Twitter Profiles and Tweets Scraper ## Overview This powerful scraper allows you to extract comprehensive data from Twitter (X) profiles and their tweets. It's designed to work with various input formats, including full profile URLs, usernames, and even Instagram profile URLs. ## Features - Scrape detailed profile information - Extract tweets with comprehensive metadata - Supports multiple input URLs, including Twitter handles and profile URLs. - Username-only input support - Configurable concurrency and retry mechanisms - Optional inclusion of posts/tweets - Proxy configuration for enhanced reliability - Extracts tweets, user information, engagement metrics, tweet sources, and more. ## Input Configuration 1. Start URLs: - Full profile URLs: https://x.com/elonmusk or https://twitter.com/elonmusk - Usernames: adrian_horning_ - Instagram URLs (for cross-platform profiles): https://www.instagram.com/robertsyfert/ 2. Max Concurrency: Set maximum concurrent requests (default: 10) 3. Min Concurrency: Set minimum concurrent requests (default: 1) 4. Max Request Retries: Set maximum retry attempts for failed requests (default: 100) ## Input Data Here's an example of how to set up a Twitter profile scraping task: json { "startUrls": [ "https://x.com/elonmusk", "elonmusk", "https://www.instagram.com/robertsyfert/" ], "maxConcurrency": 10, "minConcurrency": 1, "maxRequestRetries": 100 } ## Output Structure json ,{ "type": "tweet", "id": "1519480761749016577", "url": "https://x.com/elonmusk/status/1519480761749016577", "twitterUrl": "https://twitter.com/elonmusk/status/1519480761749016577", "text": "Next I'm buying Coca-Cola to put the cocaine back in", "retweetCount": 620127, "replyCount": 179263, "likeCount": 4434571, "quoteCount": 166686, "createdAt": "Thu Apr 28 00:56:58 +0000 2022", "lang": "en", "quoteId": null, "bookmarkCount": 21208, "isReply": false, "card": {}, "place": {}, "source": "<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>", "author": { "type": "user", "userName": "elonmusk", "url": "https://x.com/elonmusk", "twitterUrl": "https://twitter.com/elonmusk", "id": "44196397", "name": "Elon Musk", "isVerified": true, "isBlueVerified": true, "verifiedType": "none", "hasNftAvatar": false, "profilePicture": "https://pbs.twimg.com/profile_images/1845482317860450309/OrD0ovmf_normal.jpg", "coverPicture": "https://pbs.twimg.com/profile_banners/44196397/1726163678", "description": "Read @America to understand why I'm supporting Trump for President", "location": "", "followers": 201720551, "following": 787, "protected": false, "status": "", "canDm": false, "canMediaTag": false, "advertiserAccountType": "promotable_user", "analyticsType": "enabled", "createdAt": "Tue Jun 02 20:12:29 +0000 2009", "entities": { "description": { "urls": [] }, "url": { "urls": [ { "display_url": "TheAmericaPAC.org", "expanded_url": "http://TheAmericaPAC.org", "url": "https://t.co/DjyKIO6ePx", "indices": [ 0, 23 ] } ] } }, "fastFollowersCount": 0, "favouritesCount": 81232, "geoEnabled": false, "hasCustomTimelines": true, "hasExtendedProfile": false, "isTranslator": false, "mediaCount": 2592, "profileBackgroundColor": "C0DEED", "statusesCount": 54347, "translatorTypeEnum": "None", "withheldInCountries": [], "affiliatesHighlightedLabel": { "label": { "url": { "url": "https://twitter.com/X", "urlType": "DeepLink" }, "badge": { "url": "https://pbs.twimg.com/profile_images/1683899100922511378/5lY42eHs_bigger.jpg" }, "description": "X", "userLabelType": "BusinessLabel", "userLabelDisplayType": "Badge" } } }, "extendedEntities": {}, "isRetweet": false, "isQuote": false, "media": [], "otherData": { "__typename": "Tweet", "rest_id": "1828402665845322123", "core": { "user_results": { "result": { "__typename": "User", "id": "VXNlcjo0NTIwMjQxMjA5", "rest_id": "4520241209", "affiliates_highlighted_label": {}, "is_blue_verified": true, "profile_image_shape": "Circle", "legacy": { "created_at": "Fri Dec 18 02:48:59 +0000 2015", "default_profile": false, "default_profile_image": false, "description": "Public Social Media Scraping API's: https://t.co/eSvJcfOZwF", "entities": { "description": { "urls": [ { "display_url": "scrapecreators.com", "expanded_url": "https://scrapecreators.com/", "url": "https://t.co/eSvJcfOZwF", "indices": [ 36, 59 ] } ] }, "url": { "urls": [ { "display_url": "thewebscrapingguy.com", "expanded_url": "https://thewebscrapingguy.com/", "url": "https://t.co/gNUelkV9LA", "indices": [ 0, 23 ] } ] } }, "fast_followers_count": 0, "favourites_count": 97936, "followers_count": 18248, "friends_count": 1237, "has_custom_timelines": true, "is_translator": false, "listed_count": 177, "location": "Austin, TX", "media_count": 1331, "name": "Adrian | The Web Scraping Guy", "normal_followers_count": 18248, "pinned_tweet_ids_str": [ "1911900126529958135" ], "possibly_sensitive": false, "profile_banner_url": "https://pbs.twimg.com/profile_banners/4520241209/1710267319", "profile_image_url_https": "https://pbs.twimg.com/profile_images/1413647704161275904/1tTdl4v9_normal.jpg", "profile_interstitial_type": "", "screen_name": "adrian_horning_", "statuses_count": 20251, "translator_type": "none", "url": "https://t.co/gNUelkV9LA", "verified": false, "withheld_in_countries": [] }, "professional": { "rest_id": "1554172330263339015", "professional_type": "Business", "category": [] }, "tipjar_settings": { "is_enabled": false, "bandcamp_handle": "", "bitcoin_handle": "", "cash_app_handle": "", "ethereum_handle": "", "gofundme_handle": "", "patreon_handle": "", "pay_pal_handle": "", "venmo_handle": "" } } } }, "unmention_data": {}, "edit_control": { "edit_tweet_ids": [ "1828402665845322123" ], "editable_until_msecs": "1724763740000", "is_edit_eligible": false, "edits_remaining": "5" }, "is_translatable": false, "views": { "count": "494856", "state": "EnabledWithCount" }, "source": "<a href=\"https://tweethunter.io\" rel=\"nofollow\">Tweet Hunter Pro</a>", "legacy": { "bookmark_count": 1021, "bookmarked": false, "created_at": "Tue Aug 27 12:02:20 +0000 2024", "conversation_id_str": "1828402665845322123", "display_text_range": [ 0, 216 ], "entities": { "hashtags": [], "media": [ { "display_url": "pic.x.com/fh8xBzkrKI", "expanded_url": "https://x.com/adrian_horning_/status/1828402665845322123/photo/1", "id_str": "1828402662682841089", "indices": [ 217, 240 ], "media_key": "3_1828402662682841089", "media_url_https": "https://pbs.twimg.com/media/GV_KiM-W0AEa31c.png", "type": "photo", "url": "https://t.co/fh8xBzkrKI", "ext_media_availability": { "status": "Available" }, "features": { "large": { "faces": [] }, "medium": { "faces": [] }, "small": { "faces": [] }, "orig": { "faces": [] } }, "sizes": { "large": { "h": 117, "w": 182, "resize": "fit" }, "medium": { "h": 117, "w": 182, "resize": "fit" }, "small": { "h": 117, "w": 182, "resize": "fit" }, "thumb": { "h": 117, "w": 117, "resize": "crop" } }, "original_info": { "height": 117, "width": 182, "focus_rects": [ { "x": 0, "y": 0, "w": 182, "h": 102 }, { "x": 28, "y": 0, "w": 117, "h": 117 }, { "x": 35, "y": 0, "w": 103, "h": 117 }, { "x": 57, "y": 0, "w": 59, "h": 117 }, { "x": 0, "y": 0, "w": 182, "h": 117 } ] }, "media_results": { "result": { "media_key": "3_1828402662682841089" } } } ], "symbols": [], "timestamps": [], "urls": [], "user_mentions": [] }, "extended_entities": { "media": [ { "display_url": "pic.x.com/fh8xBzkrKI", "expanded_url": "https://x.com/adrian_horning_/status/1828402665845322123/photo/1", "id_str": "1828402662682841089", "indices": [ 217, 240 ], "media_key": "3_1828402662682841089", "media_url_https": "https://pbs.twimg.com/media/GV_KiM-W0AEa31c.png", "type": "photo", "url": "https://t.co/fh8xBzkrKI", "ext_media_availability": { "status": "Available" }, "features": { "large": { "faces": [] }, "medium": { "faces": [] }, "small": { "faces": [] }, "orig": { "faces": [] } }, "sizes": { "large": { "h": 117, "w": 182, "resize": "fit" }, "medium": { "h": 117, "w": 182, "resize": "fit" }, "small": { "h": 117, "w": 182, "resize": "fit" }, "thumb": { "h": 117, "w": 117, "resize": "crop" } }, "original_info": { "height": 117, "width": 182, "focus_rects": [ { "x": 0, "y": 0, "w": 182, "h": 102 }, { "x": 28, "y": 0, "w": 117, "h": 117 }, { "x": 35, "y": 0, "w": 103, "h": 117 }, { "x": 57, "y": 0, "w": 59, "h": 117 }, { "x": 0, "y": 0, "w": 182, "h": 117 } ] }, "media_results": { "result": { "media_key": "3_1828402662682841089" } } } ] }, "favorite_count": 1710, "favorited": false, "full_text": "I just scraped 2.8 million companies from crunchbase 🤯\n\nName, website, semrush stats, etc.\n\nI'm giving the entire thing away in the next 24 hours\n\nComment \"crunchbase\" and I'll send it to you. Make sure DM's are open https://t.co/fh8xBzkrKI", "is_quote_status": false, "lang": "en", "possibly_sensitive": false, "possibly_sensitive_editable": true, "quote_count": 11, "reply_count": 3149, "retweet_count": 63, "retweeted": false, "user_id_str": "4520241209", "id_str": "1828402665845322123" }, "quick_promote_eligibility": { "eligibility": "IneligibleUserUnauthorized" } } } ## Output Format The scraper provides rich, structured data for both profiles and tweets. Here's a detailed breakdown of the output fields: ### Tweet Object | Field | Type | Description | | ---------------- | ------- | ----------------------------------------------- | | type | String | Always "tweet" for tweet objects | | id | String | Unique identifier of the tweet | | url | String | Full URL to the tweet on X.com | | twitterUrl | String | Full URL to the tweet on Twitter.com | | text | String | Content of the tweet | | retweetCount | Number | Count of retweets | | replyCount | Number | Count of replies | | likeCount | Number | Count of likes | | quoteCount | Number | Count of quote tweets | | createdAt | String | Timestamp of tweet creation | | lang | String | Language code of the tweet | | quoteId | String | null | | bookmarkCount | Number | Count of bookmarks | | isReply | Boolean | Indicates if the tweet is a reply | | card | Object | Additional card information (e.g., for links) | | place | Object | Location information, if available | | source | String | Application used to post the tweet | | isRetweet | Boolean | Indicates if the tweet is a retweet | | isQuote | Boolean | Indicates if the tweet is a quote tweet | | media | Array | Contains media objects (images, videos) | | extendedEntities | Object | Additional entities (e.g., extended media info) | | otherData | Object | Additional Twitter API data including: | | | | - __typename: Type of the object (e.g., 'Tweet') | | | | - rest_id: Unique identifier for the tweet | | | | - core.user_results: User information | | | | - edit_control: Tweet edit metadata including: | | | | - edit_tweet_ids: Array of tweet edit IDs | | | | - editable_until_msecs: Timestamp until which the tweet can be edited | | | | - is_edit_eligible: Whether the tweet is eligible for editing | | | | - edits_remaining: Number of remaining edits | | | | - is_translatable: Whether the tweet can be translated | | | | - views: View count information | | | | - source: Source of the tweet (e.g., 'Twitter Web App') | | | | - legacy: Legacy Twitter API data including: | | | | - bookmark_count: Number of bookmarks | | | | - created_at: Creation timestamp | | | | - conversation_id_str: Conversation ID | | | | - display_text_range: Text range for display | | | | - entities: Tweet entities (hashtags, mentions, etc.) | | | | - favorite_count: Number of likes | | | | - full_text: Full text of the tweet | | | | - is_quote_status: Whether this is a quote tweet | | | | - quote_count: Number of quotes | | | | - reply_count: Number of replies | | | | - retweet_count: Number of retweets | | | | - user_id_str: User ID of the tweet author | | | | - quick_promote_eligibility: Promotion eligibility status | ### Author Object (User Profile) | Field | Type | Description | | -------------------------- | ------- | ----------------------------------------- | | type | String | Always "user" for profile objects | | userName | String | Twitter handle of the user | | url | String | Profile URL on X.com | | twitterUrl | String | Profile URL on Twitter.com | | id | String | Unique identifier of the user | | name | String | Display name of the user | | isVerified | Boolean | Legacy verification status | | isBlueVerified | Boolean | Twitter Blue verification status | | verifiedType | String | Type of verification | | hasNftAvatar | Boolean | Indicates if user has an NFT avatar | | profilePicture | String | URL of the profile picture | | coverPicture | String | URL of the profile banner/cover | | description | String | User's bio text | | location | String | User's location (if provided) | | followers | Number | Count of followers | | following | Number | Count of accounts followed | | protected | Boolean | Indicates if the account is private | | status | String | Current status (if set) | | canDm | Boolean | Indicates if user can receive DMs | | canMediaTag | Boolean | Indicates if user can be tagged in media | | advertiserAccountType | String | Type of advertiser account | | analyticsType | String | Type of analytics enabled | | createdAt | String | Timestamp of account creation | | entities | Object | URLs and mentions in the bio | | fastFollowersCount | Number | Count of fast followers | | favouritesCount | Number | Count of liked tweets | | geoEnabled | Boolean | Indicates if geolocation is enabled | | hasCustomTimelines | Boolean | Indicates custom timeline usage | | hasExtendedProfile | Boolean | Indicates extended profile features | | isTranslator | Boolean | Indicates if user is a Twitter translator | | mediaCount | Number | Count of media posts | | profileBackgroundColor | String | Hex code of profile background color | | statusesCount | Number | Total count of tweets/posts | | translatorTypeEnum | String | Type of translator status | | withheldInCountries | Array | Countries where account is withheld | | affiliatesHighlightedLabel | Object | Information about affiliated labels | ## Usage Tips 1. For bulk scraping, use the "Bulk edit" feature to input multiple profile URLs or usernames. 2. Adjust concurrency settings based on your network capabilities and Twitter's rate limits. 3. Use proxy configuration for large-scale scraping to avoid IP blocks. ## Explore More Scrapers If you found this Apify Smartbuyglasses Scraper useful, be sure to check out our other powerful scrapers and actors at memo23's Apify profile. We offer a wide range of tools to enhance your web scraping and automation needs across various platforms and use cases. ## Support - For issues or feature requests, please use the Issues section of this actor. - If you need customization or have questions, feel free to contact the author: - Author's website: https://muhamed-didovic.github.io/ - Email: muhamed.didovic@gmail.com ## Additional Services - Request customization or whole dataset: muhamed.didovic@gmail.com - If you need anything else scraped, or this actor customized, email: muhamed.didovic@gmail.com - For API services of this scraper (no Apify fee, just usage fee for the API), contact: muhamed.didovic@gmail.com - Email: muhamed.didovic@gmail.com