Twitter (X) Comment Scraper

# Twitter (X) Comment Scraper This Apify actor is designed to scrape comments from a specific tweet. ## Features This actor provides comprehensive data collection capabilities from Twitter comments. It scrapes comments from a specific tweet and extracts detailed information including user profiles, engagement metrics, and media content. The collected data encompasses user details such as username, follower counts, and verification status, along with tweet-specific information including content, timestamps, and engagement statistics. For media-rich tweets, it captures various media types including images, videos, and GIFs, along with their associated metadata. The actor offers flexible comment sorting options, allowing users to sort comments by relevance, newest, or most liked to better suit their data collection needs. The scraper is designed to handle Twitter's modern interface and can collect data such as: - Comprehensive user profile information including bio, location, tweet counts, and Twitter Blue verification status - Complete tweet content with engagement metrics (likes, retweets, replies, quotes) - Advanced statistics like bookmark counts and view counts - Detailed media information including video quality and duration - Tweet metadata such as creation time, language, source application, and entities (hashtags, mentions) - Conversation details like reply-to and conversation IDs - Flexible comment sorting options (relevance, newest, most liked) ### Obtaining Cookie Information You can use the Cookie Editor extension to obtain Twitter cookie information. Follow these steps: 1. Add the Cookie Editor extension to your Chrome browser. 2. Log in to Twitter. 3. Click on the Cookie Editor icon in the top right corner of your browser. 4. Click the "Export" button to export your cookie information in JSON format. 5. Use this JSON string as the cookieJson input. Note: You can validate with a tool like JSONLint to make sure your JSON input is correct. Note: If you encounter errors or if 0 comments are processed, ensure that your account is functioning normally and log in again to obtain new cookies to use as input. ## Usage 1. Run this actor in the Apify console. 2. Provide the desired inputs: - tweetUrl: The URL of the tweet you want to scrape comments from. - cookieJson: A JSON string containing your Twitter account's cookie information. - maxDepth: Maximum comment depth (0 = unlimited). - sortBy: Sort comments by relevance (default), newest, or most liked. - onlyDirectReplies: Show only replies to the tweet owner (default: true). ## Output The scraped comments are saved to the Apify dataset. The output data includes: ### User Information - user.id: Unique user ID - user.name: Username - user.screen_name: User tag - user.description: User's biography - user.location: User's location - user.followers_count: Number of followers - user.following_count: Number of following - user.tweet_count: Total number of tweets by the user - user.media_count: Number of media items uploaded by the user - user.profile_image: Profile image URL - user.profile_banner: Profile banner URL - user.is_blue_verified: Twitter Blue verification status - user.professional_info: Professional account details (if available) ### Tweet Information - tweet_id: Unique ID of the tweet/comment - conversation_id: ID of the main conversation thread - in_reply_to_tweet_id: ID of the tweet being replied to - is_quote_tweet: Whether the tweet is a quote tweet - text: Tweet content - entities: Hashtags, mentions, and URLs within the tweet - reply_to_username: Username being replied to - favorite_count: Number of likes - reply_count: Number of replies - retweet_count: Number of retweets - quote_count: Number of quotes - bookmark_count: Number of bookmarks - views_count: Number of views - created_at: Creation timestamp - lang: Tweet language - source: Tweet source ### Media Information - media: Media content (images, videos, GIFs) - type: Media type - thumb_url: Thumbnail URL - url: Media URL - video_url: Video URL (if applicable) - duration_ms: Video duration (if applicable) ## Example Output json { "tweet_id": "1951027633937899850", "conversation_id": "1951026104422244393", "in_reply_to_tweet_id": "1951026104422244393", "is_quote_tweet": false, "user": { "id": "1390762874809761793", "name": "Defiant L’s", "screen_name": "DefiantLs", "followers_count": 1592063, "following_count": 166, "description": "Exposing hypocrisy, one day at a time. | DM submissions", "location": "", "profile_image": "https://pbs.twimg.com/profile_images/1428167484406964229/Q8fS7M5X_normal.jpg", "profile_banner": "https://pbs.twimg.com/profile_banners/1390762874809761793/1710377019", "is_blue_verified": true, "tweet_count": 40291, "lists_count": 3760, "media_count": 26205, "profile_url": "https://t.co/xxxjOwrKeu", "professional_info": null }, "text": "@elonmusk Can't wait https://t.co/LaXXWUbGky", "entities": { "hashtags": [], "media": [ { "display_url": "pic.x.com/LaXXWUbGky", "expanded_url": "https://x.com/DefiantLs/status/1951027633937899850/photo/1", "id_str": "1951027625507377152", "indices": [ 21, 44 ], "media_key": "3_1951027625507377152", "media_url_https": "https://pbs.twimg.com/media/GxNxSdVXUAA8GnX.jpg", "type": "photo", "url": "https://t.co/LaXXWUbGky", "ext_media_availability": { "status": "Available" }, "features": { "large": { "faces": [ { "x": 512, "y": 88, "h": 294, "w": 294 }, { "x": 260, "y": 225, "h": 250, "w": 250 } ] }, "medium": { "faces": [ { "x": 512, "y": 88, "h": 294, "w": 294 }, { "x": 260, "y": 225, "h": 250, "w": 250 } ] }, "small": { "faces": [ { "x": 340, "y": 58, "h": 195, "w": 195 }, { "x": 172, "y": 149, "h": 166, "w": 166 } ] }, "orig": { "faces": [ { "x": 512, "y": 88, "h": 294, "w": 294 }, { "x": 260, "y": 225, "h": 250, "w": 250 } ] } }, "sizes": { "large": { "h": 1024, "w": 1024, "resize": "fit" }, "medium": { "h": 1024, "w": 1024, "resize": "fit" }, "small": { "h": 680, "w": 680, "resize": "fit" }, "thumb": { "h": 150, "w": 150, "resize": "crop" } }, "original_info": { "height": 1024, "width": 1024, "focus_rects": [ { "x": 0, "y": 0, "w": 1024, "h": 573 }, { "x": 0, "y": 0, "w": 1024, "h": 1024 }, { "x": 126, "y": 0, "w": 898, "h": 1024 }, { "x": 435, "y": 0, "w": 512, "h": 1024 }, { "x": 0, "y": 0, "w": 1024, "h": 1024 } ] }, "allow_download_status": { "allow_download": true }, "media_results": { "result": { "media_key": "3_1951027625507377152" } } } ], "symbols": [], "timestamps": [], "urls": [], "user_mentions": [ { "id_str": "44196397", "name": "Elon Musk", "screen_name": "elonmusk", "indices": [ 0, 9 ] } ] }, "reply_to_username": "elonmusk", "reply_to_user_id": "44196397", "favorite_count": 321, "reply_count": 46, "retweet_count": 54, "quote_count": 6, "bookmark_count": 21, "views_count": "38652", "created_at": "Thu Jul 31 21:10:10 +0000 2025", "lang": "en", "source": "Twitter for iPhone", "media": [ { "type": "photo", "thumb_url": "https://pbs.twimg.com/media/GxNxSdVXUAA8GnX.jpg", "url": "https://x.com/DefiantLs/status/1951027633937899850/photo/1" } ] } This example output shows the structure of a single comment. The actual output will be a list of similar objects for all scraped comments. ## Notes - It is designed to scrape comments visible on the tweet screen. Therefore, it cannot process nested comments and can retrieve a maximum of 200 comments due to limitations. - The collected data is stored in Apify’s default data store.