Twitter Follower Scraper A Twitter Scraper that scrapes followers/following and verified follower/following data using authenticated requests. This scraper can extract detailed user information including verification status, profile data, and social metrics. ## Features - Multiple Scraping Types: Scrape followers, following, or filter for verified users only - Batch Processing: Process multiple usernames in a single run - Rich User Data: Extract comprehensive profile information including: - Basic profile data (name, username, description) - Social metrics (followers, following, tweets count) - Verification status and details - Profile images (standard and high-resolution) - Account creation date and location - Verification Filtering: Option to scrape only verified users - Rate Limit Handling: Built-in delays to respect Twitter's API limits - Error Handling: Robust error handling with detailed logging ## Input Parameters Configure the scraper using these input parameters: ### Required Parameters | Parameter | Type | Description | |-----------|------|-------------| | username | Array[String] | List of Twitter usernames to scrape (without @ symbol) | | cookies | Array[Object] | Authentication cookies from your Twitter session | ### Optional Parameters | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | scrape_type | String | "followers" | Type of data to scrape | | follower_limit | Number | 100 | Maximum number of followers/following to scrape per username | ### Scrape Types - "followers" - Scrape followers of the specified users - "following" - Scrape users that the specified users are following - "verified_followers" - Scrape only verified followers - "verified_following" - Scrape only verified users that the specified users are following ## Authentication Setup This scraper requires Twitter authentication cookies. Here's how to obtain them: 1. Login to Twitter in your browser 2. Open Developer Tools (F12) 3. Go to Application/Storage tab → Cookies → https://twitter.com 4. Export the following cookies: json [ {"name": "auth_token", "value": "your_auth_token_value"}, {"name": "ct0", "value": "your_ct0_value"}, {"name": "twid", "value": "your_twid_value"} ] ### Cookie Format Each cookie object should contain: json { "name": "cookie_name", "value": "cookie_value" } ## Input Example json { "username": ["elonmusk", "sundarpichai", "satyanadella"], "scrape_type": "followers", "follower_limit": 500, "cookies": [ {"name": "auth_token", "value": "your_auth_token"}, {"name": "ct0", "value": "your_csrf_token"}, {"name": "twid", "value": "your_twitter_id"} ] } ## Output Data The scraper extracts the following data for each user: ### Basic Information - id - Twitter user ID - username - Twitter handle (without @) - name - Display name - description - Bio/description ### Social Metrics - followers_count - Number of followers - following_count - Number of accounts following - tweets_count - Total number of tweets ### Verification Details - verified - Overall verification status (boolean) - verification_details - Detailed verification information: - verified_type - Type of verification - blue_verified - Twitter Blue verification - legacy_verified - Legacy blue checkmark - government_verified - Government account verification - business_verified - Business account verification ### Profile Assets - profile_image_url - Standard profile image URL - profile_image_url_hd - High-resolution profile image URL - profile_banner_url - Profile banner image URL - profile_background_image_url - Background image URL ### Additional Data - created_at - Account creation date - location - User's location (if provided) - url - Website URL (if provided) - protected - Whether the account is private - scraped_from - Username this data was scraped from - scrape_type - Type of scraping performed ## Usage Tips 1. Start Small: Begin with a low follower_limit to test your setup 2. Monitor Rate Limits: The scraper includes delays, but be mindful of Twitter's limits 3. Verified Users: Use verified filtering to get higher-quality data 4. Batch Processing: Process multiple usernames efficiently in one run 5. Data Storage: Results are automatically saved to Apify dataset ## Rate Limiting The scraper implements several rate limiting strategies: - 1-second delay between pagination requests - 2-second delay between different usernames - Automatic error handling for rate limit responses ## Error Handling The scraper handles various error scenarios: - Invalid usernames - Authentication failures - Network connectivity issues - Rate limiting - Private/suspended accounts All errors are logged with detailed information for debugging. ## Requirements - Apify Account: Required to run this actor - Valid Twitter Cookies: Fresh authentication cookies from an active Twitter session - Internet Connection: For API requests to Twitter ## Limitations - Requires valid Twitter authentication - Subject to Twitter's rate limiting - Cannot access private accounts unless you follow them - Suspended or deleted accounts will be skipped ## Troubleshooting ### Authentication Issues - Ensure cookies are fresh (less than 24 hours old) - Check that all required cookies are provided - Verify cookie values are complete and unmodified ### Rate Limiting - Reduce follower_limit if hitting rate limits - Increase delays between requests if needed - Use fewer usernames per run ### No Data Returned - Verify usernames exist and are public - Check if accounts have followers/following to scrape - Ensure proper scrape_type is selected ## Legal and Ethical Considerations - Respect Twitter's Terms of Service - Use scraped data responsibly - Consider privacy implications - Don't overwhelm Twitter's servers - Comply with applicable data protection laws