Twitter/X Scraper (Python) A powerful Apify actor to scrape Twitter/X data without requiring authentication. Uses Playwright for browser automation and extracts data directly from the Twitter/X website. ## Features ✅ No Authentication Required - Works without login cookies ✅ Search Tweets - Find tweets by keywords, hashtags ✅ User Timelines - Get tweets from any public user ✅ User Profiles - Extract profile information ✅ Media & Replies - Filter for specific content types ✅ Engagement Metrics - Likes, retweets, replies, views ✅ Anti-Detection - Built-in stealth features ## How It Works 1. Playwright Browser - Uses a real browser to access Twitter 2. Guest Token - Obtains guest credentials automatically 3. DOM Extraction - Parses tweets directly from the page 4. Infinite Scroll - Automatically scrolls to load more tweets ## Input Parameters ### Search Configuration | Parameter | Type | Description | |-----------|------|-------------| | search_terms | Array | Search queries (e.g., #AI, bitcoin) | | usernames | Array | Twitter usernames to scrape (without @) | | tweet_urls | Array | Direct URLs to specific tweets | ### Scraping Options | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | scrape_type | String | tweets | What to scrape: tweets, profile, replies, media | | max_tweets | Integer | 50 | Maximum tweets per search/user | | include_replies | Boolean | false | Include reply tweets | | include_retweets | Boolean | true | Include retweets | ### Filters | Parameter | Type | Description | |-----------|------|-------------| | min_likes | Integer | Minimum likes filter | | min_retweets | Integer | Minimum retweets filter | ## Output Format ### Tweet Data json { "tweet_id": "1234567890", "text": "Tweet content here...", "author_name": "Elon Musk", "author_username": "elonmusk", "likes": 50000, "retweets": 10000, "replies": 5000, "views": 1000000, "created_at": "2024-01-15T10:30:00.000Z", "url": "https://x.com/elonmusk/status/1234567890", "is_retweet": false, "is_reply": false, "media": [{"type": "image", "url": "..."}], "hashtags": ["#tech"], "mentions": ["@OpenAI"], "scraped_at": "2024-01-15T12:00:00.000Z" } ### Profile Data json { "name": "Elon Musk", "username": "elonmusk", "bio": "...", "location": "Austin, TX", "website": "tesla.com", "join_date": "Joined June 2009", "followers": "170M", "following": "500", "is_verified": true, "profile_image_url": "...", "url": "https://x.com/elonmusk", "scraped_at": "2024-01-15T12:00:00.000Z" } ## Local Development ### Prerequisites - Python 3.9+ - pip ### Setup bash # Navigate to the actor directory cd twitter-scraper-python # Create virtual environment python -m venv venv # Activate (Windows) .\venv\Scripts\activate # Activate (Linux/Mac) source venv/bin/activate # Install dependencies pip install -r requirements.txt # Install Playwright browsers playwright install chromium ### Run Locally bash # Using Apify CLI apify run # Or directly python -m src.main ### Test Input Edit apify_storage/key_value_stores/default/INPUT.json: json { "usernames": ["elonmusk"], "max_tweets": 10 } ## Deployment to Apify bash # Login to Apify apify login # Push to Apify apify push ## Limitations - Rate Limits: Twitter may rate-limit after many requests - Login Walls: Some content requires authentication - Dynamic Changes: Twitter's DOM may change ## Tips - Use residential proxies for better success rates - Start with small limits to test - Use Apify Proxy when deploying to Apify platform ## License ISC License