Twitter Video Downloader A powerful Apify Actor that downloads videos from Twitter/X posts with support for both direct post URLs and username-based scraping. Built for reliable video downloads and authenticated profile scraping. A template for web scraping data from a single web page in Python. The URL of the web page is passed in via input, which is defined by the input schema. The template uses the HTTPX to get the HTML of the page and the Beautiful Soup to parse the data from it. The data are then stored in a dataset where you can easily access them. The scraped data in this template are page headings but you can easily edit the code to scrape whatever you want from the page. ## Included features - Apify SDK for Python - a toolkit for building Apify Actors and scrapers in Python - Input schema - define and easily validate a schema for your Actor's input - Request queue - queues into which you can put the URLs you want to scrape - Dataset - store structured data where each object stored has the same attributes - HTTPX - library for making asynchronous HTTP requests in Python - Beautiful Soup - library for pulling data out of HTML and XML files ## How It Works ### Input Modes 1. Direct Post URLs json { "postUrls": [ "https://x.com/username/status/123456789" ], "videoQuality": "best" } 2. Username-Based Scraping json { "usernames": ["elonmusk", "NASA"], "maxPostsPerUser": 20, "videoQuality": "720p" } ### Workflow 1. Input Processing - Actor accepts either post URLs or usernames 2. Profile Scraping (username mode) - Playwright browser collects post URLs from user profiles 3. Video Download - downloads videos with selected quality 4. Metadata Extraction - OpenCV extracts video specifications (resolution, FPS, codec, duration, bitrate) 5. Cloud Upload - Videos uploaded to Apify key-value store 6. Dataset Output - Structured data saved with download links ## Output Each downloaded video includes: json { "tweet_url": "https://x.com/username/status/123456789", "id": "123456789", "title": "Tweet title", "uploader": "username", "filename": "123456789.mp4", "resolution": "1920x1080", "download_status": "finished", "download_url": "https://api.apify.com/v2/key-value-stores/.../records/video_123456789.mp4", "storage_key": "video_123456789.mp4", "downloaded_at": "2025-11-13T00:00:00.000000", "video_meta": { "width": 1920, "height": 1080, "fps": 30.0, "duration": 45.23, "video_codec": "avc1", "ext": "mp4", "filesize_bytes": 28640256, "total_bitrate_kbps": 5064.12, "aspect_ratio": 1.78 } } ## Input Parameters | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | postUrls | Array | No | [] | List of Twitter/X post URLs to download | | usernames | Array | No | [] | List of usernames to scrape videos from (without @) | | maxPostsPerUser | Integer | No | 20 | Maximum posts to scrape per username (1-100) | | videoQuality | String | No | best | Video quality: best, 720p, 480p, worst | | browserCookies | String | No | - | Browser cookies in JSON format (for profile scraping) | | minDelayBetweenRequests | Integer | No | 2 | Minimum delay between requests (1-30 seconds) | | maxDelayBetweenRequests | Integer | No | 5 | Maximum delay between requests (1-60 seconds) | | humanizeBehavior | Boolean | No | true | Enable human-like mouse movements and scrolling | ## Use Cases ### 1. Content Archival Archive videos from your favorite Twitter accounts before they're deleted: json { "usernames": ["NASA", "SpaceX"], "maxPostsPerUser": 50, "videoQuality": "best" } ### 2. Research & Analysis Collect videos for social media research: json { "usernames": ["researcher1", "researcher2"], "maxPostsPerUser": 100, "videoQuality": "720p" } ### 3. Specific Video Download Download individual videos: json { "postUrls": [ "https://x.com/username/status/123456", "https://x.com/username/status/789012" ], "videoQuality": "best" } ## Cookie Configuration ### Why Use Cookies? Cookies help avoid rate limits and access protected content. The actor uses a dual cookie system: 1. Browser Cookies (browserCookies parameter) - For Playwright profile scraping 2. cookies.txt File - For reliable video downloads ### Getting Cookies Option 1: Browser Extension 1. Install "Get cookies.txt" extension for your browser 2. Navigate to x.com and log in 3. Click extension icon to export cookies 4. Save as cookies.txt in actor root directory Option 2: Manual Export Use browser DevTools to export cookies in JSON format and paste into browserCookies input field. ### Default Cookies The actor includes default browser cookies for basic profile scraping. For heavy usage, provide your own authenticated cookies. ## Anti-Detection Features - Human-like delays - Random delays between 2-5 seconds (configurable) - Mouse movements - Simulated cursor movements during scraping - Scroll behavior - Natural scrolling patterns - Firefox browser - Less detectable than Chrome for automation - Custom user agents - Realistic browser fingerprints - Rate limit protection - Automatic detection of blocks and warnings ## Local Development ### Prerequisites - Python 3.12+ - Apify CLI (npm install -g apify-cli) ### Setup 1. Clone the repository: bash cd Twitter/twitter-video-downloader-cli 2. Install dependencies: bash pip install -r requirements.txt playwright install firefox 3. Create input file .actor/INPUT.json: json { "postUrls": ["https://x.com/username/status/123456"], "videoQuality": "best" } 4. (Optional) Add cookies.txt for authenticated downloads 5. Run locally: bash apify run ### Testing The actor creates local storage in ./storage/: - datasets/default/ - Output data with metadata - key_value_stores/default/ - Input configuration - Downloaded videos are uploaded to cloud storage ## Deployment ### Deploy to Apify Platform 1. Login to Apify: bash apify login 2. Push to Apify: bash apify push 3. Configure Secrets (optional): - Add cookies.txt as a secret file in Actor settings - Set environment variables for sensitive data ### Deploy from GitHub 1. Go to Apify Console 2. Click "Link Git Repository" 3. Connect your GitHub repository 4. Configure build settings: - Build tag: latest - Dockerfile: ./Dockerfile ## Performance & Limits ### Speed - Profile scraping: ~2-5 seconds per username - Video download: 5-30 seconds per video (depends on quality/size) - Metadata extraction: < 100ms per video (OpenCV) ### Rate Limits - Default delays: 2-5 seconds between requests - Recommended max: 100 posts per run without authentication - With cookies: 500+ posts per run (monitor for blocks) ### Quality vs. File Size | Quality | Resolution | Typical Size | Use Case | |---------|-----------|--------------|----------| | best | 1080p-4K | 20-100 MB | Archival, high quality | | 720p | 1280x720 | 10-40 MB | Balanced quality/size | | 480p | 854x480 | 5-15 MB | Fast downloads | | worst | 360p-240p | 2-8 MB | Bandwidth limited | ## Troubleshooting ### "No video found" Error - Tweet may not contain a video (only images/GIFs) - Video may be age-restricted or geo-blocked - Try adding cookies.txt for authentication ### Rate Limited / Blocked - Increase minDelayBetweenRequests and maxDelayBetweenRequests - Enable humanizeBehavior - Add authenticated browser cookies - Reduce maxPostsPerUser ### Video Quality Issues - our actor selects best available format within quality constraint - Some tweets only have lower quality versions - Check video_meta in output for actual resolution ### Missing Metadata (FPS, Duration) - Some HLS streams don't expose all metadata upfront - Metadata is extracted post-download via OpenCV - Check logs for extraction errors ## API Integration ### Fetch Results python from apify_client import ApifyClient client = ApifyClient('YOUR_API_TOKEN') # Start actor run run = client.actor('YOUR_ACTOR_ID').call(run_input={ "usernames": ["NASA"], "maxPostsPerUser": 10, "videoQuality": "best" }) # Fetch dataset items dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items for item in dataset_items: print(f"Video: {item['download_url']}") ### Download Videos Videos are stored in key-value store. Use the download_url from output: bash curl -o video.mp4 "https://api.apify.com/v2/key-value-stores/{STORE_ID}/records/{KEY}" ## Limitations - Only downloads videos (not images or GIFs) - Requires video to be embedded in tweet (not external links) - Private/protected accounts require authenticated cookies - Age-restricted content requires authentication - Twitter's rate limits apply (use delays and cookies) ## Support - Apify Discord Community - Apify Documentation ## License This project is for educational and personal use. Respect Twitter's Terms of Service and robots.txt. Always ensure you have rights to download and use the content. --- Built with ❤️ by CrawlerBros