TikTok Hashtag Scraper with Video Downloader A powerful and efficient Apify actor that scrapes TikTok videos by hashtag, extracting comprehensive metadata and optionally downloading videos directly to Apify storage for easy access. ## 🌟 Features - 🏷️ Multi-Hashtag Scraping - Scrape videos from multiple hashtags in a single run - ⚡ Parallel Processing - Fast data extraction using batch processing (3x faster) - 📥 Video Download - Download videos in HD quality and store them in Apify's key-value store - 📊 Rich Metadata - Comprehensive data including: - Author details (username, nickname, avatar, verification status, bio) - Video metadata (duration, resolution, cover image, download URL) - Music information (song title, artist, cover image) - Engagement statistics (likes, comments, shares, views, saves) - Timestamps (creation date, scrape date) - Download information (storage key, quality, file size) - 🔐 Proxy Support - Built-in Apify Proxy integration for reliable access - 🎯 Smart Extraction - Uses multiple extraction methods (JSON + DOM) for reliability - 💾 Real-time Export - Data pushed to dataset as it's collected - 🎬 HD Quality - Automatically downloads videos in highest quality available ## 🚀 Quick Start 1. Add hashtags you want to scrape (without # symbol) 2. Set number of results per hashtag (1-100) 3. Enable video download (optional) to save videos to storage 4. Enable Apify Proxy (RESIDENTIAL recommended) 5. Run the actor and watch results appear in real-time! ## 📥 Input Configuration ### Required Settings | Field | Type | Description | |-------|------|-------------| | Hashtags | Array | List of hashtags to scrape (e.g., ["travel", "food", "tech"]) | ### Optional Settings | Field | Type | Default | Description | |-------|------|---------|-------------| | Results Per Hashtag | Integer | 20 | Number of videos to scrape per hashtag (1-100) | | Download Videos | Boolean | false | Download actual video files to Apify key-value store (HD quality when available) | | Get Detailed Data | Boolean | true | Visit each video page for complete details (slower but more data) | | Proxy Configuration | Object | - | Apify Proxy settings (highly recommended) | | Download Covers | Boolean | false | Download video cover images (not yet implemented) | ### Example Input json { "hashtags": ["travel", "foodie", "tech"], "resultsPerPage": 30, "shouldDownloadVideos": true, "getDetailedData": true, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] } } ## 📤 Output Format Each scraped video contains: json { "videoUrl": "https://www.tiktok.com/@username/video/1234567890", "videoId": "7484313881280564526", "text": "Video description with hashtags...", "hashtag": "travel", "authorMeta": { "name": "username", "nickname": "Display Name", "avatar": "https://...", "verified": false, "signature": "User bio text" }, "videoMeta": { "duration": 15, "width": 720, "height": 1280, "coverUrl": "https://...", "downloadUrl": "https://..." }, "musicMeta": { "musicId": "7249132457248737282", "musicName": "Song Title", "musicAuthor": "Artist Name", "musicOriginal": false, "coverUrl": "https://..." }, "stats": { "likesCount": 122600, "commentsCount": 255, "sharesCount": 4867, "viewsCount": 775200, "savedCount": 13806 }, "createTime": "1742577652", "createTimeISO": "2025-03-21T17:20:52.000Z", "scrapedAt": "2024-12-14T16:39:49.802993", "downloadInfo": { "success": true, "storage_key": "video_7484313881280564526", "quality": "HD", "file_size": 2458932 } } ### Accessing Downloaded Videos When shouldDownloadVideos is enabled, videos are stored in Apify's key-value store: 1. Go to Storage tab in your actor run 2. Click on Key-Value Store 3. Find videos named video_{videoId} 4. Download or access videos directly from there Each video is stored with a unique key in the format: video_7484313881280564526 ## 🎬 Video Download Feature ### How It Works 1. Actor scrapes video metadata from hashtag pages 2. For each video, it fetches the HD download URL using tikwm.com API 3. Videos are downloaded and stored in Apify's key-value store 4. Download information is added to the dataset output ### Video Quality - HD Quality: Automatically downloads HD version when available - Fallback: Downloads standard quality if HD is unavailable - File Format: MP4 format, compatible with all devices ### Storage & Access - Videos stored in Apify Key-Value Store - Accessible via Apify Console → Storage → Key-Value Store - Can be downloaded individually or via API - Storage key format: video_{videoId} ### Performance Impact - Downloading videos increases run time (1-2 seconds per video) - Increases storage usage (videos typically 2-10 MB each) - May increase actor costs due to longer runtime and storage ## 🔒 Proxy Configuration ### Why Proxies Are Required TikTok restricts access from: - Data center IPs - Cloud server IPs (AWS, Google Cloud, etc.) - Known scraping/bot IPs Without proxy: Actor will fail to connect or get blocked quickly ### Recommended Setup json { "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] } } ### Proxy Options 1. Apify Residential Proxy (Recommended) - Best success rate for TikTok - Slower but more reliable - Higher cost but worth it 2. Apify Datacenter Proxy - Faster but may get blocked - Lower cost - Use as backup option ### Alternative Options - Use your own proxy server (configure in input) - Deploy on servers in TikTok-supported regions - Use VPN before running locally ## 📊 Performance - Speed: ~2-3 minutes for 40 videos (with detailed data) - With Download: ~3-5 minutes for 40 videos (including video downloads) - Accuracy: 95%+ data extraction success rate - Download Success: 90%+ video download success rate - Reliability: Automatic fallback mechanisms if extraction fails ## 🎯 Use Cases - Social Media Analytics - Track hashtag performance and trends - Content Research - Discover popular content in your niche - Competitor Analysis - Monitor competitor hashtags and engagement - Content Archive - Download and archive viral videos for analysis - Market Research - Understand audience preferences and behaviors - Influencer Discovery - Find creators in specific niches - Trend Monitoring - Stay updated with viral content - Content Repurposing - Download videos for editing or compilation (respect copyright) ## ⚠️ Limitations - Maximum 100 videos per hashtag per run - Video download requires stable internet connection - TikTok may change their structure (actor maintained regularly) - CAPTCHA challenges may appear (rare with good proxies) - Some videos may be region-restricted or private - Rate limiting applies (built-in delays prevent blocking) - Downloaded videos increase storage costs ## 🔧 Troubleshooting ### Actor fails to connect Solution: Enable Apify Proxy with RESIDENTIAL group in input settings ### No videos found Possible causes: - Hashtag has no recent content - TikTok changed their HTML structure - CAPTCHA appeared (check logs) - Proxy issue (try different proxy group) ### Slow performance Solutions: - Set getDetailedData: false for faster scraping (less data) - Disable shouldDownloadVideos for metadata-only scraping - Reduce resultsPerPage number - Use RESIDENTIAL proxies (faster than datacenter) ### Video download fails Possible causes: - Video is private or deleted - Download API (tikwm.com) is temporarily unavailable - Network timeout - Video URL is invalid Solution: Check logs for specific error messages. Some videos may fail while others succeed. ### Missing data fields Cause: TikTok occasionally updates their data structure Solution: Check for actor updates or report issue ## 📝 Best Practices 1. Start Small - Test with 10-20 videos first 2. Use Residential Proxies - Better success rate and speed 3. Enable Detailed Data - For complete information (slightly slower) 4. Monitor Storage - Video downloads increase storage usage 5. Monitor Costs - Check usage in Apify Console 6. Respect Rate Limits - Don't scrape thousands of videos at once 7. Regular Updates - Keep actor updated for best performance 8. Test Downloads - Try with a few videos before bulk downloading ## 💰 Cost Considerations ### Without Video Download - Typical cost: $0.05-0.15 per 100 videos scraped - Storage: Minimal (only metadata, < 1 MB per 100 videos) ### With Video Download - Typical cost: $0.15-0.40 per 100 videos (includes download time) - Storage: 200-1000 MB per 100 videos (depends on video length) - Network: Higher bandwidth usage Tip: Start without downloads to test, then enable for selected runs ## 🔐 Legal & Ethics - ✅ Respect TikTok's Terms of Service - ✅ Use scraped data responsibly - ✅ Do not scrape private or sensitive content - ✅ Be mindful of creator rights and privacy - ✅ Respect video copyrights when downloading - ✅ Do not redistribute downloaded videos without permission - ✅ Follow data protection regulations (GDPR, etc.) - ✅ Rate limiting is built-in to prevent abuse Note: This tool is for research and analytics purposes. Downloaded videos should only be used for personal analysis, research, or with proper permissions. Users are responsible for how they use the scraped data and downloaded videos. ## 🆘 Support - Issues: Report bugs or request features in Apify Console - Documentation: Apify Documentation - Community: Apify Discord ## 🏆 Why Choose This Actor? - ⚡ Fast - Parallel processing for maximum speed - 📊 Comprehensive - Extracts 20+ data fields per video - 🎬 Complete Solution - Metadata + actual video downloads - 🔒 Reliable - Multiple extraction methods + error handling - 💾 Easy Access - Videos stored in Apify storage, ready to download - 💰 Affordable - Optimized to minimize compute costs - 🎯 Easy - Simple configuration, no coding required - 🔄 Maintained - Regular updates for TikTok changes - 📦 HD Quality - Downloads highest quality available ## 🔄 Version History ### Latest Update - ✨ Added video download functionality with HD quality support - 📥 Videos stored in Apify key-value store for easy access - 📊 Download information included in dataset output - 🎬 Automatic quality detection (HD/Standard) - ⚡ Optimized download process with error handling --- Ready to scrape TikTok? Start by adding your hashtags and enable the video download feature to get complete access to viral content! 🚀