📺 YouTube Shorts Scraper YouTube Shorts Scraper is an Apify Actor that scrapes video data and comments from a YouTube channel. It extracts key details such as video title, URL, view count, streaming time (in days), hashtags, description, comments count, likes count, and individual comments with user IDs. ## 🚀 Features - ✅ Scrapes video details: title, URL, views (converted to a number), streaming time (in days), hashtags, and description. - ✅ Extracts engagement metrics: Comments count, likes count, and individual comments (with user IDs and comment text). - ✅ Advanced anti-blocking: Intelligent detection and handling of YouTube consent pages and blocking mechanisms. - ✅ Robust data extraction: Dual extraction strategy - from individual video pages and grid view with automatic fallback. - ✅ Smart parsing: Converts view counts (e.g., "11K views" → 11000) and time descriptions (e.g., "2 weeks ago" → 14 days). - ✅ Configurable scrolling: Loads additional videos using infinite scrolling with a maximum number of scroll attempts. - ✅ Limits processing: Limits the number of videos processed per channel using configurable parameters. - ✅ Uses Puppeteer Stealth and Proxy: Supports proxy configurations for bypassing bot detection if needed. - ✅ Stores data in Apify Dataset: Outputs structured JSON data for easy export and further processing. - ✅ Make.com Integration: Real-time webhook delivery with batch processing and rate limiting. - ✅ Revenue Optimized: Default 100 videos per run for maximum results and ROI ($20/1000 results). ## 📥 Input Parameters The Actor accepts the following input parameters: | Parameter | Type | Description | Default Value | |--------------|-----------|-----------------------------------------------------------------------------------------------|-----------------------------------------| | channel | string | A YouTube channel URL (supports both /videos and /shorts endpoints). | "https://www.youtube.com/@Dior" | | max_videos | integer | The maximum number of videos to process from the channel (revenue optimized). | 100 | | webhook_url | string | Optional Make.com webhook URL for real-time data delivery. | - | | batch_size | integer | Number of videos to process in each batch (helps with rate limiting). | 25 | | delay_between_batches | integer | Delay between batches in milliseconds (helps with Make.com rate limits). | 2000 | ### Example Input json { "channel": "https://www.youtube.com/@CHANEL/shorts", "max_videos": 100, "webhook_url": "https://hook.eu1.make.com/YOUR_WEBHOOK_ID", "batch_size": 25, "delay_between_batches": 2000 } ## 📤 Output The Actor outputs a JSON dataset containing video details for each processed video. A sample output object is shown below: json { "id": "lQnhM3rW4GU", "title": "Dior Spring-Summer 2025 Haute Couture Show", "url": "https://www.youtube.com/watch?v=lQnhM3rW4GU", "views": 1600000, "time_text": 21, "hashtags": "", "description": "Spring-Summer 2025 Haute Couture by Maria Grazia Chiuri is an exploration of imaginative escapism, emanating a youthful energy influenced by Yves Saint Laurent's debut 'Trapèze' collection for the House in 1958. The A-line silhouette liberates the body as sweeping and ruffled fabrics represent metamorphosis...", "comments_count": 1376, "likes_count": 15420, "comments": [ { "user_id": "AA-ke5cu", "user_href": "https://www.youtube.com/@AA-ke5cu", "text": "My hats off to the floor engineer; this is not easy to do..." }, { "user_id": "anotherUser", "user_href": "https://www.youtube.com/@anotherUser", "text": "Amazing work by DIOR!" } ], "batch_id": "batch_1", "processed_at": "2025-01-20T10:30:00.000Z", "webhook_sent": true } ## 🔧 Technical Features ### Advanced Anti-Blocking System - Consent page detection: Automatically detects and handles YouTube consent popups - Content validation: Identifies blocked or empty pages with intelligent fallback - HTTP headers optimization: Mimics real browser behavior to avoid detection - Retry mechanisms: Implements smart retry logic for failed page loads ### Robust Data Extraction - Dual extraction strategy: - Primary: Extract from individual video pages - Fallback: Use data from channel grid view - Smart parsing functions: - Views: "11K views" → 11000, "1.2M views" → 1200000 - Time: "2 weeks ago" → 14, "1 month ago" → 30 - Error handling: Graceful fallback for failed extractions ### YouTube Shorts Optimization - Grid-based extraction: Leverages Shorts grid view for reliable metadata - Adaptive timeouts: Different timeout strategies for Shorts vs. regular videos - Content detection: Identifies and handles Shorts-specific page structures ## 🔗 Make.com Integration This Actor is specifically optimized for Make.com (formerly Integromat) integration, providing real-time data delivery and efficient workflow automation. ### 🚀 Make.com Features - Real-time webhook delivery - Send results directly to Make.com as they're processed - Batch processing - Handle large volumes with configurable batch sizes - Rate limiting control - Adjustable delays to respect Make.com and YouTube rate limits - Enhanced output format - Includes batch IDs, timestamps, and webhook status ### 📋 Setup Instructions 1. Create webhook in Make.com - Add a Webhook module to your scenario 2. Copy webhook URL - Use the generated webhook URL in your Actor input 3. Configure batch settings - Adjust batch size and delays as needed 4. Handle incoming data - Process webhook data in real-time workflows ### 🔄 Workflow Examples - Content Monitoring → YouTube Scraper → Webhook → Slack Notifications - Data Analysis → YouTube Scraper → Webhook → AI Analysis → Database - Content Curation → YouTube Scraper → Webhook → Content Filter → Social Media - Engagement Tracking → YouTube Scraper → Webhook → Analytics Dashboard ### 📊 Enhanced Output Fields Each result now includes Make.com specific data: json { "batch_id": "batch_1", "processed_at": "2025-01-20T10:30:00.000Z", "webhook_sent": true } For detailed Make.com integration instructions, advanced filtering, and sorting strategies, see MAKE_INTEGRATION.md. ## ⚙️ How It Works ### Channel Page Processing: The Actor navigates to the provided YouTube channel URL (supports both /videos and /shorts endpoints) and scrolls down to load more videos, based on the max_videos parameter. ### Video Processing: It extracts unique video URLs and processes up to max_videos videos. For each video, it opens a new page, waits for the metadata and comments to load, and then scrapes the required details. ### Data Transformation: The Actor converts view counts (e.g., "1.6M views" → 1600000) and converts time descriptions (e.g., "Streamed 3 weeks ago" → 21 days) using robust parsing functions. ### Result Storage: All scraped data is pushed to an Apify Dataset for easy access and export. ## 📈 Recent Improvements (v1.0.41+) ### Enhanced Data Extraction - ✅ Likes count extraction - Now captures video likes from YouTube Shorts - ✅ Robust view parsing - Handles "11K views", "1.2M views", "344K views" formats - ✅ Time parsing improvements - Better handling of various time formats - ✅ Grid-based fallback - Uses channel grid data when individual page extraction fails ### Anti-Blocking Enhancements - ✅ Consent page handling - Automatically accepts YouTube consent popups - ✅ Content validation - Detects blocked pages and implements fallback strategies - ✅ HTTP headers optimization - Enhanced browser fingerprinting - ✅ Retry mechanisms - Intelligent retry logic for failed requests ### Performance Optimizations - ✅ Adaptive timeouts - Different strategies for Shorts vs. regular videos - ✅ Resource management - Better page cleanup and memory management - ✅ Error recovery - Graceful handling of extraction failures --- ### ⚠️ Legal Disclaimer This project is intended for educational and research purposes only. The use of this Actor must comply with YouTube's Terms of Service and robots.txt policies. You are responsible for ensuring your use case does not violate YouTube website terms. YouTube's content and trademarks are the property of YouTube, Inc. Avoid aggressive scraping that could negatively impact YouTube's infrastructure. If you intend to use this Actor for commercial purposes, consider reaching out to the YouTube API team for official data access. ## Related Resources - YouTube Channel Scraper - Vimeo Video Scraper