🎬 YouTube Shorts Scraper — Apify Actor 🔎 🎯 Purpose: A lightweight Apify Actor that scrapes YouTube Shorts metadata from one or more channel URLs. Designed to run in Apify using the Python SDK, it focuses on ⚙️ reliability and 🧭 proxy support (Apify proxy or custom proxy URLs). --- ## ✨ Key Features ✅ Extracts Shorts videoIds from channel pages (tries multiple channel endpoints). 📄 Fetches per-short details — title, description, duration, thumbnails, view counts, tags, captions URL, publish date, and channel info. 🌐 Built-in support for Apify Proxy (RESIDENTIAL/DATACENTER groups) via Actor.create_proxy_configuration(). 🔐 Accepts custom proxy strings (e.g., http://user:pass@host:port or host:port) and normalizes the scheme. 🔁 Retries with exponential backoff, configurable ⏱️ timeouts, and concurrency. 📤 Incremental Actor.push_data() for streaming results to the Apify dataset. 💾 Writes a local JSON file shorts_filled_{channel_or_id}.json with final results. --- ## ⚙️ Input (Actor Input JSON) Provide actor input using the Apify run calls 🧩 | 🏷️ Field | Type | Required | 💡 Default / Notes | | ----------------------- | ---------------------------------- | :------: | ------------------------------------------------------------------------------------------------------------------------------------------------ | | startUrls 🌐 | array<{url: string}> or string | ✅ | One or more channel URLs. Example: [{"url":"https://www.youtube.com/PenMovies/shorts"}] or ["https://www.youtube.com/PenMovies"] | | maxshorts 🔢 | integer | ✅ | Limit of Shorts per channel to collect IDs for further detail fetching. | | proxyConfiguration 🧠 | object | ✅ | Standard Apify proxy editor object. Example: { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] } or use a custom proxy (see below). | | concurrentWorkers ⚡ | integer | ✅ | Overrides default concurrency (CONCURRENT_WORKERS env var). | --- ### 🌍 Custom Proxy (in proxyConfiguration) * 🧩 If you provide a custom proxy string, use the customProxy / proxyUrl / proxy key in proxyConfiguration. The actor automatically adds http:// if you only provide host:port. * 🚀 If useApifyProxy is set to true, the actor will use Apify’s proxy configuration via proxy_conf.new_url(). --- Would you like me to also add emojis in the Output, Examples, and How it works sections (if you have them)? I can style the whole README consistently like a polished Apify Store project. ## Input Example : json { "startUrls": [ { "url": "https://www.youtube.com/@TseriesKannadaMusic/shorts" } ], "maxshorts": 30, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"], } } --- ## Output Example : json [ { "title": "🎬 Romeo S3 – July 18 → Prime Video + Pen Movies.", "type": "shorts", "id": "BJsbRhaXBPg", "video_url": "https://www.youtube.com/watch?v=BJsbRhaXBPg", "thumbnailUrl": "https://i.ytimg.com/vi/BJsbRhaXBPg/maxres2.jpg?sqp=-oaymwEoCIAKENAF8quKqQMcGADwAQH4AbYIgAKAD4oCDAgAEAEYZSBKKD8wDw==&rs=AOn4CLBl_qKaB5Hbo7MWFFCG6SCt7KM1-Q", "viewCount": "35.1k", "viewCountRaw": 35069, "uploadDate": "2025-07-17T06:20:47-07:00", "duration": 15, "liveBroadcastDetails": { "live_status": "not_live", "is_live": false }, "processingDetails": { "status": "processed", "playable_in_embed": true }, "description": "Romeo S3 – Streaming July 18\nOn Amazon Prime Video & Pen Movies YouTube!\nEnjoy Super Hit Movies Collection :- \nAcharya - https://youtu.be/MZGKv-UqD4c\nJaya Janaki Nayaka KHOONKHAR - https://youtu.be/1lFk1LSsI_c \nSHER KA SHIKAAR - https://youtu.be/FE435iFT6jw \nTholi Prema - https://youtu.be/J4nBcaYMuI4\nEntertainment- https://youtu.be/LXXkiUKDK4w \n\n\nFor your favorite Bollywood Movies, Subscribe now: \nhttp://bit.ly/2VXistY\n\nSUBSCRIBE for the best Bollywood clips, movies and scenes all in your PEN Multiplex channel \nhttp://bit.ly/2QxELp3\n\nFor your favorite Gujarati Movies, Subscribe now: http://bit.ly/2VZm9Q6\n\nFor Action-Packed Movies, Click on the below link now...\nPen Action: http://bit.ly/2XaSS6t\n\n\nEnjoy & stay connected with us! \nLike us on Facebook: https://www.facebook.com/PenMovies\nFollow us on Twitter: https://twitter.com/penmovies\nFollow us on Instagram: https://www.instagram.com/penmovies 👉 Log On To Our Official Website : https://penstudios.in/", "channelName": "Pen Movies", "channelUrl": "https://www.youtube.com/channel/UC3ar28GS6o1p0m_wabfk2zw", "channelId": "UC3ar28GS6o1p0m_wabfk2zw", "inputChannelUrl": "https://www.youtube.com/PenMovies/shorts", "hashtags": [], "fromYTUrl": "https://www.youtube.com/watch?v=BJsbRhaXBPg", "tags": [], "durationAlt": "15s", "transcriptSample": [] }, { "title": "No popcorn needed. Just WiFi. 🎬 Romeo S3 – July 18 → Prime Video + Pen Movies.", "type": "shorts", "id": "s9Pr4rydv_8", "video_url": "https://www.youtube.com/watch?v=s9Pr4rydv_8", "thumbnailUrl": "https://i.ytimg.com/vi/s9Pr4rydv_8/maxres2.jpg?sqp=-oaymwEoCIAKENAF8quKqQMcGADwAQH4AbYIgAKAD4oCDAgAEAEYciBWKEAwDw==&rs=AOn4CLCj8NKhwTuqOhxvoXMYqlEcYAlLag", "viewCount": "16.6k", "viewCountRaw": 16641, "uploadDate": "2025-07-13T07:30:39-07:00", "duration": 20, "liveBroadcastDetails": { "live_status": "not_live", "is_live": false }, "processingDetails": { "status": "processed", "playable_in_embed": true }, "description": "\"No popcorn needed. Just WiFi.\n 🎬 Romeo S3 – July 18 → Prime Video + Pen Movies.\"\nEnjoy Super Hit Movies Collection :- \nAcharya - https://youtu.be/MZGKv-UqD4c\nJaya Janaki Nayaka KHOONKHAR - https://youtu.be/1lFk1LSsI_c \nSHER KA SHIKAAR - https://youtu.be/FE435iFT6jw \nTholi Prema - https://youtu.be/J4nBcaYMuI4\nEntertainment- https://youtu.be/LXXkiUKDK4w \n\n\nFor your favorite Bollywood Movies, Subscribe now: \nhttp://bit.ly/2VXistY\n\nSUBSCRIBE for the best Bollywood clips, movies and scenes all in your PEN Multiplex channel \nhttp://bit.ly/2QxELp3\n\nFor your favorite Gujarati Movies, Subscribe now: http://bit.ly/2VZm9Q6\n\nFor Action-Packed Movies, Click on the below link now...\nPen Action: http://bit.ly/2XaSS6t\n\n\nEnjoy & stay connected with us! \nLike us on Facebook: https://www.facebook.com/PenMovies\nFollow us on Twitter: https://twitter.com/penmovies\nFollow us on Instagram: https://www.instagram.com/penmovies 👉 Log On To Our Official Website : https://penstudios.in/", "channelName": "Pen Movies", "channelUrl": "https://www.youtube.com/channel/UC3ar28GS6o1p0m_wabfk2zw", "channelId": "UC3ar28GS6o1p0m_wabfk2zw", "inputChannelUrl": "https://www.youtube.com/PenMovies/shorts", "hashtags": [], "fromYTUrl": "https://www.youtube.com/watch?v=s9Pr4rydv_8", "tags": [], "durationAlt": "20s", "transcriptSample": [] }, { "title": "Why \"Retro\" Matters? | Suriya | Pooja Hegde | Karthik Subbaraj | Pen Studios | 1st May 25", "type": "shorts", "id": "wR4C5mCLxSM", "video_url": "https://www.youtube.com/watch?v=wR4C5mCLxSM", "thumbnailUrl": "https://i.ytimg.com/vi/wR4C5mCLxSM/maxres2.jpg?sqp=-oaymwEoCIAKENAF8quKqQMcGADwAQH4AbYIgAKAD4oCDAgAEAEYZSBVKEwwDw==&rs=AOn4CLDWPuQjNpaeeQk04mKXxSNtGgxrtQ", "viewCount": "19.2k", "viewCountRaw": 19178, "uploadDate": "2025-04-30T05:08:39-07:00", "duration": 51, "liveBroadcastDetails": { "live_status": "not_live", "is_live": false }, "processingDetails": { "status": "processed", "playable_in_embed": true }, "description": "With Suriya, it’s always magic — whether on the big screen or on satellite. The audience just can’t get enough,” says Dr. Jayantilal Gada.\nBook your ticket here - https://linktr.ee/RetroTheMovie\n\nEnjoy Super Hit Movies Collection :- \nAcharya - https://youtu.be/MZGKv-UqD4c\nJaya Janaki Nayaka KHOONKHAR - https://youtu.be/1lFk1LSsI_c \nSHER KA SHIKAAR - https://youtu.be/FE435iFT6jw \nTholi Prema - https://youtu.be/J4nBcaYMuI4\nEntertainment- https://youtu.be/LXXkiUKDK4w \n\n\nFor your favorite Bollywood Movies, Subscribe now: \nhttp://bit.ly/2VXistY\n\nSUBSCRIBE for the best Bollywood clips, movies and scenes all in your PEN Multiplex channel \nhttp://bit.ly/2QxELp3\n\nFor your favorite Gujarati Movies, Subscribe now: http://bit.ly/2VZm9Q6\n\nFor Action-Packed Movies, Click on the below link now...\nPen Action: http://bit.ly/2XaSS6t\n\n\nEnjoy & stay connected with us! \nLike us on Facebook: https://www.facebook.com/PenMovies\nFollow us on Twitter: https://twitter.com/penmovies\nFollow us on Instagram: https://www.instagram.com/penmovies 👉 Log On To Our Official Website : https://penstudios.in/", "channelName": "Pen Movies", "channelUrl": "https://www.youtube.com/channel/UC3ar28GS6o1p0m_wabfk2zw", "channelId": "UC3ar28GS6o1p0m_wabfk2zw", "inputChannelUrl": "https://www.youtube.com/PenMovies/shorts", "hashtags": [], "fromYTUrl": "https://www.youtube.com/watch?v=wR4C5mCLxSM", "tags": [ "hindi songs 2025", "hindi songs new", "bollywood songs 2025", "bollywood movies 2025", "tseries", "tseries songs", "retro trailer hindi", "retro hindi trailer", "retro suriya", "suriya retro trailer hindi", "retro suriya movie", "suriya 44", "retro suriya 44", "retro trailer suriya", "retro movie trailer hindi", "suriya retro movie", "suriya 44 movie", "retro suriya 44 trailer", "suriya movie trailer", "retro movie trailer suriya", "retro movie pooja hegde", "suriya and pooja hegde movie", "suriya 44 trailer", "retro" ], "durationAlt": "51s", "transcriptSample": [] }, { "title": "The Story Behind Movie \"Retro\" | Karthik Subbaraj | Suriya | Pooja Hegde | Pen Studios | 1st May 25", "type": "shorts", "id": "psKewJxFXgc", "video_url": "https://www.youtube.com/watch?v=psKewJxFXgc", "thumbnailUrl": "https://i.ytimg.com/vi/psKewJxFXgc/maxres2.jpg?sqp=-oaymwEoCIAKENAF8quKqQMcGADwAQH4AbYIgAKAD4oCDAgAEAEYZSBdKEcwDw==&rs=AOn4CLA3JXOHia2mN6b427Bev_jyG0AaFg", "viewCount": "27.2k", "viewCountRaw": 27210, "uploadDate": "2025-04-30T05:08:39-07:00", "duration": 35, "liveBroadcastDetails": { "live_status": "not_live", "is_live": false }, "processingDetails": { "status": "processed", "playable_in_embed": true }, "description": "Watch Karthik Subbaraj talk about what makes this film truly special.\nBook your ticket here - https://linktr.ee/RetroTheMovie\n\nEnjoy Super Hit Movies Collection :- \nAcharya - https://youtu.be/MZGKv-UqD4c\nJaya Janaki Nayaka KHOONKHAR - https://youtu.be/1lFk1LSsI_c \nSHER KA SHIKAAR - https://youtu.be/FE435iFT6jw \nTholi Prema - https://youtu.be/J4nBcaYMuI4\nEntertainment- https://youtu.be/LXXkiUKDK4w \n\n\nFor your favorite Bollywood Movies, Subscribe now: \nhttp://bit.ly/2VXistY\n\nSUBSCRIBE for the best Bollywood clips, movies and scenes all in your PEN Multiplex channel \nhttp://bit.ly/2QxELp3\n\nFor your favorite Gujarati Movies, Subscribe now: http://bit.ly/2VZm9Q6\n\nFor Action-Packed Movies, Click on the below link now...\nPen Action: http://bit.ly/2XaSS6t\n\n\nEnjoy & stay connected with us! \nLike us on Facebook: https://www.facebook.com/PenMovies\nFollow us on Twitter: https://twitter.com/penmovies\nFollow us on Instagram: https://www.instagram.com/penmovies 👉 Log On To Our Official Website : https://penstudios.in/", "channelName": "Pen Movies", "channelUrl": "https://www.youtube.com/channel/UC3ar28GS6o1p0m_wabfk2zw", "channelId": "UC3ar28GS6o1p0m_wabfk2zw", "inputChannelUrl": "https://www.youtube.com/PenMovies/shorts", "hashtags": [], "fromYTUrl": "https://www.youtube.com/watch?v=psKewJxFXgc", "tags": [], "durationAlt": "35s", "transcriptSample": [] }, { "title": "The Magic of \"Retro\" | Suriya | Pooja Hegde | Karthik Subbaraj | Pen Studios | 1st May 25", "type": "shorts", "id": "1cNBR6iW1-A", "video_url": "https://www.youtube.com/watch?v=1cNBR6iW1-A", "thumbnailUrl": "https://i.ytimg.com/vi/1cNBR6iW1-A/maxresdefault.jpg?sqp=-oaymwEmCIAKENAF8quKqQMa8AEB-AG2CIACgA-KAgwIABABGHIgSCgzMA8=&rs=AOn4CLBJ76T8vZ2He3QBvKkhFPTA6a8mvg", "viewCount": "30.8k", "viewCountRaw": 30839, "uploadDate": "2025-04-30T05:08:39-07:00", "duration": 107, "liveBroadcastDetails": { "live_status": "not_live", "is_live": false }, "processingDetails": { "status": "processed", "playable_in_embed": true }, "description": "🎬 Suriya spills the beans on what it was like shooting for this time-travelling ride!\nBook your ticket here - https://linktr.ee/RetroTheMovie\n\nEnjoy Super Hit Movies Collection :- \nAcharya - https://youtu.be/MZGKv-UqD4c\nJaya Janaki Nayaka KHOONKHAR - https://youtu.be/1lFk1LSsI_c \nSHER KA SHIKAAR - https://youtu.be/FE435iFT6jw \nTholi Prema - https://youtu.be/J4nBcaYMuI4\nEntertainment- https://youtu.be/LXXkiUKDK4w \n\n\nFor your favorite Bollywood Movies, Subscribe now: \nhttp://bit.ly/2VXistY\n\nSUBSCRIBE for the best Bollywood clips, movies and scenes all in your PEN Multiplex channel \nhttp://bit.ly/2QxELp3\n\nFor your favorite Gujarati Movies, Subscribe now: http://bit.ly/2VZm9Q6\n\nFor Action-Packed Movies, Click on the below link now...\nPen Action: http://bit.ly/2XaSS6t\n\n\nEnjoy & stay connected with us! \nLike us on Facebook: https://www.facebook.com/PenMovies\nFollow us on Twitter: https://twitter.com/penmovies\nFollow us on Instagram: https://www.instagram.com/penmovies 👉 Log On To Our Official Website : https://penstudios.in/", "channelName": "Pen Movies", "channelUrl": "https://www.youtube.com/channel/UC3ar28GS6o1p0m_wabfk2zw", "channelId": "UC3ar28GS6o1p0m_wabfk2zw", "inputChannelUrl": "https://www.youtube.com/PenMovies/shorts", "hashtags": [], "fromYTUrl": "https://www.youtube.com/watch?v=1cNBR6iW1-A", "tags": [ "hindi songs 2025", "hindi songs new", "bollywood songs 2025", "bollywood movies 2025", "tseries", "tseries songs", "retro trailer hindi", "retro hindi trailer", "retro suriya", "suriya retro trailer hindi", "retro suriya movie", "suriya 44", "retro suriya 44", "retro trailer suriya", "retro movie trailer hindi", "suriya retro movie", "suriya 44 movie", "retro suriya 44 trailer", "suriya movie trailer", "retro movie trailer suriya", "retro movie pooja hegde", "suriya and pooja hegde movie", "suriya 44 trailer", "retro" ], "durationAlt": "107s", "transcriptSample": [] }, ] ## Field reference Video metadata | Field | Type | Description | | -------------- | -------------- | ----------------------------------------------------- | | title | string | Video title | | video_id | string | YouTube video ID | | uploader | string | null | Uploader name | | upload_date | string | null | Original upload date (YYYYMMDD) | | publish_date | string | null | ISO date derived from upload/timestamp (YYYY-MM-DD) | | duration | integer | null | Duration in seconds | | duration_str | string | null | Human-readable duration (H:MM:SS or M:SS) | | description | string | null | Video description | | categories | array | null | Video categories | | tags | array | null | Video tags | | thumbnail | string | null | Thumbnail URL | | url | string | Video URL | | subtitles | object | null | Subtitles map from yt-dlp | Engagement (raw + humanized) | Field | Type | Description | | ------------------- | -------------- | ---------------------------- | | view_count | integer | null | View count (raw) | | view_count_str | string | null | View count (e.g., 1.4M) | | like_count | integer | null | Like count (raw) | | like_count_str | string | null | Like count (e.g., 320K) | | comment_count | integer | null | Comment count (raw) | | comment_count_str | string | null | Comment count (e.g., 5.6K) | | dislike_count | integer | null | May be unavailable | Channel enrichment | Field | Type | Description | | ------------------------------ | -------------- | ------------------------------------------------- | | channel_name | string | null | Channel display name | | channel_url | string | null | Channel URL | | channel_id | string | null | Channel ID | | channel_subscriber_count | integer| null | Subscribers (raw, if exposed) | | channel_subscriber_count_str | string | null | Subscribers (e.g., 2.3M) | | about | string | null | Channel “About” text | | tiktok | string | null | First TikTok link found on the channel profile | | instagram | string | null | First Instagram link found on the channel profile | | x | string | null | First X/Twitter link found on the channel profile | --- ## Troubleshooting checklist (fast) * ✅ Did you pick the right proxy group? (RESIDENTIAL often works best for YouTube scraping.) * ✅ Check your Apify proxy credits / permissions. * ✅ If using custom proxy, ensure it accepts HTTPS and supports SNI. * ✅ Check actor logs for failed to fetch entries — these contain the exception message showing whether it was timeout, TLS, or HTTP error. * ✅ If you get few or zero videoIds, try enabling Playwright-based scraping (outside scope of this actor) or increase maxshorts and retries. ---