🎥 Video Link Crawler The Video Link Crawler is a powerful web crawler designed to extract 🎞️ video links from websites, starting from a specified URL, it navigates through web pages, discovers video content, and compiles the extracted links into a structured dataset. It supports customizable crawling settings and efficient video extraction. Whether you're building a video catalog or analyzing video trends, this tool is your go-to solution for efficient video discovery. --- ## 📝 Description The Video Link Crawler specializes in extracting video URLs from various HTML elements, including: - <video> and <source> tags - <embed> and <object> tags - <iframe> tags (e.g., YouTube, Vimeo) - <a> tags linking to video files (e.g., .mp4, .webm) With advanced features like proxy support, crawl depth control, and retry mechanisms, it adapts seamlessly to your web scraping needs. Extracted data is stored in an Apify Dataset, ready for analysis or integration into your workflows. --- ## ✨ Key Features - Customizable Crawling ⬩ Define starting URLs ⬩ Restrict crawling using glob patterns ⬩ Set maximum crawl depth and request limits - Video URL Extraction ⬩ Extract video links from multiple HTML tags ⬩ Convert relative URLs to absolute links for seamless results - Proxy Support ⬩ Ensure anonymity with Apify Proxy ⬩ Optionally specify proxy groups - Error Handling ⬩ Detailed logs for failed requests ⬩ Handles redirect loops and HTTP 403 errors gracefully - Data Output ⬩ Structured and easy-to-use dataset with video links --- ## ⚙️ How It Works - Initialization Reads input parameters such as startUrl, glob patterns, and crawl depth to configure the crawler. - Crawling Begins at the startUrl, follows links matching the glob patterns, and stops upon reaching the crawl limits. - Video Extraction Detects video links from supported tags and resolves relative URLs to absolute ones. - Data Storage Extracted URLs and video links are saved to an Apify Dataset. - Error Logging Logs all failed or skipped requests for review and troubleshooting. --- ## 🔧 Input Configuration The Video Link Crawler accepts the following input parameters in JSON format: | Field | Type | Required | Description | |----------------------|------------|----------|----------------------------------------------------------------------------------------------| | startUrl | string | Yes | The URL where the crawler begins. | | globs | string[] | Yes | Array of glob patterns to restrict crawling to specific domains or paths. | | maxRequestsPerCrawl| number | Optional | Maximum number of requests the crawler processes. | | maxCrawlDepth | number | Optional | Maximum depth of pages to crawl from the starting URL. | | proxyConfig | object | Optional | Proxy configuration for anonymized crawling (e.g., Apify Proxy). | ### 🛡️ Proxy Configuration - useApifyProxy (boolean): Enable Apify Proxy for anonymized crawling. - apifyProxyGroups (string[]): Specify proxy groups for more granular control (optional). --- ## 📝 Example Input json { "startUrl": "https://example.com", "globs": ["https://example.com/**"], "maxRequestsPerCrawl": 50, "maxCrawlDepth": 3, "proxyConfig": { "useApifyProxy": true, "apifyProxyGroups": ["SHARED"] } } --- ## 📄 Example Output The output of the Actor is saved to an Apify Dataset in the following structured format: json [ { "url": "https://example.com/page1", "videoLinks": [ "https://example.com/videos/video1.mp4", "https://youtube.com/watch?v=abc123" ] }, { "url": "https://example.com/page2", "videoLinks": [] } ] ## ℹ️ Need Help? If you have any questions, suggestions, or need assistance, feel free to: 💬 Open an issue in the Issues section. 📧 Contact us: infoweaver7@gmail.com