DuckDuckGo Search Scraper A powerful web scraping actor built with Apify SDK and Crawlee that extracts search results from DuckDuckGo in multiple search modes. ## Features - Multiple Search Modes: Default search, image search, news search, and video search - Batch Processing: Process multiple search queries in a single run (comma-separated or array) - Auto Pagination: Automatically navigate through multiple pages to collect more results - SEO Data Extraction: Extract meta tags, structured data, and page insights - Social Media Data: Extract Open Graph and Twitter Card meta tags - Configurable Proxy: Optional Apify proxy integration for better scraping success - Robust Extraction: Intelligent title and snippet extraction with fallback mechanisms - Anti-Detection: Built-in measures to avoid being blocked - Screenshot Support: Optional debugging screenshots for troubleshooting - Clean Output: Results in a clean, structured array format ## Search Modes - Default: Standard web search results - Images: Image search results with thumbnails - News: News article search results - Videos: Video search results ## Input Parameters - searchQuery (required): The search term to search for (can be comma-separated for multiple queries) - multipleSearchQueries (optional): Array of search terms for batch processing - searchMode (required): Which search mode to use (default, images, news, videos) - maxResults: Maximum number of results to extract (default: 10) - maxPages: Maximum number of pages to scrape (default: 1, max: 10) - autoPagination: Automatically navigate to next pages (default: false) - timeout: Page load timeout in seconds (default: 60) - extractSEOData: Extract additional SEO information (default: false) - extractSocialMedia: Extract social media meta tags (default: false) - proxyConfiguration: Proxy configuration object ## Output Format The actor outputs a clean array of search results, each containing: json { "title": "Readable title of the result", "url": "URL of the result", "snippet": "Description or snippet text", "position": 1, "type": "default|images|news|videos", "page": 1 } ### Enhanced Output (when SEO features enabled) When extractSEOData or extractSocialMedia are enabled, additional fields are included: json { "title": "Readable title of the result", "url": "URL of the result", "snippet": "Description or snippet text", "position": 1, "type": "default|images|news|videos", "seoData": { "metaTitle": "Page title from meta tag", "metaDescription": "Meta description", "canonicalUrl": "Canonical URL", "structuredData": "JSON-LD structured data" }, "socialMedia": { "openGraph": { "title": "Open Graph title", "description": "Open Graph description", "image": "Open Graph image URL" }, "twitterCard": { "title": "Twitter Card title", "description": "Twitter Card description", "image": "Twitter Card image URL" } } } ## Legal This actor is provided for educational and research purposes only. Users are responsible for: - Complying with DuckDuckGo's Terms of Service - Respecting robots.txt and website terms - Following applicable laws and regulations - Using the tool responsibly and ethically The developers are not responsible for any misuse of this tool or violations of terms of service. ## Usage Examples ### Single Search Query json { "searchQuery": "artificial intelligence", "searchMode": "default", "maxResults": 10 } ### Multiple Search Queries (comma-separated) json { "searchQuery": "machine learning, deep learning, neural networks", "searchMode": "default", "maxResults": 5 } ### Multiple Search Queries (array) json { "multipleSearchQueries": ["SEO optimization", "content marketing", "link building"], "searchMode": "default", "maxResults": 5 } ### SEO Data Extraction json { "searchQuery": "web development", "searchMode": "default", "maxResults": 5, "extractSEOData": true, "extractSocialMedia": true } ### Image Search with SEO json { "searchQuery": "nature photography", "searchMode": "images", "maxResults": 10, "extractSEOData": true } ### Multi-Page Search with Pagination json { "searchQuery": "web development tutorials", "searchMode": "default", "maxResults": 50, "maxPages": 5, "autoPagination": true } ### Batch Search with Pagination json { "multipleSearchQueries": ["SEO tips", "content marketing", "link building"], "searchMode": "default", "maxResults": 30, "maxPages": 3, "autoPagination": true }