Universal E-Commerce Scraper An intelligent, universal e-commerce scraper for Apify that automatically detects 6+ store systems, finds product data via sitemaps or deep crawling, and normalizes it into a clean, structured format. --- ## Core Features | Feature | Description | Status | | ----------------------- | -------------------------------------------------------------------------------------------------------- | ----------- | | Automatic Platform Detection | Detects Shopify, WooCommerce, Magento, PrestaShop, BigCommerce, Shopware, and generic systems. | ✅ Active | | Intelligent Product Discovery | 1. Sitemap Analysis: Finds product URLs via sitemap.xml (fast & efficient). | ✅ Active | | | 2. Deep Product Crawler: "Screaming Frog"-style crawling for intelligent product page identification. | ✅ Active | | Robust Data Extraction | Uses APIs (WooCommerce, Shopify), GraphQL, JSON-LD, and HTML parsing for maximum accuracy. | ✅ Active | | Data Normalization | Cleans and standardizes prices, currencies, availability, and URLs for consistent output. | ✅ Active | | Ethical Crawling | Optional respect for robots.txt, Crawl-Delay, and bot identification. | ✅ Active | | Proxy & User-Agent Management | Full Apify Proxy integration and User-Agent rotation to prevent blocking. | ✅ Active | | Anti-Marketplace Protection | A blacklist prevents crawling large marketplaces like Amazon, eBay, etc. | ✅ Active | | Flexible Output Formats | Exports data as JSON, CSV, or XLSX. | ✅ Active | --- ## 🚫 Marketplace Policy This scraper is designed exclusively for standalone online stores. It is NOT suitable for large marketplaces such as Amazon, eBay, Alibaba, Otto, Kaufland, Walmart, Etsy, etc. Attempts to crawl these sites will be actively blocked. --- ## 🚀 Quick Start on the Apify Platform 1. Click "Try actor". 2. Enter your start URLs (e.g., https://your-shopify-store.com/, https://your-magento-store.com/). 3. Configure the options (see "Input Parameters"). 4. Click "Start" to begin the crawl. You will find the results in the Dataset tab once the run is complete. --- ## ⚙️ Input Parameters For a full description of all input parameters, please refer to the Input Schema Tab. | Key Parameters | Description | | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | startUrls | The URLs of the stores you want to crawl. | | maxProducts | The maximum number of products to extract. Ideal for quick tests. | | enableDeepCrawler | (Warning: Can be expensive!) Activates the intelligent deep crawler if sitemap analysis fails. | | proxyConfiguration | Configure Apify Proxy or your own custom proxies. (Recommended) | | outputFormat | Choose the output format: json, csv, or xlsx. | | includeDescription, includeImages, includeSKU | Specify which optional fields should be included in the results. | | respectRobotsTxt | Ensures the crawler respects the website's robots.txt rules. | | rotateUserAgents | Rotates user agents for each request to avoid getting blocked. | | debugMode, saveHTML | Enable these options for detailed debugging. Saves raw HTML to the Key-Value Store. | --- ## 📊 Example Output (JSON) The extracted data is provided in a clean, standardized format. json [ { "name": "Stylish T-Shirt - Blue", "description": "A comfortable and stylish t-shirt made from 100% organic cotton. Perfect for everyday wear.", "price": 29.99, "currency": "USD", "sku": "TS-BL-M-01", "availability": "InStock", "image": "https://your-store.com/images/stylish-t-shirt-blue.jpg", "url": "https://your-store.com/products/stylish-t-shirt-blue", "platform": "shopify", "variants": [ { "name": "Color", "value": "Blue" }, { "name": "Size", "value": "M" } ] } ] Note: The attributes field (for detailed specifications) is primarily supported for WooCommerce and Magento stores.