Shopify Scraper A professional-grade Shopify store scraper that extracts comprehensive product data from any Shopify-based e-commerce site. Supports full store crawling, collection scraping, individual product extraction, and search functionality. ## Why Use This Scraper? - Market Research: Analyze competitor pricing, product descriptions, and variants - Trend Monitoring: Track new product launches and stock status changes - Data Aggregation: Build comprehensive catalogs from multiple Shopify stores - Marketing Insights: Understand how brands structure their product metadata and categories ## Key Features 🚀 Store-Wide Crawling — Automatically discovers and extracts all products from an entire store 🎯 Precision Targeting — Scrape specific collections or individual product URLs 🔍 Search Support — Search for products within a store using keywords 💱 Price Normalization — Prices stored as integers (cents) to avoid floating-point errors 📦 Comprehensive Data — Extracts titles, descriptions, variants, images, options, SKUs, barcodes, and stock status 🛡️ Anti-Bot Resilience — Built-in rate limiting, retry logic, and proxy rotation 🔄 Auto-Detection — Automatically verifies if a site is Shopify-powered ⚡ Fast & Reliable — Optimized extraction with automatic fallback mechanisms ## How It Works 1. Automatic Shopify Detection — The scraper automatically verifies each URL is a Shopify store 2. Smart URL Classification — Detects if URL is a store root, collection, product, or search page 3. Data Extraction — Extracts comprehensive product data with automatic pagination 4. Deduplication — Automatically removes duplicate products across collections ## Input Parameters | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | startUrls | Array | ✅ | - | List of Shopify URLs (store, collection, or product) | | maxProducts | Integer | No | 0 | Max products to scrape (0 = unlimited) | | searchQuery | String | No | - | Search term for finding specific products | | includeOutOfStock | Boolean | No | true | Include out-of-stock products | | currency | String | No | Auto | Currency code (USD, EUR, GBP, etc.) | | proxy | Object | No | Apify Proxy | Proxy configuration | | useHeadlessFallback | Boolean | No | true | Enable fallback extraction method | | requestTimeout | Integer | No | 30000 | Request timeout in milliseconds | | retryCount | Integer | No | 3 | Number of retry attempts | ## Input Examples ### Scrape Entire Store json { "startUrls": [ { "url": "https://www.allbirds.com" } ], "proxy": { "useApifyProxy": true } } ### Scrape Specific Collection json { "startUrls": [ { "url": "https://www.allbirds.com/collections/mens-shoes" } ], "maxProducts": 100 } ### Scrape Single Product json { "startUrls": [ { "url": "https://www.allbirds.com/products/mens-wool-runners" } ] } ### Search Within Store json { "startUrls": [ { "url": "https://www.allbirds.com" } ], "searchQuery": "wool runners", "maxProducts": 20 } ### Multiple URLs json { "startUrls": [ { "url": "https://www.allbirds.com/collections/mens" }, { "url": "https://www.allbirds.com/collections/womens" } ], "maxProducts": 50 } ## Output Schema Each product includes: json { "source": { "id": "7654321098765", "handle": "mens-wool-runners", "url": "https://www.allbirds.com/products/mens-wool-runners", "retailer": "www.allbirds.com", "scrapedAt": "2025-12-13T10:30:00Z" }, "title": "Men's Wool Runners", "description": "Our original wool shoe...", "descriptionHtml": "<p>Our original wool shoe...</p>", "vendor": "Allbirds", "productType": "Shoes", "tags": ["mens", "shoes", "wool"], "createdAt": "2025-01-15T00:00:00Z", "updatedAt": "2025-12-10T00:00:00Z", "publishedAt": "2025-01-15T08:00:00Z", "variants": [ { "id": "42345678901234", "title": "8 / Natural Grey", "sku": "WR-M-NG-8", "barcode": "1234567890123", "price": 11000, "compareAtPrice": null, "currency": "USD", "available": true, "inventoryQuantity": null, "requiresShipping": true, "weight": 0.5, "weightUnit": "kg", "option1": "8", "option2": "Natural Grey", "option3": null } ], "images": [ { "id": "12345678901234", "url": "https://cdn.shopify.com/s/files/...", "alt": "Men's Wool Runners", "width": 1200, "height": 1500, "position": 1 } ], "options": [ { "name": "Size", "position": 1, "values": ["7", "8", "9", "10", "11", "12"] }, { "name": "Color", "position": 2, "values": ["Natural Grey", "Black", "Navy"] } ] } ## Extracted Data Fields | Product Fields | Variant Fields | Media & Options | |---------------|----------------|-----------------| | Title | SKU | All Images | | Description (text & HTML) | Barcode | Image Dimensions | | Vendor/Brand | Price (in cents) | Alt Text | | Product Type | Compare-at Price | Options (Size, Color) | | Tags | Availability | Option Values | | Created/Updated Dates | Inventory Quantity | Positions | | URL & Handle | Weight & Unit | | | Retailer | Shipping Required | | ## URL Types Supported | URL Pattern | Type | Example | |-------------|------|---------| | domain.com | Full Store | https://www.allbirds.com | | /collections/{handle} | Collection | https://www.allbirds.com/collections/mens | | /products/{handle} | Single Product | https://www.allbirds.com/products/wool-runners | | /search?q={query} | Search Results | https://www.allbirds.com/search?q=wool | ## Troubleshooting | Issue | Possible Cause | Solution | |-------|---------------|----------| | 0 Results | Site is not Shopify-based | Check logs - scraper auto-detects and warns | | 403 / Access Denied | IP flagged | Enable useApifyProxy with residential proxies | | Incorrect Prices | Integer format | Divide by 100 (e.g., 2995 → $29.95) | | Missing Products | Rate limiting | Increase retryCount, use proxies | ## Important Notes - Price Format: All prices are integers in cents (e.g., 2995 = $29.95) - Variants: Each product contains all variants in the variants array - Deduplication: Products are automatically deduplicated by ID - Pagination: Handles pagination automatically across large catalogs ## Cost Estimation - Speed: Typically 500-2,000 products per minute - Compute Units: ~0.1-0.2 CUs per 1,000 products - Proxy: Residential proxies recommended for best results Actual costs vary based on store size and anti-bot measures.