Shopify Scraper (GraphQL)

An Apify actor that extracts product data from Shopify stores. It crawls a store's sitemap.xml to find product pages and then uses the Shopify Storefront GraphQL API to fetch detailed product information efficiently.

Overview

This actor is designed for speed and cost-effectiveness. It processes stores by batching GraphQL requests, supports incremental runs to avoid re-scraping, and can filter products based on their last modified date. Output is structured as one record per product, with all variants nested inside.

Key Features

• Sitemap-based Crawling: Discovers product URLs from a store's sitemap.xml (specifically /products/<handle> paths).
• Efficient GraphQL Queries: Batches multiple product requests into single GraphQL calls using aliases, reducing network overhead.
• Incremental Processing: Can skip products that have already been processed in previous runs.
• Date Filtering: Optionally ignores products not updated since a given date (updatedSince).
• Performance Tuning: Configurable batch size, concurrency, and buffered dataset writes.
• Extensible: Use extendScraperFunction for custom logic during the scraping lifecycle and extendOutputFunction to transform final records.

Input / Configuration

Core parameters required to run the actor:

• startUrls: Array containing your target sitemap.xml URL(s).
• storefrontApiVersion: The Shopify Storefront API version to use (e.g., 2024-07).
• storefrontAccessToken: Your store's Storefront API access token.

Essential performance and utility settings:

• maxRequestsPerCrawl, maxConcurrency, maxRequestRetries, proxyConfiguration: Standard Apify crawl controls.
• updatedSince: ISO date string; skips products with a <lastmod> older than this.
• batchSize: Number of product handles to query per GraphQL request (default: 10).
• perHostConcurrency: Parallel GraphQL requests allowed per store host (default: 2).
• bufferWrites & bufferSize: Controls buffering for dataset writes to improve performance.

How to Use

Running Locally

1. Install dependencies:
   bash npm install
2. Create an input file at apify_storage/key_value_stores/default/INPUT.json:
   json { "startUrls": [{ "url": "https://example.com/sitemap.xml" }], "storefrontApiVersion": "2024-07", "storefrontAccessToken": "<YOUR_TOKEN>", "maxRequestsPerCrawl": 50 }
3. Start the actor:
   bash npm start
   For development with auto-restart:
   bash npm run dev

Docker Quick Start

Using the provided Makefile:

make init  # Creates .env and INPUT.json from templates
make run   # Builds and starts the Docker container

Output datasets will be in apify_storage/datasets/default.

Output

The actor saves one item per product to the dataset. The product's variants are available within the additional.variants property of each record. The structure is based on the response from the Shopify Storefront GraphQL API.

Extensibility

You can inject custom logic at specific points:

• extendScraperFunction: Provides lifecycle hooks (SETUP, FILTER_SITEMAP_URL, PRENAVIGATION, POSTNAVIGATION, RUN, FINISHED) for custom actions.
• extendOutputFunction: Allows you to modify or filter the final product record before it is saved to the dataset.

Project Info

• License: Apache License 2.0 (see LICENSE and NOTICE files).
• CI/CD: GitHub Actions workflows (ci.yml, codeql.yml) handle testing, linting, and security analysis on pushes and pull requests.