Lidl Scraper

Overview

This Apify actor scrapes product data from Lidl.de, providing an alternative to Lidl's unavailable free API. It's designed for e-commerce data extraction, automation, and developer use cases.

Key Features

• Keyword Search: Scrape results for any search term.
• Category Scraping: Extract products from any category or sub-category page.
• Product Detail Extraction: Retrieve detailed information for individual products.
• Delivery Date Scraping: Get delivery dates for all scraped products.

How to Use

Configure the actor using a JSON input object. The scraper will visit the specified pages, extract data, and store the results in an Apify dataset for export.

Input Parameters

The input is a JSON object with the following fields:

• search: (Optional, String) A keyword to search on Lidl.de.
• startUrls: (Optional, Array) A list of Lidl.de URLs to scrape. Provide list pages (category/search) or direct product detail URLs.
• endPage: (Optional, Number) The final page number to scrape for each search or startUrl. Default is Infinite.
• maxItems: (Optional, Number) A limit for the total number of items to scrape.
• proxy: (Required, Proxy Object) Proxy configuration. Using a proxy is required. You can use Apify Proxy or your own proxies.

Tip: To scrape a specific list, paste its URL into startUrls. Set endPage to 1 to scrape only the first page of that list. You can also scrape page ranges by providing a start page URL and setting endPage accordingly.

Input Example

{
  "startUrls": [
    "https://www.lidl.de/c/sport-freizeit/s10067763",
    "https://www.lidl.de/p/parkside-performance-20-v-8-ah-smart-akku-paps-208-a1-8-ah/p100368990"
  ],
  "search": "trink",
  "endPage": 2,
  "maxItems": 1000,
  "proxy": {
    "useApifyProxy": true
  }
}

Performance Notes

The actor is optimized for speed. With minimal blocking, it can scrape approximately 100 product listings in 2 minutes, consuming ~0.03-0.05 compute units.

Output

During the run, the actor logs its progress, indicating which page is being processed and how many items have been loaded.

All scraped results are stored in the actor's associated dataset. Each product is a separate item in the dataset. You can then manage and export this data using the Apify API in various languages (Python, PHP, Node.js). See the Apify API documentation for details on working with datasets.

Development & Support

This scraper is under active development. For bug reports or feature requests, please create an issue on the project's GitHub repository.