dm Scraper ## dm scraper Since dm doesn't provide a good and free API, this actor should help you to retrieve data from it. The dm data scraper supports the following features: - Search Result Scraping - Collect product information directly from search results using either keywords and/or URLs. - Category Scraping - Extract product data from categories and/or subcategories of your choice from URLs. - Product Detail Scraping - Retrieve comprehensive and detailed information for each individual product. - Location-Based Availability - When a ZIP code is provided, the scraper automatically identifies the nearest store and updates product availability information accordingly. - The scraper automatically detects the current page of a given category or search URL and continues scraping from that point. ## Limitations - For each input URL (whether a category or search result), the scraper can only scrape a maximum of 1000 products. - This limit is applied per request, regardless of the offset or start value provided. For example, even if the crawl begins from offset 5 or 10, the scraper will still only return up to 1000 items total for that URL. - If the category or search contains more than 1000 products, the scraper will not retrieve the products beyond the 1000th item. - There is no limitation on the overall result. If you provide multiple URLs and/or search keyword(s), the scraper will return 1000 products per URL and/or search keyword(s). ## Bugs, fixes, updates, and changelog This scraper is under active development. If you have any feature requests you can create an issue from here. ## Input Parameters The input of this scraper should be JSON containing the list of pages on dm that should be visited. Required fields are: - zipcode:(Optional) (String) Zipcode value that enforces website to update the availability of the product. - startUrls: (Optional) (Array) List of dm URLs. You should only provide a search, category, or product URLs. - search: (Optional) (Array) Keyword(s) that you want to search on dm. - endPage: (Optional) (Number) Final number of page that you want to scrape. The default is Infinite. This applies to all search requests and startUrls individually. - maxItems: (Optional) (Number) You can limit the number of scraped items. This should be useful when you search through the big lists or search results. - customMapFunction: (Optional) (String) Function that takes each object's handle as an argument and returns the object with executing the function. - proxy: (Required) (Proxy Object) Proxy configuration. This solution requires the use of Proxy servers, either your own proxy servers or you can use Apify Proxy. ### Tip When you want to scrape over a specific list URL, just copy and paste the link as one of the startUrl. If you would like to scrape only the first page of a list then put the link for the page and have the endPage as 1. With the last approach that is explained above you can also fetch any interval of pages. If you provide the 5th page of a list and define the endPage parameter as 6 then you'll have the 5th and 6th pages only. ### Compute Unit Consumption The actor is optimized to run blazing fast and scrape as many items as possible. Therefore, it forefronts all the detailed requests. If the actor doesn't block very often it'll scrape 100 listings in 2 minutes with ~0.03-0.05 compute units. ### dm Scraper Input example json { "startUrls": [ "https://www.dm.de/balea-tuchmaske-big-dreams-p4067796168532.html", "https://www.dm.de/pflege-und-parfum", "https://www.dm.de/haare/haar-accessoires", "https://www.dm.de/pflege-und-parfum?allCategories.id0=020000&pageSize0=10&sort0=editorial_relevance&currentPage0=2", "https://www.dm.de/search?query=tasche&searchProviderType=dm-products", "https://www.dm.de/search?query=tasche&searchProviderType=dm-products&currentPage=1&loadPrev=false" ], "keywords": [ "tasche", "handcreme", "haarspray" ], "maxItems": 50, "endPage": 5, "zipcode": "80335", "proxy": { "useApifyProxy": true } } ## During the Run During the run, the actor will output messages letting you know what is going on. Each message always contains a short label specifying which page from the provided list is currently specified. When items are loaded from the page, you should see a message about this event with a loaded item count and total item count for each page. If you provide incorrect input to the actor, it will immediately stop with a failure state and output an explanation of what is wrong. ## dm Export During the run, the actor stores results into a dataset. Each item is a separate item in the dataset. You can manage the results in any language (Python, PHP, Node JS/NPM). See the FAQ or our API reference to learn more about getting results from this dm actor. ## Scraped dm Properties The structure of each item in dm looks like this: ### Item Detail json { "dan": 3064116, "gtin": 4067796168532, "a11yLabel": "Produktdetailseite: Balea, Tuchmaske Big Dreams, 1 St. Preis: 0,95 €", "breadcrumbs": [ "Pflege & Parfum", "Gesichtspflege", "Gesichtsmaske" ], "title": { "brandImage": "https://products.dm-static.com/images/f_auto,q_auto,c_fit,w_280,h_180/v1750686892/assets/mp/brands/subbrand_6169_default/subbrand_6169_default.png", "headline": "Tuchmaske Big Dreams, 1 St", "brand": "Balea" }, "brand": { "name": "Balea", "image": { "src": "https://products.dm-static.com/images/f_auto,q_auto,c_fit,w_280,h_180/v1750686892/assets/mp/brands/subbrand_6169_default/subbrand_6169_default.png", "alt": "Balea" } }, "rating": { "ratingCount": 7, "ratingValue": 4.4286 }, "pills": [ "Vegan" ], "images": [ { "zoomSrc": "https://products.dm-static.com/images/f_auto,q_auto,c_fit,h_1200,w_1200/v1753139673/assets/pas/images/ab4bb7c2-23a0-47e0-8b7b-59736de0bd26/balea-tuchmaske-big-dreams", "backgroundColor": "#F9F1F3", "src": "https://products.dm-static.com/images/f_auto,q_auto,c_fit,h_440,w_500/v1753139673/assets/pas/images/ab4bb7c2-23a0-47e0-8b7b-59736de0bd26/balea-tuchmaske-big-dreams", "thumbnailSrc": "https://products.dm-static.com/images/f_auto,q_auto,c_fit,h_116,w_116/v1753139673/assets/pas/images/ab4bb7c2-23a0-47e0-8b7b-59736de0bd26/balea-tuchmaske-big-dreams", "alt": "Tuchmaske Big Dreams Balea" }, { "zoomSrc": "https://products.dm-static.com/images/f_auto,q_auto,c_fit,h_1200,w_1200/v1753139680/assets/pas/images/c893bb4b-acc2-4efc-8565-8402189bb0de/balea-tuchmaske-big-dreams", "backgroundColor": "#F9F1F6", "src": "https://products.dm-static.com/images/f_auto,q_auto,c_fit,h_440,w_500/v1753139680/assets/pas/images/c893bb4b-acc2-4efc-8565-8402189bb0de/balea-tuchmaske-big-dreams", "thumbnailSrc": "https://products.dm-static.com/images/f_auto,q_auto,c_fit,h_116,w_116/v1753139680/assets/pas/images/c893bb4b-acc2-4efc-8565-8402189bb0de/balea-tuchmaske-big-dreams", "alt": "Tuchmaske Big Dreams Balea" }, { "zoomSrc": "https://products.dm-static.com/images/f_auto,q_auto,c_fit,h_1200,w_1200/v1753139687/assets/pas/images/844fc276-60fc-4dbb-9a1e-2e653c3a2fa8/balea-tuchmaske-big-dreams", "backgroundColor": "#F9F1F5", "src": "https://products.dm-static.com/images/f_auto,q_auto,c_fit,h_440,w_500/v1753139687/assets/pas/images/844fc276-60fc-4dbb-9a1e-2e653c3a2fa8/balea-tuchmaske-big-dreams", "thumbnailSrc": "https://products.dm-static.com/images/f_auto,q_auto,c_fit,h_116,w_116/v1753139687/assets/pas/images/844fc276-60fc-4dbb-9a1e-2e653c3a2fa8/balea-tuchmaske-big-dreams", "alt": "Tuchmaske Big Dreams Balea" } ], "eyecatchers": [ { "src": "https://media.dm-static.com/images/f_auto,q_auto,c_fit,w_150/v1748357417/products/pim/eyecatcher_new_de/new_de", "alt": "Neu im Sortiment" }, { "src": "https://media.dm-static.com/images/f_auto,q_auto,c_fit,w_150/v1748357418/products/pim/eyecatcher_dmbrand_de/dm_brand_stoerer_old", "alt": "Unsere Marke nur bei uns" } ], "seals": [ { "description": "Die Veganblume garantiert, dass weder tierische Inhaltsstoffe enthalten sind, noch Tierversuche durchgeführt werden.", "image": { "src": "https://media.dm-static.com/images/v1538636894/products/pim/v6210_vegan_society_vegan_logo1/v6210_vegan_society_vegan_logo.svg", "alt": "Veganblume" }, "src": "https://media.dm-static.com/images/v1538636894/products/pim/v6210_vegan_society_vegan_logo1/v6210_vegan_society_vegan_logo.svg", "title": "Veganblume" }, { "image": { "src": "https://media.dm-static.com/images/f_auto,q_auto,c_fit,h_ih,w_480/v1754316243/products/pim/stoererrom/dm_ohne_mikroplastik-stoerer_rgb_300px.png", "alt": "Rezeptur ohne Mikroplastik Siegel" }, "src": "https://media.dm-static.com/images/f_auto,q_auto,c_fit,h_ih,w_480/v1754316243/products/pim/stoererrom/dm_ohne_mikroplastik-stoerer_rgb_300px.png" } ], "descriptionGroups": [ { "header": "Produktbeschreibung", "contentBlock": [ { "bulletpoints": [ "Gesichtsmaske", "Spendet intensiv Feuchtigkeit", "Reduziert Spannungsgefühle und Trockenheitsfältchen", "Mit Hibiskus-Extrakt, Hyaluron und Aloe Vera", "Geeignet für jeden Hauttyp" ] }, { "texts": [ "Die Balea Tuchmaske Big Dreams verwöhnt die Haut mit einer erfrischenden Pflegeformel. Angereichert mit Hibiskus-Extrakt, Hyaluron und Aloe Vera spendet sie intensive Feuchtigkeit und sorgt für ein geschmeidiges Hautgefühl. Spannungsgefühle und Trockenheitsfältchen können gemildert werden, während der Teint frischer und strahlender wirkt. Die Maske ist für alle Hauttypen geeignet und bietet eine Auszeit für die Haut." ] }, { "texts": [ "dm-Artikelnummer: 3064116", "GTIN: 4067796168532" ] } ], "isExpanded": true, "showInInfoHintLayer": false }, { "header": "Produktmerkmale", "contentBlock": [ { "descriptionList": [ { "description": "Vegan", "title": "Produkteigenschaften" } ] } ], "isExpanded": false, "showInInfoHintLayer": false }, { "header": "Inhaltsstoffe", "contentBlock": [ { "texts": [ "Ingredients: Aqua Glycerin Pentylene Glycol Propylene Glycol Xanthan Gum Sodium Citrate Parfum Tocopherol Helianthus Annuus Seed Oil Citric Acid Hibiscus Sabdariffa Flower Extract Sorbitol Linalool Sodium Hyaluronate Aloe Barbadensis Leaf Juice Tetramethyl acetyloctahydronaphthalenes Hexyl Cinnamal Citronellol Phenoxyethanol Vanillin Acetyl Cedrene Ethylhexylglycerin Sodium Benzoate Potassium Sorbate" ] } ], "isExpanded": false, "showInInfoHintLayer": false }, { "header": "Verwendungshinweise", "contentBlock": [ { "texts": [ "Die Maske ist nicht wiederverwendbar! Nach dem Öffnen des Sachets sofort verbrauchen! Direkten Augenkontakt vermeiden. Nicht zur Pflege von Kindern verwenden!" ] } ], "isExpanded": false, "showInInfoHintLayer": false }, { "header": "Anschrift des Unternehmens", "contentBlock": [ { "texts": [ "dm-drogerie markt GmbH + Co. KG\nAm dm-Platz 1\n76227 Karlsruhe\nservice@dm.de" ] } ], "isExpanded": false, "showInInfoHintLayer": false }, { "header": "Hergestellt in", "contentBlock": [ { "texts": [ "Deutschland" ] } ], "isExpanded": false, "showInInfoHintLayer": false } ], "self": "/balea-tuchmaske-big-dreams-p4067796168532.html", "metadata": { "currency": "EUR", "description": "Die Balea Tuchmaske Big Dreams verwöhnt die Haut mit einer erfrischenden Pflegeformel. Angereichert mit Hibiskus-Extrakt, Hyaluron und Aloe Vera spendet sie...", "categoryCodes": [ "020207" ], "canonical": "/balea-tuchmaske-big-dreams-p4067796168532.html", "price": 0.95, "pageTitle": "Balea Tuchmaske Big Dreams, 1 St dauerhaft günstig online kaufen | dm.de", "robots": "index, follow, noodp" }, "price": { "infos": [ "1 St (0,95 € je 1 St)", "inkl. MwSt. zzgl. <linking>Versand</linking>" ], "paybackInfo": "Je 2 Euro Warenwert erhalten Sie 1 PAYBACK °Punkt", "price": { "current": { "value": "0,95 €" } } }, "netPrice": { "infos": [ "1 St (0,80 € je 1 St)", "exkl. MwSt. zzgl. <linking>Versand</linking>" ], "price": { "current": { "value": "0,80 €" } } }, "confidenceBuildingMeasuresWeb": [ "Kostenloser Versand ab 59 € mit dm-Konto", "Kostenlose Rückgabe auch im dm-Markt", "Kauf auf Rechnung möglich" ], "about": { "heading": "Infos zum Produkt" }, "isPurchasable": true, "isUserInExpressDeliveryArea": false, "quantitySelection": [ 1, 2, 3, 4 ], } ## Contact Please visit us through epctex.com to see all the products that are available for you. If you are looking for any custom integration or so, please reach out to us through the chat box in epctex.com. In need of support? business@epctex.com is at your service.