Willhaben.at Car Scraper

# Willhaben.at Car Scraper A lightweight scraper for extracting car listings from willhaben.at. ## Related scrapers: - Checkout Simple result Willhaben.at Car Scraper - Checkout Full result Willhaben.at Car Scraper ## Features - Extract car listings from custom search URLs - Control pagination with configurable page limits - Export data in JSON format with detailed vehicle information - Support for multiple search URLs in a single run > Note: Willhaben.at limits results to maximum 20 pages per search. For larger datasets, split your searches into smaller chunks using different filter combinations in your URLs. --- ## Configuration ### start_urls (required) An array of URL objects containing Willhaben.at search pages to scrape. - Type: Array of objects with url property - Format: Each object must contain a url field with a valid Willhaben.at search URL - Example: json "start_urls": [ { "url": "https://www.willhaben.at/iad/gebrauchtwagen/auto/gebrauchtwagenboerse?rows=30" } ] How to get search URLs: 1. Go to willhaben.at 2. Navigate to the car section (Gebrauchtwagen/Auto) 3. Use the search filters (make, model, price range, location, etc.) 4. Copy the URL from your browser's address bar 5. Add it to the start_urls array ### scrape_page_limit (optional) Limits the number of search result pages to scrape from each start URL. - Type: Number - Default: No limit (scrapes all available pages, up to Willhaben.at's maximum of 20) - Example: "scrape_page_limit": 1 (scrapes only the first page) - Use case: Useful for testing or when you only need recent listings ### Configuration Examples #### Minimal configuration (scrape all pages) json { "start_urls": [ { "url": "https://www.willhaben.at/iad/gebrauchtwagen/auto/gebrauchtwagenboerse?rows=30" } ] } #### With page limit (test mode) json { "start_urls": [ { "url": "https://www.willhaben.at/iad/gebrauchtwagen/auto/gebrauchtwagenboerse?rows=30" } ], "scrape_page_limit": 1 } #### Multiple search URLs json { "start_urls": [ { "url": "https://www.willhaben.at/iad/gebrauchtwagen/auto/gebrauchtwagenboerse?PRICE_FROM=0&PRICE_TO=10000" }, { "url": "https://www.willhaben.at/iad/gebrauchtwagen/auto/gebrauchtwagenboerse?PRICE_FROM=10000&PRICE_TO=20000" } ], "scrape_page_limit": 5 } --- ## Output ### Output Data Structure Each scraped car listing includes detailed information about: - Basic Info: ID, URL, heading/description - Vehicle Details: Make, model, model specification, year, car type (body style) - Pricing: Price amount, formatted price - Technical Specs: Power (kW/PS), fuel type, transmission, mileage, number of seats - Features: Equipment list (resolved to readable names) - Condition: Condition report, number of previous owners, warranty status - Location: Country, state, district, postal code, exact coordinates - Seller Info: Type (private/dealer), organization details - Media: Main image, all photo URLs - Additional: Published date, last updated timestamp, exterior color ### Output Example json { "id": "1654277607", "url": "https://www.willhaben.at/iad/gebrauchtwagen/d/auto/bmw-5er-reihe-525d-xdrive-1654277607/", "description": "BMW 5er-Reihe 525d xdrive", "photos": [ "https://cache.willhaben.at/mmo/7/165/427/7607_1988883143.jpg", "https://cache.willhaben.at/mmo/7/165/427/7607_-2021590471.jpg", "https://cache.willhaben.at/mmo/7/165/427/7607_-804916147.jpg" ], "highlight": [ { "prefix": null, "value": "2009", "postfix": "EZ" }, { "prefix": null, "value": "352.000", "postfix": "km" }, { "prefix": null, "value": "197", "postfix": "PS (145 kW)" } ], "attributes": { "attribute": [ { "name": "BODY_DYN", "values": [ "Seit 2013 in meinem Besitz. Es wurde immer alles fachgerecht repariert..." ] }, { "name": "CAR_MODEL/MAKE", "values": ["BMW"] }, { "name": "CAR_MODEL/MODEL", "values": ["5er-Reihe"] }, { "name": "CAR_MODEL/MODEL_SPECIFICATION", "values": ["525d xdrive"] }, { "name": "PRICE/AMOUNT", "values": ["4900.0"] }, { "name": "PRICE_FOR_DISPLAY", "values": ["€ 4.900"] }, { "name": "YEAR_MODEL", "values": ["2009"] }, { "name": "MILEAGE", "values": ["352000"] }, { "name": "ENGINE/EFFECT", "values": ["145"] }, { "name": "ENGINE/FUEL_RESOLVED", "values": ["Diesel"] }, { "name": "TRANSMISSION_RESOLVED", "values": ["Automatik"] }, { "name": "CAR_TYPE", "values": ["Kombi / Family Van"] }, { "name": "CONDITION_RESOLVED", "values": ["Gebrauchtwagen"] }, { "name": "NO_OF_OWNERS", "values": ["2"] }, { "name": "NOOFSEATS", "values": ["5"] }, { "name": "LOCATION", "values": ["Wien, 04. Bezirk, Wieden"] }, { "name": "POSTCODE", "values": ["1040"] }, { "name": "STATE", "values": ["Wien"] }, { "name": "COORDINATES", "values": ["48.191558837890625,16.370309829711914"] }, { "name": "EXTERIORCOLOURMAIN", "values": ["5"] }, { "name": "PUBLISHED_String", "values": ["2024-08-27T00:00:00Z"] }, { "name": "EQUIPMENT_RESOLVED", "values": [ "Regensensor", "Fahrerairbag", "Nebelscheinwerfer", "Leichtmetall-/Alufelgen", "Antriebsschlupfregelung (ASR)", "CD Player", "Nichtraucherfahrzeug", "Mittelarmlehne", "Lederlenkrad", "Kurvenfahrlicht", "Kopfairbags", "Schisack", "Multifunktionslenkrad", "elektr. Sitze", "Lichtsensor", "Parksensor hinten", "ISOFIX", "Beifahrerairbags", "Head-Up-Display", "Tempomat", "Bluetooth / USB", "Stabilitätsprogramme (ESP)", "Lederausstattung", "Radio", "Klimaautomatik", "elektr. Spiegel", "Dachreling", "Servolenkung", "Parksensor vorne & hinten", "ABS", "Tagfahrlicht", "elektr. Fensterheber", "Zentralverriegelung", "Klimaanlage", "Reifendrucksensor", "Navigationssystem", "Bordcomputer", "Xenon" ] }, { "name": "WARRANTY_RESOLVED", "values": ["Nein"] } ] }, "seller": { "type": "Privat", "iconType": "NONE", "upsellingOrgLogo": null } } --- ## Limitations - Willhaben.at limits search results to maximum 20 pages per query - No built-in proxy support in this version - Single domain support (willhaben.at) - Country-specific domains may have variations (not all tested) Workaround for large datasets: Split your searches using different filter combinations: - By price range: €0-10,000, €10,000-20,000, etc. - By location/region (federal states) - By make/model - By year range - By body type (Limousine, Kombi, SUV, etc.) --- ## Support If this scraper doesn't match your needs or you encounter issues, please contact: - Email: realspidery@gmail.com - X/Twitter: @realspidery For feature requests or bugs, please provide: - Your input configuration - Error messages or logs - Expected vs actual behavior