🔥Fast Indeed Job Scraper

# Fast Indeed Job Scraper A powerful and configurable web scraper for extracting job listings from Indeed.com. This actor efficiently collects job metadata and full descriptions, supporting various search parameters and customization options for production-scale data collection. ## Features - Comprehensive Job Data: Extracts job titles, companies, locations, salaries, posting dates, and detailed descriptions. - Flexible Search Options: Accepts full search URLs or builds queries from keywords and locations. - Pagination Support: Automatically handles multiple pages of search results. - Customizable Scraping: Control concurrency, proxy usage, and session management. - Output to Dataset: Results are stored in Apify's dataset for easy export and integration. ## Input Configuration The actor accepts a JSON input object with the following parameters. All fields are optional, with sensible defaults applied. ### Search Parameters | Parameter | Type | Description | Default | |-----------|------|-------------|---------| | searchUrl | string | Full Indeed search URL. If provided, keyword, location, and country are ignored. | - | | keyword | string | Search keywords for job queries. | - | | location | string | Geographic location filter. | - | | country | string | Country-specific Indeed domain (e.g., "United States", "United Kingdom"). | "United States" | | posted_date | string | Filter by posting date (e.g., "1d", "7d", "30d" for days). | - | ### Scraping Options | Parameter | Type | Description | Default | |-----------|------|-------------|---------| | maxItems | number | Maximum number of job listings to collect. | 100 | | collectDetails | boolean | Whether to fetch full job descriptions from detail pages. | false | | maxConcurrency | number | Number of parallel requests to make. | 5 | | cookies / cookiesJson | object/string | Required for high-volume runs: session cookies that keep you logged in and reduce blocking. | - | | proxyConfiguration | object | Proxy settings for requests. | - | ## Output Schema The actor outputs job listings to the default dataset. Each item is a JSON object with the following structure: | Field | Type | Description | |-------|------|-------------| | title | string | Job title | | company | string | Company name | | location | string | Job location | | postedAt | string | Human-readable posting date | | salary | string | Salary information (if available) | | description_html | string | Full job description in HTML format | | description_text | string | Plain text version of the job description | | url | string | Direct link to the job posting | | source | string | Data source identifier ("indeed") | | search_url | string | Original search URL where the job was found | ## Usage ### Basic Example To scrape software engineering jobs in remote locations: json { "keyword": "software engineer", "location": "Remote", "maxItems": 100, "collectDetails": true } ### Advanced Configuration For large-scale scraping with proxies: json { "searchUrl": "https://www.indeed.com/jobs?q=developer&l=New+York&fromage=7", "maxItems": 500, "collectDetails": true, "maxConcurrency": 3, "proxyConfiguration": { "useApifyProxy": true, "groups": ["RESIDENTIAL"] } } ### Running the Actor 1. Via Apify Console: - Navigate to the actor page - Configure input parameters - Click "Run" and monitor progress 2. Via API: bash curl -X POST "https://api.apify.com/v2/acts/shahidirfan100~fast-indeed-job-scraper/runs" \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"keyword": "data scientist", "location": "San Francisco"}' 3. Via Apify CLI: bash apify run shahidirfan100/fast-indeed-job-scraper --input '{"keyword": "marketing", "maxItems": 50}' ## Configuration Tips - Optimizing Performance: Adjust maxConcurrency based on your proxy setup. Higher values speed up scraping but may trigger rate limits. - Handling Large Datasets: Provide valid cookies/cookiesJson for any run above ~50 jobs; Indeed blocks anonymous traffic quickly at scale. - Proxy Usage: Enable residential proxies for better success rates on large crawls. - Authentication: Provide cookies from a logged-in session to access premium features or reduce blocking. - Strict Locales: Markets such as South Africa, China, and UAE enforce heavier bot controls—keep maxConcurrency ≤ 4 and expect slower warm-up even with cookies. ### Getting Cookies via Cookie-Editor (Chrome) 1. Install the free Cookie-Editor extension from the Chrome Web Store. 2. Open https://www.indeed.com (or your target country domain) and sign in manually. 3. Click the Cookie-Editor icon → Export → copy the JSON string. 4. Paste that JSON into the actor input under cookiesJson. 5. For large, long-running jobs, refresh cookies regularly to keep sessions valid. ## Troubleshooting ### Common Issues - Incomplete Results: Verify search parameters and increase maxItems if needed. - Rate Limiting: Reduce maxConcurrency or add delays between requests. - Blocking/CAPTCHA: Use high-quality proxies and valid session cookies. - No Results: Check if the search URL is valid and accessible. ### Best Practices - Start with small test runs to validate configuration. - Monitor the actor's log for errors and adjust settings accordingly. - Use the dataset export features for data analysis and integration. ## Support For issues or feature requests, please contact support through the Apify platform. --- This actor is designed for ethical web scraping. Always respect website terms of service and robots.txt files.