Universal Apify Email & Metadata Scraper (Puppeteer + Crawlee)

Overview

This Actor is a production-ready template for building web crawlers that require JavaScript execution. It uses Crawlee's PuppeteerCrawler with headless Chrome to scrape websites, making it suitable for extracting data like emails and metadata from dynamic pages.

Key Features

• Puppeteer Crawler: Parallel crawling using headless Chrome via Puppeteer to handle JavaScript-rendered content.
• Proxy Management: Built-in configurable proxy support using Actor.createProxyConfiguration() to circumvent IP blocking. Works with Apify Proxy or custom proxy URLs.
• Structured Output: Stores scraped data in an Apify Dataset, where each item has consistent attributes.
• Input Validation: Uses an input schema (INPUT_SCHEMA.json) to define and validate run parameters.
• Enhanced Performance: Configured for deeper crawls (up to ~500 requests), higher concurrency (25-50 workers), and longer timeouts (90s).

How to Use

The Actor's logic is centered around a PuppeteerCrawler instance and a custom request router.

1. Input & Setup: The Actor starts by fetching input (like start URLs) from INPUT.json. A proxy configuration is created to manage IP rotation.
2. Crawler Initialization: A PuppeteerCrawler is instantiated with the proxy config and a requestHandler. The handler uses a router defined in routes.js to process pages.
3. Request Handling: In routes.js, you define handlers for different URL patterns using createPuppeteerRouter().
   • Use router.addDefaultHandler() for general page processing.
   • Use router.addHandler(label, ...) for specific page types (e.g., product detail pages).
4. Data Extraction: Inside your handlers, use Puppeteer's page object to interact with the page (e.g., page.title(), page.$eval()). Push structured results to the dataset with Dataset.pushData().
5. Execution: Start the crawl with crawler.run(startUrls).

Example Handler:

router.addHandler('detail', async ({ request, page, log }) => {
    const title = await page.title();
    // Add your custom scraping logic here
    await Dataset.pushData({
        url: request.loadedUrl,
        title,
    });
});

Input/Output

• Input: Configured via INPUT.json. Typical parameters include an array of startUrls and proxy settings. The exact schema is defined in INPUT_SCHEMA.json.
• Output: The Actor stores results in an Apify Dataset. Each item is a JSON object containing the scraped fields you define (e.g., URL, page title, extracted emails, metadata).

Resources

• Crawlee + Apify Platform guide
• PuppeteerCrawler documentation and examples
• Proxy configuration guide
• Custom routing with createPuppeteerRouter