Arab chat عربي شات

A TypeScript-based Actor template using PuppeteerCrawler from Crawlee. It's a production-ready boilerplate for building web crawlers that need to handle JavaScript-rendered pages. This template is designed for automation tasks and is open source.

Overview

This Actor provides a structured foundation for parallel web crawling with Puppeteer (headless Chrome). It's particularly useful for scraping dynamic websites where content is loaded or modified by JavaScript. The template handles common concerns like proxy rotation, input validation, and data storage, letting you focus on the specific page interaction logic.

Key Features

• Puppeteer Crawler: Core framework for parallel crawling using headless Chrome.
• Configurable Proxy: Integrated proxy management via Actor.createProxyConfiguration() to circumvent IP blocking. Supports Apify Proxy or custom proxy URLs.
• Input Schema: Validates and defines the structure for the Actor's input (e.g., start URLs).
• Dataset Storage: Outputs structured data to an Apify Dataset, where each item shares the same attributes.
• Custom Routing: Uses a router (routes.js) to define specific handlers for different URL patterns or page types.

How to Use

The Actor's workflow is straightforward:

1. Input: The Actor fetches its configuration, like start URLs, from INPUT.json.
2. Setup: It creates a proxy configuration and initializes a PuppeteerCrawler instance.
3. Routing: Page handling is defined in routes.js using createPuppeteerRouter(). You can set a default handler and add specific handlers (e.g., for product detail pages).
4. Execution: The crawler runs, processes pages based on your router logic, and stores results.

Example of a custom handler in routes.js:

router.addHandler('detail', async ({ request, page, log }) => {
    const title = await page.title();
    // Add your custom data extraction here
    await Dataset.pushData({
        url: request.loadedUrl,
        title,
    });
});

Input/Output

• Input: Typically provided via INPUT.json. The primary input is an array of startUrls to begin crawling. The exact structure can be customized using the input schema.
• Output: Data is pushed to an Apify Dataset using Dataset.pushData(). Each entry is a structured object containing the fields you extract (like url and title in the example above).

Resources

For further development and examples:
* Crawlee + Apify Platform guide
* PuppeteerCrawler Documentation and examples
* Apify SDK Documentation
* How to scale Puppeteer and Playwright

Getting Started

To build and run this Actor, refer to the Apify Actor development guide. The main steps are building the Actor in the Apify Console and then running it with your desired input.