NEPSE Live Actor

An Apify actor built on the PuppeteerCrawler template for automated web scraping. It's designed to crawl JavaScript-rendered websites, making it suitable for extracting live data from dynamic pages like the Nepal Stock Exchange (NEPSE).

Overview

This actor uses Crawlee's PuppeteerCrawler to run headless Chrome browsers, allowing it to interact with and extract data from modern web applications. It includes a structured input schema, proxy rotation support, and organized request handling via a custom router.

Key Features

• Puppeteer Crawler: Parallel crawling using headless Chrome to handle JavaScript-heavy sites.
• Proxy Configuration: Integrated support for Apify Proxy or custom proxies to manage IP blocking, configured via Actor.createProxyConfiguration().
• Input Schema: Validates run configuration and start URLs through a defined JSON schema.
• Structured Output: Saves scraped data into an Apify Dataset for easy export and integration.
• Custom Routing: Uses a routes.js file to define specific handlers for different URL patterns.

How to Use

The actor's workflow is straightforward:

1. Input: Provide your configuration (like start URLs) via the INPUT.json file or the Apify console.
2. Setup: The actor initializes a proxy configuration and creates a PuppeteerCrawler instance.
3. Handling: Page requests are processed by a custom router (routes.js). You define handlers for specific page types (e.g., a 'detail' handler for product pages).
4. Execution: The crawler runs through the provided URLs, executing JavaScript and extracting data as defined in your handlers.
5. Output: Results are pushed to an Apify Dataset.

A basic handler in routes.js looks like this:

router.addHandler('detail', async ({ request, page, log }) => {
    const title = await page.title();
    await Dataset.pushData({
        url: request.loadedUrl,
        title,
    });
});

Input/Output

Input:
The actor expects a JSON input containing startUrls (an array of URLs to crawl). You can extend the input schema to include custom parameters like maximum pages or specific selectors.

Output:
Data is stored in an Apify Dataset. Each item's structure depends on your custom handlers, typically containing the URL and extracted data points (like titles, prices, or live figures).

Resources

• Crawlee + Apify Platform guide
• PuppeteerCrawler Documentation and Examples
• Apify SDK Documentation
• Input Schema Guide