my-actor: Single-Page Web Scraper

Overview

A Node.js template for scraping structured data from a single web page. You provide a URL via input, and the actor fetches the HTML, parses it, and stores the results in an Apify dataset. It's built for ecommerce, SEO, and open-source use cases, but you can modify it to extract any data from a page.

Key Features

• Apify SDK: Provides the scaffolding for building and running the actor.
• Input Schema: Validates the input configuration, primarily the target URL.
• Axios Client: Handles the HTTP request to fetch the page HTML.
• Cheerio: Parses the HTML using a jQuery-like syntax to extract data.
• Dataset Storage: Outputs structured data (like page headings) into a dataset for easy access and export.

Input/Output

Input: The actor expects a JSON input containing the url of the page to scrape.

{
  "url": "https://example.com"
}

Output: By default, the actor scrapes all heading elements (h1 through h6). The data is pushed to an Apify dataset, where each item is an object containing the heading text and its tag name. You can modify the Cheerio selector in the code to extract different elements.

How to Use

1. Configure Input: Set the target page URL in the actor's input.
2. Run the Actor: Start the run via the Apify Console, API, or CLI.
3. Access Data: Once finished, retrieve the scraped data from the actor's dataset tab in JSON, CSV, or other formats.

The core scraping logic works as follows:
1. Actor.getInput() retrieves the input configuration.
2. axios.get(url) fetches the HTML of the target page.
3. cheerio.load(response.data) loads the HTML for parsing.
4. A Cheerio selector (e.g., $("h1, h2, h3, h4, h5, h6")) iterates through elements to extract data.
5. Actor.pushData() stores the extracted records into the dataset.

To customize what data is scraped, edit the Cheerio selector within the actor's code.

Local Development

You can pull this actor to your local machine for development using the Apify CLI.

1. Install the CLI:
   ```bash
   # Using npm
   npm -g install apify-cli

# Using Homebrew
brew install apify-cli
```

1. Pull the actor using its unique name or ID:
   bash apify pull <ActorId>
   Find the Actor ID or unique name (like trayv/my-actor) in the Apify Console.

Resources

• Web scraping in Node.js with Axios and Cheerio
• Apify SDK for JavaScript documentation
• Apify Platform documentation
• Join our developer community on Discord

For a full guide on building actors, see this article.