Project Gutenberg Research Scraper

A Node.js template for scraping structured data from a single web page. You provide a URL via input, and the actor fetches the HTML, parses it, and stores the results in an Apify dataset. It's built with Axios for HTTP requests and Cheerio for HTML parsing.

Key Features

• Apify SDK: Provides the foundation for building and running the actor.
• Input Schema: Validates the input configuration, primarily the target URL.
• Dataset Storage: Outputs structured data (like page headings) into a queryable Apify dataset.
• Axios & Cheerio: Uses Axios for reliable page fetching and Cheerio for jQuery-style HTML parsing.
• Customizable Code: The scraping logic is simple to edit for targeting different page elements.

Input/Output

Input: The actor expects an input object containing the url of the page to scrape.

{
  "url": "https://example.com"
}

Output: Scraped data is saved to the actor's default dataset. The default template extracts page headings, producing items like:

{
  "heading": "Example Domain",
  "level": "h1"
}

You can modify the code to extract any other data from the page.

How to Use

Basic Usage (Apify Console)

1. Build the Actor using this template in the Apify Console.
2. Configure the Input by providing the target URL.
3. Run the Actor. The results will be available in the dataset tab.

How It Works

The actor's flow is straightforward:
1. Fetches the input configuration using Actor.getInput().
2. Downloads the HTML from the provided URL using axios.get(url).
3. Loads the HTML into Cheerio with cheerio.load(response.data) for parsing.
4. Executes the parsing logic (e.g., $("h1, h2, h3").each(...)) to extract data.
5. Saves each extracted item using Actor.pushData().

Local Development

To develop locally, pull the actor using the Apify CLI:
1. Install the CLI:
bash npm -g install apify-cli
2. Pull the actor by its unique name or ID:
bash apify pull <ActorId>
You can find the Actor ID or unique name in the Apify Console.

Resources

• Web scraping in Node.js with Axios and Cheerio
• Apify SDK for JavaScript documentation
• Apify Platform documentation
• Join our developer community on Discord