My Actor

A web scraping Actor built with Crawlee and CheerioCrawler, designed to extract data from websites. It's categorized for AI, E-commerce, and Social Media use cases.

Overview

This Actor is a JavaScript template that uses Cheerio to parse HTML and scrape structured data from web pages. It starts from a list of URLs, crawls pages, extracts information (like page titles and URLs), and saves the results to a dataset. It's built on the Apify SDK and is suitable for scraping static HTML content.

Key Features

• CheerioCrawler: Uses the fast Cheerio library for parsing HTML, ideal for static websites.
• Structured Data Output: Saves scraped items into an Apify dataset with a defined schema.
• Configurable Input: Control the crawl via startUrls and maxPagesPerCrawl parameters.
• Proxy Support: Includes configuration for proxy rotation to help avoid blocks.
• Local & Cloud Development: Full local development workflow with easy deployment to the Apify platform.

How to Use

Local Development & Run

1. Install dependencies: npm install
2. Start the Actor locally: apify run
3. Deploy to Apify Console:
   bash apify login apify push

Project Structure

.actor/
├── actor.json          # Actor configuration and settings
├── input_schema.json   # Defines and validates Actor input
└── dataset_schema.json # Defines the structure of output data
src/
└── main.js            # Main Actor entry point and logic

How It Works

1. The crawler begins with URLs from the input's startUrls.
2. It fetches each page and uses Cheerio to parse the HTML.
3. The requestHandler function extracts data (e.g., page title and URL) from each page.
4. Extracted items are saved to the dataset and logged.
5. The crawl respects the maxPagesPerCrawl limit set in the input.

Input / Output

Input (via input_schema.json):
* startUrls (array): List of URLs to start crawling from.
* maxPagesPerCrawl (number, optional): Limit for the total number of pages to scrape.

Output:
* Dataset: Contains saved items, typically including url and title for each scraped page, following the structure in dataset_schema.json.

Included Tools & Resources

• Apify SDK - Toolkit for building Actors.
• Crawlee - Web scraping and browser automation library.
• Cheerio - Library for parsing and manipulating HTML/XML.
• Input schema - For defining and validating Actor input.
• Dataset - For storing structured output data.
• Proxy configuration - Support for IP rotation.

Helpful Links:
* Quick Start for building Actors.
* Video tutorial on building a scraper with CheerioCrawler.
* Written tutorial on using CheerioCrawler.
* Web scraping with Cheerio in 2023
* How to create Actors from templates

Getting Started for Local Development

To pull an existing Actor from the Apify Console for local work:
1. Install the Apify CLI:
* Homebrew: brew install apify-cli
* NPM: npm -g install apify-cli
2. Pull the Actor using the CLI.