Chiso: Python Web Scraping Actor Template

A Python Actor template built on Apify's platform for web scraping using Crawlee and Beautiful Soup. It provides a structured starting point for building data extraction tools that parse HTML content.

Overview

This Actor is a template that scrapes website data using the Beautiful Soup library, wrapped within Crawlee's BeautifulSoupCrawler. It extracts page titles and URLs by default, with a codebase you can modify for custom scraping logic. The template includes pre-configured project structure, input validation, and local/cloud deployment workflows.

Key Features

• Beautiful Soup Integration: Uses BeautifulSoupCrawler from Crawlee for Python to parse and navigate HTML/XML.
• Structured Data Output: Saves scraped results to an Apify dataset, ensuring consistent JSON formatting.
• Configurable Crawling: Control the crawl via input parameters like startUrls and maxPagesPerCrawl.
• Local Development & Cloud Deployment: Full support for local testing with apify run and easy pushing to the Apify platform.
• Built-in Best Practices: Includes proxy configuration support, input/output schemas, and the Apify SDK for Actor development.

How to Use

Local Development

1. Install dependencies and run the Actor locally:
   bash apify run
2. To deploy to the Apify platform:
   bash apify login apify push

Project Structure

The main files and directories are:
* .actor/ – Configuration files (actor.json, input/output schemas).
* src/main.py – The main Actor entry point and crawling logic.
* storage/ – Local storage for datasets, key-value stores, and request queues (mirrors cloud structure).
* Dockerfile – Container definition for the Actor runtime.

How It Works

The crawler begins with URLs from the startUrls input. For each page, the requestHandler uses Beautiful Soup to parse the HTML. The default script extracts the page title and URL, saves each result as an item in the dataset, and logs the output. You can modify main.py to change the data extraction logic.

Input/Output

Input (via input_schema.json):
* startUrls (array): The list of URLs to begin crawling.
* maxPagesPerCrawl (number, optional): Limit the total number of pages scraped.

Output:
* Dataset: Contains JSON objects for each scraped page. By default, each item includes:
json { "url": "https://example.com", "title": "Example Page Title" }

Resources

• Apify SDK for Python
• Crawlee for Python Documentation
• Apify Actors Quick Start
• Input Schema Documentation
• Beautiful Soup on PyPI