Pro Web Content Crawler (With Images)

A Python-based web scraping Actor built on Apify's platform. It systematically crawls websites starting from provided URLs, extracts content and images using BeautifulSoup, and outputs structured data. Built with Crawlee for Python for robust crawling and queue management.

Overview

This Actor is a template for scraping web content and images. You provide starting URLs via the input configuration. The Actor crawls from those points, following links according to your settings, and extracts data from each page using BeautifulSoup to parse HTML. Extracted data is stored in an Apify dataset for easy retrieval and export.

It's designed for automation and AI data collection workflows, handling the complexities of request queues, retries, and data storage.

Key Features

• Crawlee for Python: Handles the crawling logic, request queues, and concurrency.
• BeautifulSoup Integration: Extracts and parses data from HTML/XML content.
• Managed Request Queue: (RequestQueue) Controls the flow of URLs to be scraped.
• Structured Data Storage: (Dataset) Stores all scraped results in a consistent format.
• Input Schema: Validates and defines the configuration for each Actor run.
• Apify SDK: Provides the foundation for building and running the Actor on the Apify platform.

Input/Output

Input (Configured via the Actor's input schema):
* startUrls: (Required) List of URLs where the crawl will begin.
* maxDepth: (Optional) How many link levels deep the crawler should go.
* maxPages: (Optional) Limit on the total number of pages to scrape.
* extractImages: (Optional) Boolean to enable/disable image URL extraction.
* customCssSelectors: (Optional) Define specific CSS selectors for targeted data extraction.

Output:
The Actor stores its results in an Apify dataset. Each item typically includes:
* url: The source URL of the page.
* title: The page title.
* text: The main textual content extracted.
* images: (If enabled) A list of image URLs found on the page.
* metadata: Such as scrape timestamp and page depth.

Data can be exported as JSON, CSV, XML, or via the Apify API.

How to Use

On the Apify Platform

1. Configure the Actor's input with your startUrls and desired parameters (max depth, page limit, etc.).
2. Run the Actor. It will begin crawling and extracting data.
3. Once finished, access the scraped data from the dataset tab in the Actor run console. You can preview, export, or connect it to other apps via Apify integrations.

Local Development

To modify or run the Actor locally, use the Apify CLI to pull the code:

1. Install the Apify CLI:
   ```bash
   # Using npm
   npm -g install apify-cli

   Or using Homebrew

   brew install apify-cli
   ```

2. Pull the Actor using its unique name or ID (found in the Apify console):
   bash apify pull <ActorId>

3. Develop locally. The core logic resides in the request handler function where you define your BeautifulSoup parsing.

Resources

• Apify SDK for Python documentation
• Crawlee for Python documentation
• Apify Platform documentation
• Apify integrations (Make, Zapier, Google Drive, etc.)
• Join the developer community on Discord