Scrapy Books Example

A Python Scrapy project example that scrapes book data from books.toscrape.com. It's designed to run both as a standard Scrapy project locally and as a deployable Apify Actor.

Overview

This actor demonstrates how to integrate a Scrapy web scraper with the Apify platform. It crawls the books.toscrape.com website, extracting details like titles, prices, and ratings, and outputs the data in a structured format.

Key Features

• Scrapy Integration: Built as a standard Scrapy project (book_spider), making it familiar for Python developers.
• Dual Execution: Can be run locally as a Scrapy project or as an Apify Actor.
• Apify Platform Ready: Easily deployable to the Apify cloud for scheduling, scaling, and integration.
• Example Dataset: Scrapes book titles, prices, availability, and ratings from a dedicated testing site.

How to Use

Prerequisites

• Apify CLI: Install from Apify CLI docs.
• Python & uv: Ensure Python is installed. Then install the uv package manager:
  bash pip install uv

Running Locally

1. Install dependencies and activate the virtual environment:
   bash make install-dev source .venv/bin/activate
2. Run as a Scrapy project:
   bash scrapy crawl book_spider -o books.json
3. Run as an Apify Actor locally:
   bash apify run --purge

Deploying to Apify

1. Log in with your Apify API Token:
   bash apify login
2. Deploy the actor to the Apify Platform:
   bash apify push
   The actor will appear under Actors -> My Actors.

Input/Output

• Input: This example actor does not require specific input parameters. It is configured to start scraping from the books.toscrape.com homepage.
• Output: The actor outputs a dataset containing a list of book objects. Each object typically includes fields such as title, price, availability, rating, and the product url. The default storage is a dataset on the Apify platform or a local JSON file when run with Scrapy directly.

Documentation Reference

• Integrating Scrapy projects
• Apify SDK for Python
• Apify Platform
• Join our developer community on Discord