CSLB (Contractors State License Board) Scraper

Overview

This is an Apify Actor for scraping contractor license data from the CSLB website. It's built on a Python template using HTTPX for asynchronous requests and Beautiful Soup for HTML parsing. The Actor systematically crawls from a starting URL, follows nested links up to a defined depth, and extracts structured data into a dataset.

Key Features

• Built with Apify SDK for Python – Provides the framework for building and running the Actor.
• Request Queue Management – Handles the enqueuing and processing of URLs, supporting recursive crawling.
• Structured Data Storage – Outputs scraped data into a dataset where each item has consistent attributes.
• HTTPX & Beautiful Soup – Uses HTTPX for efficient HTTP requests and Beautiful Soup for parsing HTML/XML.
• Input Schema – Validates run configuration via a defined input schema.
• Error Handling – Catches and logs exceptions during the scraping process.

How to Use

The Actor is designed to be run from the Apify platform. You configure the run via input, then execute it. The scraper will:
1. Read the starting URLs and maximum crawl depth from the input.
2. Place the starting URLs into a request queue with depth 0.
3. Process each URL: fetch the page, parse it with Beautiful Soup, and extract the target data.
4. If the current page's depth is less than max_depth, find new links on the page and add them to the queue with an incremented depth.
5. Push the extracted data from each page to the default dataset.

For local development, you can pull the Actor code using the Apify CLI.

Input/Output

Input (via JSON):

{
  "start_urls": [
    { "url": "https://www.cslb.ca.gov/..." }
  ],
  "max_depth": 1
}

• start_urls: An array of URLs where scraping should begin.
• max_depth: Defines how many levels of nested links the scraper should follow (0 = only the start URLs).

Output:
The scraped data is stored in the Actor's default dataset. Each item in the dataset contains the data extracted from a single scraped page (e.g., links, page content). You can access the results via the Apify dataset API, export them to JSON, CSV, or other formats, or process them in connected applications.

Resources

• BeautifulSoup Scraper
• Beautifulsoup Scraper tutorial
• Python tutorials in Academy
• Web scraping with Beautiful Soup and Requests
• Integration with Make, GitHub, Zapier, Google Drive, and other apps
• Video guide on getting scraped data using Apify API