ScraperCodeGenerator

An Apify actor that automatically scrapes websites and generates the Python code to do it yourself. You provide a URL and a description of the data you want; it returns the extracted data and a ready-to-run BeautifulSoup script.

Overview

This actor eliminates manual scraping work. Instead of writing code, you describe your goal in plain English. The actor tests multiple scraping strategies (like Cheerio, Playwright, etc.) in parallel, uses Claude AI to evaluate and select the best results, extracts your data, and then generates a custom Python script tailored to the target website. You get both the structured data and the reusable code.

Key Features

• AI-Powered Scraping: Uses Claude AI to understand your goal, evaluate results from different methods, and ensure accurate data extraction.
• Parallel Strategy Testing: Runs several scraping approaches simultaneously for speed and resilience. If one method fails, others can succeed.
• Custom Code Generation: The core output is a personalized, standalone Python script (using BeautifulSoup) that you can run, modify, and integrate into your own projects.
• Zero Coding Required: Input is a simple JSON object with a URL and a plain-English description.
• Production-Ready Output: The generated code is clean, documented, and saved as a separate downloadable file.

How to Use

Configure the actor with a JSON input containing:

• targetUrl: The website you want to scrape.
• userGoal: A specific description of the data you need (e.g., "product titles and prices").
• claudeApiKey: Your Anthropic Claude API key for the AI analysis.

Example Input:

{
  "targetUrl": "https://books.toscrape.com/",
  "userGoal": "Get me a list of all the books on the first page. For each book, I want its title, price, star rating, and whether it is in stock.",
  "claudeApiKey": "sk-ant-..."
}

The actor will process your request and provide the outputs below.

Input / Output

Input:
A JSON object with the three required fields: targetUrl, userGoal, and claudeApiKey.

Output:
The actor saves two primary results:

1. To the Dataset: The extracted data, structured according to your goal, along with performance scores for each scraping method and an indication of the best method.
2. To the Key-Value Store (GENERATED_SCRIPT): The custom Python scraping script. After the run, download this file from the "Key-value store" tab in your run details and rename it with a .py extension.

You get the immediate data and a reusable script to scrape the same site in the future without needing the actor again.