Apify Template for Scrapy Spiders This repository serves as a template for deploying Scrapy spiders to Apify. It is automatically updated by a GitHub Actions workflow in the central repository (getdataforme/central_repo) when changes are pushed to spider files in src/spiders/ or src/custom/. Below is an overview of the automated tasks performed to keep this repository in sync. ## Automated Tasks The following tasks are executed by the GitHub Actions workflow when a spider file (e.g., src/spiders/example/example_parser_spider.py) is modified in the central repository: 1. Repository Creation: - Creates a new Apify repository (e.g., example_apify) from this template (apify_template) using the GitHub API, if it doesn't already exist. - Grants push permissions to the scraping team in the getdataforme organization. 2. Spider File Sync: - Copies the modified spider file (e.g., example_parser_spider.py) from the central repository to src/spiders/ in this repository. - Copies the associated requirements.txt (if present) from the spider's directory (e.g., src/spiders/example/) to the root of this repository. 3. Input Schema Generation: - Runs generate_input_schema.py to create .actor/input_schema.json. - Parses the spider's __init__ method (e.g., def __init__(self, location:str, item_limit:int=100, county:str="Japan", *args, **kwargs)) to generate a JSON schema. - Supports types: string, integer, boolean, number (for Python str, int, bool, float). - Uses prefill for strings and default for non-strings, with appropriate editor values (textfield, number, checkbox). - Marks parameters without defaults (e.g., location) as required. 4. Main Script Update: - Runs update_main.py to update src/main.py. - Updates the actor_input section to fetch input values matching the spider's __init__ parameters (e.g., location, item_limit, county). - Updates the process.crawl call to pass these parameters to the spider (e.g., process.crawl(Spider, location=location, item_limit=item_limit, county=county)). - Preserves existing settings, comments, and proxy configurations. 5. Actor Configuration Update: - Updates .actor/actor.json to set the name field based on the repository name, removing the _apify suffix (e.g., example_apify → example). - Uses jq to modify the JSON file while preserving other fields (e.g., title, description, input). 6. Commit and Push: - Commits changes to src/spiders/$spider_file, requirements.txt, .actor/input_schema.json, src/main.py, and .actor/actor.json. - Pushes the changes to the main branch of this repository. ## Repository Structure - src/spiders/: Contains the Scrapy spider file (e.g., example_parser_spider.py). - src/main.py: Main script to run the spider with Apify Actor integration. - .actor/input_schema.json: JSON schema defining the spider's input parameters. - .actor/actor.json: Actor configuration with the repository name and metadata. - requirements.txt: Python dependencies for the spider. - Dockerfile: Docker configuration for running the Apify Actor. ## Prerequisites - The central repository (getdataforme/central_repo) must contain: - generate_input_schema.py and update_main.py in the root. - Spider files in src/spiders/ or src/custom/ with a valid __init__ method. - The GitHub Actions workflow requires a GITHUB_TOKEN with repository creation and write permissions. - jq and python3 are installed in the workflow environment. ## Testing To verify the automation: 1. Push a change to a spider file in src/spiders/ or src/custom/ in the central repository. 2. Check the generated Apify repository (e.g., getdataforme/example_apify) for: - Updated src/spiders/$spider_file. - Correct input_schema.json with parameters matching the spider's __init__. - Updated src/main.py with correct actor_input and process.crawl lines. - Updated .actor/actor.json with the correct name field. ## Notes > Warning: This Apify actor repository is automatically generated and updated by the GitHub Actions workflow in getdataforme/central_repo. Do not edit this repository directly. To modify the spider, update the corresponding file in src/spiders/ or src/custom/ in the central repository, and the workflow will sync changes to this repository, including: > - Copying the spider file to src/spiders/. > - Generating .actor/input_schema.json based on the spider’s __init__ parameters. > - Updating src/main.py with correct input handling and spider execution. > - Setting the name field in .actor/actor.json (e.g., example for example_apify). > > Verification: After the workflow completes, verify the actor by checking: > - src/spiders/$spider_file matches the central repository. > - .actor/input_schema.json includes all __init__ parameters with correct types and defaults. > - src/main.py has updated actor_input and process.crawl lines. > - .actor/actor.json has the correct name. > - Optionally, deploy the actor to Apify and test with sample inputs to ensure functionality. - The workflow supports multiple spider types (scrapy, hrequest, playwright) based on the file path (src/spiders/, src/custom/*/hrequest/, src/custom/*/playwright/). - Commits with [apify] in the message update only Apify repositories; [internal] updates only internal repositories; otherwise, both are updated. - Ensure the spider's __init__ uses supported types (str, int, bool, float) to avoid schema generation errors. For issues, check the GitHub Actions logs in the central repository or contact the scraping team.