Zip Download and Extraction Scraper

A Python utility for downloading and optionally extracting ZIP files from a list of URLs. It's designed for both local use and deployment as an Apify actor.

Overview

This tool reads a list of URLs from a JSON input, downloads the ZIP files in parallel batches, and saves them to an output folder. It includes retry logic for network errors and can automatically extract the contents of downloaded archives.

Key Features

• Flexible Input: Reads URLs from a local input.json file or the APIFY_INPUT environment variable (for Apify platform runs).
• Parallel Downloads: Uses configurable threading for faster batch downloads.
• Robust Operation: Implements retries on network failures and skips already-downloaded files by default.
• Optional Extraction: Can automatically extract the contents of downloaded ZIP files.
• Apify Integration: Ready to be deployed and run as an Apify actor.

How to Use

Local Execution

1. Prepare Input: Create an input.json file with a urls array.
   json { "urls": ["https://example.com/file1.zip", "https://example.com/file2.zip"], "extract": true, "batch": 4, "retries": 3 }
2. Install Dependencies:
   bash pip install -r requirements.txt
3. Run the Script:
   bash python src/download.py --input input.json

Common Command-Line Options:
* --output <folder>: Specify output directory (default: output).
* --batch N: Set number of parallel downloads (default: 4).
* --retries N: Set retry attempts per URL (default: 3).
* --overwrite: Overwrite existing files in the output folder.
* --extract: Extract ZIP contents after download.

Apify Platform

Deploying the Actor:
1. Install the Apify CLI: npm install -g @apify/cli
2. Log in: apify login
3. Push from the project root: apify push

Running the Actor:
Provide input via the Apify console or API using JSON like:

{
    "urls": ["https://github.com/psf/requests/archive/refs/heads/main.zip"],
    "extract": true
}

The actor receives this input via the APIFY_INPUT environment variable.

Docker Execution

You can also build and run the tool locally using Docker:

docker build -t zip-downloader .
docker run --rm -e APIFY_INPUT='{"urls":["https://example.com/file.zip"],"extract":true}' zip-downloader

Input / Output

Input Format:
The tool accepts a JSON object, either from a file or an environment variable. Key properties are:
* urls (required): Array of URLs to download.
* extract (optional, boolean): If true, extracts downloaded ZIPs.
* batch (optional, integer): Number of parallel downloads.
* retries (optional, integer): Retry attempts per URL.

Output:
* Downloaded .zip files are saved to the specified output folder (default output/).
* If extraction is enabled, contents are extracted into a folder named after the ZIP file (e.g., output/archive-name/).