Data.gov.uk Scraper

An Apify actor for scraping dataset metadata from the UK government's open data portal, data.gov.uk. It automates the collection of detailed information, supporting both direct URL scraping and search-based discovery.

Overview

This tool extracts structured data from data.gov.uk, eliminating the need for manual research. It's designed for developers, data analysts, and researchers who need to systematically gather UK open data intelligence. You can scrape specific dataset pages directly or use search filters to find relevant datasets based on keywords, publishers, topics, and formats.

Key Features

The scraper collects comprehensive metadata for each dataset, including:

• Dataset Titles & Descriptions: Full names and detailed summaries.
• Publisher Information: The originating government department or organization.
• Temporal Data: Last updated dates.
• Categorization: Topics (e.g., Business and economy, Transport, Health).
• Technical Details: Available file formats (CSV, JSON, XML, PDF) and licensing info (primarily Open Government License).
• Access Links: Direct URLs to dataset pages and download links for the actual data files.
• Contact Information: Provided enquiry links for datasets.

How to Use

Configure the actor run via input JSON. You can use it in two primary ways.

Option 1: Direct URL Scraping

Provide the exact URLs of the dataset pages you want to scrape.

{
  "startUrl": [
    "https://www.data.gov.uk/dataset/economic-review",
    "https://www.data.gov.uk/dataset/regional-economic-indicators"
  ],
  "maxItems": 10
}

Option 2: Search with Filters

Use search parameters to find and scrape datasets dynamically.

• searchQuery: Keywords (e.g., "transport", "health").
• publisher: Specific government department.
• topic: Category like Transport or Environment.
• format: File format (CSV, JSON, etc.).
• oglOnly: Set to true to filter for Open Government License.
• sort: Order results by "best" match or "recent" updates.
• maxItems: Limit the number of datasets scraped (required for free users).

Example: Basic Search

{
  "searchQuery": "economics",
  "sort": "best",
  "maxItems": 50
}

Example: Advanced Filtered Search

{
  "searchQuery": "transport",
  "publisher": "Department for Transport",
  "topic": "Transport",
  "format": "CSV",
  "oglOnly": true,
  "sort": "recent",
  "maxItems": 100
}

Input/Output

Input: Configure the scraper using the JSON input schema as shown in the examples above.

Output: The actor stores the scraped dataset metadata in the Apify dataset associated with the run. You can then download this structured data in multiple formats including JSON, CSV, Excel, XML, or HTML for further processing and analysis.