Data Go Kr Scraper

An Apify actor that programmatically extracts dataset listings from South Korea's official open data portal (data.go.kr). It collects metadata from thousands of public datasets, which include APIs, downloadable files, and linked data from various government agencies.

Target Users: Developers, data analysts, and researchers who need to discover or catalog Korean public data for applications, integration, or analysis.

Key Features

The scraper collects comprehensive dataset information, including:
* Core Metadata: Dataset ID, title, description, categories, and tags.
* Source Info: Providing organization and publisher contact details.
* Access Details: API endpoints, documentation URLs, file formats, and direct download links.
* Technical Data: Data fields, parameters (for APIs), and service type.
* Usage Stats: View counts, download statistics, and license information.
* Timestamps: Dataset creation and last update dates.

How to Use

Configure the actor via its input settings to define your search scope. The actor will navigate the portal, scrape the specified datasets, and output the structured results to an Apify dataset.

Input Configuration

Configure the scrape using the following input parameters. Provide them via the Apify console or as a JSON object.

• startUrl: A direct URL to a pre-filtered search results page from data.go.kr. Useful for replicating a specific browser search.
• keyword: A search term to query across dataset titles, descriptions, and keywords (e.g., 'weather').
• svcType: Filter by dataset type.
  • FILE: Downloadable files (CSV, XML, Excel, etc.).
  • API: REST API endpoints.
  • STD: Standardized format datasets.
  • LINKED: Semantic web linked data.
• recmSe: Filter by portal recommendation status (Y for recommended/high-quality datasets, N for all).
• conditionType: Search condition mode ('init' for default state, 'search' for active filtering).
• kwrdArray: Comma-separated list for an advanced keyword search (e.g., 'temperature,precipitation').
• maxItems: The maximum number of datasets to scrape. Free users are limited to 100. Paid users can set a limit up to 1,000,000 or leave empty for no limit.

Example Input:

{
  "keyword": "weather",
  "svcType": "API",
  "maxItems": 100
}

Output

The actor stores its results in an Apify dataset. You can download the data in multiple formats: JSON, CSV, Excel, XML, or HTML.

Example Output Item:

{
  "dataset_id": "15059093",
  "title": "JSON AsosDaIyInfoService",
  "organization": "Science and technology research",
  "dataset_type": "API",
  "description": "Provides daily weather observation data.",
  "api_endpoint": "http://apis.data.go.kr/...",
  "formats": ["JSON"],
  "tags": ["weather", "observation", "daily"],
  "view_count": 1250,
  "modified_date": "2023-11-15"
}