GitHub Repository Intelligence - API-Based Data Scraper

Extract structured repository data from GitHub using the official REST API. This actor fetches metadata, READMEs, topics, language stats, and more, outputting clean JSON. It's designed for AI training, developer research, and competitive analysis.

Overview

This tool provides a legal and reliable way to scrape GitHub via its public API. It avoids browser automation, using API calls for stability and speed. You can search for repositories by keywords and filters or directly target specific repo URLs.

Key Features

• Dual Scraping Modes:
  • Search Mode: Find repositories using GitHub's search syntax (e.g., language:python stars:>1000).
  • Direct Mode: Fetch data from a list of specific repository URLs.
• Comprehensive Data Extraction: Get repository metadata (stars, forks, issues), README content, programming language breakdowns, topics, license info, and timestamps.
• Official API Integration: Uses GitHub's REST API v3. Optional authentication with a Personal Access Token increases the rate limit from 60 to 5,000 requests per hour.
• Structured Output: Data is returned as consistent JSON, easily exported to CSV, JSON, or Excel for analysis.

How to Use

Configure the actor using an input JSON object. The core parameters are mode (either search or direct) and your target criteria.

Input Parameters

Example Input (Search Mode):

{
  "mode": "search",
  "searchQuery": "language:python stars:>1000",
  "sortBy": "stars",
  "maxResults": 50,
  "includeReadme": true,
  "includeTopics": true,
  "githubToken": "ghp_xxxxxxxxxxxx"
}

Example Input (Direct Mode):

{
  "mode": "direct",
  "repositoryUrls": "https://github.com/apify/apify-js\nhttps://github.com/nodejs/node",
  "includeReadme": true
}

Output

The actor outputs a dataset where each item contains the full data for one repository. The output is a detailed JSON object.

Example Output Item:

{
  "name": "tensorflow",
  "full_name": "tensorflow/tensorflow",
  "owner": {
    "login": "tensorflow",
    "type": "Organization"
  },
  "description": "An Open Source Machine Learning Framework for Everyone",
  "url": "https://github.com/tensorflow/tensorflow",
  "language": "C++",
  "stars": 185000,
  "forks": 74000,
  "open_issues": 1850,
  "topics": ["machine-learning", "deep-learning", "tensorflow", "python"],
  "license": "Apache License 2.0",
  "created_at": "2015-11-07T01:19:20Z",
  "updated_at": "2025-01-12T10:30:45Z",
  "readme": {
    "content": "# TensorFlow ...",
    "html_url": "https://github.com/tensorflow/tensorflow/blob/master/README.md"
  },
  "languages": {
    "C++": 60.5,
    "Python": 35.2,
    "Other": 4.3
  }
}

The dataset can be downloaded directly from the Apify platform in JSON, CSV, Excel, or other formats.