🤖 GitHub Documentation Intelligence > AI-powered documentation extraction and analysis for GitHub repositories Extract, structure, and analyze documentation from any GitHub repository in seconds. Perfect for building RAG systems, onboarding developers, and auditing documentation quality. --- ## 🎯 What It Does Automatically extracts and structures: - ✅ README files - Main project documentation - ✅ Documentation folders - All markdown files from docs/, documentation/, etc. - ✅ Code documentation - Docstrings from Python, JavaScript, TypeScript files - ✅ Metadata - Repository info, stars, language, topics - ✅ Statistics - Word counts, file counts, documentation coverage --- ## 🚀 Quick Start ### Input Example json { "url": "https://github.com/pallets/flask", "maxFiles": 20, "extractCodeDocs": true } ### Output Example json { "status": "success", "metadata": { "name": "flask", "description": "The Python micro framework", "language": "Python", "stars": 65000, "url": "https://github.com/pallets/flask" }, "readme": { "filename": "README.md", "content": "...", "sections": [...], "word_count": 450 }, "documentation_files": [...], "code_documentation": [...], "combined_markdown": "...", "statistics": { "has_readme": true, "documentation_files_count": 23, "code_files_with_docs": 15, "total_words": 12500, "total_docstrings": 87 } } --- ## ⭐ Key Features ### 📊 Comprehensive Extraction - Extracts README, docs folders, and code docstrings - Supports Python, JavaScript, TypeScript - Handles nested documentation structures - Preserves markdown formatting and sections ### 🎯 Structured Output - Clean JSON format ready for processing - Pydantic models for type safety - Combined markdown for easy reading - Detailed statistics and metadata ### 🛡️ Robust & Reliable - Proper error handling - Rate limit management - Partial success handling - Detailed logging ### ⚡ Fast & Efficient - Async operations - Smart file filtering - Configurable limits - Optimized API usage --- ## 💡 Use Cases ### 🤖 RAG Systems Extract clean documentation for training AI models: python # Use extracted docs for embeddings docs = result['combined_markdown'] chunks = create_embeddings(docs) ### 👨‍💻 Developer Onboarding Generate comprehensive repo overviews: - Understand project structure - Find key documentation - Identify important files ### 📈 Documentation Audits Analyze documentation quality: - Check completeness - Identify gaps - Track improvements ### 🔍 Code Search Enable semantic search over codebases: - Search through docstrings - Find relevant code examples - Understand APIs --- ## 🔧 Configuration ### GitHub Token (Recommended) For private repos and higher rate limits (5,000 vs 60 requests/hour): 1. Go to https://github.com/settings/tokens 2. Generate new token (classic) 3. Select scopes: repo or public_repo 4. Add to input: "githubToken": "ghp_your_token" ### Options | Option | Type | Default | Description | |--------|------|---------|-------------| | maxFiles | integer | 100 | Maximum files to process | | extractCodeDocs | boolean | true | Extract code docstrings | --- ## 📊 Statistics Provided - has_readme: Whether README exists - documentation_files_count: Number of doc files found - code_files_with_docs: Number of code files with docstrings - total_words: Total documentation words - total_lines: Total documentation lines - total_docstrings: Total docstrings extracted --- ## 🛠️ Development ### Local Testing bash # Install dependencies pip install -r requirements.txt # Run locally apify run ### Project Structure . ├── src/ │ ├── main.py # Actor entry point │ ├── extractor.py # Extraction logic │ ├── models.py # Data models │ └── utils.py # Helper functions ├── .actor/ │ ├── actor.json # Actor configuration │ └── input_schema.json # Input schema ├── requirements.txt # Dependencies └── Dockerfile # Container config --- ## 🤝 Contributing Issues and pull requests welcome! This is an active project participating in the Apify $1M Challenge. --- ## 📝 License Apache 2.0 --- ## 💬 Support - Questions? Join Apify Discord - Issues? Open a GitHub issue - Need help? Check Apify documentation --- ## 🎯 Coming Soon - 🔜 Documentation quality scoring (A-F grades) - 🔜 MCP server for AI agents - 🔜 Change detection and tracking - 🔜 Multi-repo comparison - 🔜 PDF documentation support - 🔜 Website documentation scraping ##FAQs Q: Why did extraction fail? A: Common reasons: 1.Repository doesn't exist (check URL) 2.Repository is private (add GitHub token) 3.Rate limit exceeded (add token for 5000/hour) 4.Repository is too large (reduce maxFiles) Q: What if I hit rate limits? A: Without token: 60 requests/hour With token: 5,000 requests/hour Get token: https://github.com/settings/tokens Q: Can I extract from private repos? A: Yes! Add your GitHub token in the input: json{ "source": { "url": "...", "githubToken": "ghp_your_token" } } Q: What's the maximum repository size? A: 1.Max 500 files per run 2.Max 5MB per file 3.Max 50MB total data 4.Adjust maxFiles if needed Q: Why are some files skipped? A: Files are skipped if they: 1.Are too large (>5MB) 2.Can't be decoded (binary files) 3.Cause encoding errors Q: How long does extraction take? A: 1.Small repos (<100 files): 2-5 seconds 2.Medium repos (100-500 files): 10-30 seconds 3.Large repos (500+ files): 30-60 seconds 4.Max timeout: 4 minutes --- Built with ❤️ for the Apify $1M Challenge ⭐ If you find this useful, please star the Actor!