LLM-Ready Documentation Scraper Crawl any documentation website and get clean, formatted Markdown perfect for LLMs and RAG (Retrieval-Augmented Generation) applications. ## 🎯 Problem Developers and AI agents need to read documentation (Stripe Docs, Next.js Docs, etc.), but standard scrapers return messy HTML with navbars, footers, and ads. This Actor solves that by delivering pure, clean Markdown. ## ✨ Features - Clean Markdown Output: Strips navigation, sidebars, footers, scripts, and ads - Smart Content Detection: Automatically finds the main content area - Token Counting: Each page includes token count for LLM context planning - Merge Mode: Combine all pages into a single full_documentation.md file - Configurable Depth: Control how deep to crawl - URL Filtering: Include/exclude patterns using globs ## 📥 Input | Field | Type | Description | |-------|------|-------------| | startUrl | String | The root URL of the documentation site | | maxDepth | Number | Maximum link depth to crawl (default: 10) | | maxPages | Number | Maximum pages to scrape (default: 100) | | includeGlobs | Array | URL patterns to include | | excludeGlobs | Array | URL patterns to exclude | | excludeElements | String | CSS selectors to remove | | contentSelector | String | CSS selector for main content | | mergeOutput | Boolean | Combine all pages into one file | ## 📤 Output Each page is stored in the dataset with: json { "url": "https://docs.example.com/api/auth", "title": "Authentication", "markdown": "# Authentication\n\nThis guide covers...", "tokenCount": 1523, "scrapedAt": "2024-01-15T10:30:00.000Z" } When mergeOutput is enabled, a combined full_documentation.md is saved to the Key-Value Store. ## 🚀 Usage Examples ### Crawl Stripe Docs json { "startUrl": "https://stripe.com/docs/api", "maxPages": 50, "mergeOutput": true } ### Crawl with Custom Content Selector json { "startUrl": "https://nextjs.org/docs", "contentSelector": ".docs-content", "excludeElements": "nav, footer, .sidebar, .carbon-ads", "maxDepth": 3 } ## 🔧 Technical Details - Built with TypeScript and the Apify SDK - Uses CheerioCrawler for fast HTML parsing - Turndown library for HTML-to-Markdown conversion - gpt-tokenizer for accurate token counting ## 📝 License ISC