RAG Knowledge Loader Scrapes documentation sites (GitBook, ReadTheDocs, Notion public pages) and converts them into vector-ready JSON format for RAG applications. ## Features - Crawls entire documentation sites recursively - Extracts clean, structured content - Removes navigation, headers, footers automatically - Outputs vector-ready JSON format - Supports GitBook, ReadTheDocs, Notion, and custom doc sites ## Use Cases - Build "Chat with Docs" chatbots - Feed LLMs with up-to-date documentation - Create knowledge bases for RAG pipelines - Automated documentation updates for vector databases ## Input Parameters ### Required - Start URLs (required): Array of documentation site URLs to scrape - Example: https://docs.apify.com/, https://your-gitbook-site.com ### Optional Configuration - Max pages to crawl (default: 1000): Maximum number of pages to scrape - Minimum: 1 - Include URL patterns (globs) (default: []): Only crawl URLs matching these patterns - Example: ["**/api/**", "**/guides/**"] - Exclude URL patterns (globs) (default: ["**/*.pdf", "**/*.zip", "**/login**", "**/signup**"]): Skip URLs matching these patterns - Content CSS Selectors (default: "article, main, .content, .markdown-body, #content, [role='main']"): Comma-separated CSS selectors for main content area - Remove CSS Selectors (default: "nav, header, footer, .sidebar, #sidebar, .navigation, .cookie-banner, script, style, iframe"): Selectors for elements to remove like navigation and headers - Output Format (default: "vector-ready"): - "vector-ready": Flat structure optimized for embeddings - "hierarchical": Nested structure with full metadata - Crawler Type (default: "cheerio"): - "cheerio": Fast HTTP crawler for static sites - "playwright": Browser-based crawler for JavaScript-heavy sites ### Example Input JSON { "startUrls": [ { "url": "https://docs.example.com/" }, { "url": "https://your-gitbook.com/docs" } ], "maxPages": 500, "excludeUrlGlobs": ["/.pdf", "/login", "/signup"], "includeUrlGlobs": ["/docs/"], "contentSelectors": "article, main, .markdown-body", "removeSelectors": "nav, footer, .sidebar", "outputFormat": "vector-ready", "crawlerType": "cheerio" } ### Minimal Input Example { "startUrls": [ { "url": "https://docs.example.com/" } ] } ## Output Format ### Vector-Ready Format (Default) Optimized for direct ingestion into vector databases: { "metadata": { "crawledAt": "2025-12-06T08:11:00.000Z", "totalPages": 150, "startUrls": ["https://docs.example.com/"], "readyForEmbedding": true }, "documents": [ { "id": "unique-doc-id-123", "text": "Full page content with all text extracted and cleaned...", "metadata": { "source": "https://docs.example.com/page", "title": "Page Title", "url": "https://docs.example.com/page", "scrapedAt": "2025-12-06T08:11:00.000Z", "wordCount": 1234 } } ] } ### Hierarchical Format Includes full document structure with headings and metadata: { "metadata": { "crawledAt": "2025-12-06T08:11:00.000Z", "totalPages": 150, "startUrls": ["https://docs.example.com/"] }, "documents": [ { "id": "unique-doc-id-123", "url": "https://docs.example.com/page", "title": "Page Title", "content": "Full page content...", "metadata": { "description": "Page meta description", "keywords": "api, documentation", "scrapedAt": "2025-12-06T08:11:00.000Z", "headings": [ { "level": 1, "text": "Introduction" }, { "level": 2, "text": "Getting Started" } ], "wordCount": 1234, "characterCount": 5678 } } ] } ## Integration with Vector Databases The output is ready to use with popular RAG frameworks: - LangChain: Use JSONLoader to load documents - LlamaIndex: Import as Document objects - Pinecone/Weaviate: Batch upsert with metadata - Chroma*: Add to collection with embeddings