Intelligent Website Scraper An Apify actor that uses LangChain and LLM to intelligently scrape and process website content based on high-level goals. ## Features - Universal Website Scraping: Works on any website, not limited to specific platforms - Intelligent Content Processing: Uses LangChain + OpenAI to extract and summarize content - Multiple Task Types: Support for summarization, product extraction, service extraction, and FAQ extraction - Configurable Crawling: Optional internal link following with depth control - Clean Content Extraction: Removes scripts, styles, and irrelevant content ## Input The actor accepts the following input format: json { "startUrls": [ { "url": "https://example.com" } ], "taskType": "extractServices", "maxDepth": 1, "followInternalLinks": false } ### Input Parameters | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | startUrls | Array | Yes | - | Array of objects with url property | | taskType | String | No | summarize | Type of content processing task | | maxDepth | Number | No | 1 | Maximum depth for internal link following | | followInternalLinks | Boolean | No | false | Whether to follow internal links | ### Supported Task Types | Task Type | Description | |-----------|-------------| | summarize | Summarize entire site content | | extractProducts | Identify and extract product-related sections | | extractServices | Extract service listings or offerings | | extractFAQs | Pull FAQ-like content from the page | ## Output The actor outputs structured data for each processed URL: json { "url": "https://example.com", "title": "Example Website", "taskType": "extractServices", "processedContent": "AI-processed content based on task type...", "rawContent": "First 1000 characters of raw content...", "scrapedAt": "2024-01-01T00:00:00.000Z", "metadata": { "wordCount": 1500, "linksFound": 25, "imagesFound": 10 } } ## Environment Variables | Variable | Required | Description | |----------|----------|-------------| | OPENAI_API_KEY | Yes | Your OpenAI API key for LangChain integration | ## Example Usage ### Basic Summarization json { "startUrls": [ { "url": "https://example.com" } ], "taskType": "summarize" } ### Extract Services with Internal Link Following json { "startUrls": [ { "url": "https://example.com" } ], "taskType": "extractServices", "followInternalLinks": true, "maxDepth": 2 } ### Extract Products from Multiple URLs json { "startUrls": [ { "url": "https://shop1.com" }, { "url": "https://shop2.com" } ], "taskType": "extractProducts" } ## How It Works 1. Content Extraction: Uses Puppeteer to load pages and Cheerio to extract clean content 2. Intelligent Processing: LangChain processes content based on the specified task type 3. Structured Output: Returns processed content with metadata and original URL 4. Optional Crawling: Can follow internal links to gather more comprehensive data ## Installation 1. Clone this repository 2. Install dependencies: npm install 3. Set your OPENAI_API_KEY environment variable 4. Run the actor: npm start ## Development - npm start - Run the actor - npm run format - Format code with Prettier - npm run lint - Run ESLint - npm run lint:fix - Fix ESLint issues ## Architecture - src/main.js - Main entry point and input validation - src/routes.js - Request routing - src/handlers/websiteScraper.js - Main scraping logic - src/services/langchainService.js - LangChain integration and task processing - src/puppeteerLauncher.js - Puppeteer browser configuration