Patch.com News Scraper A robust web scraper built with Apify SDK and Playwright to extract news articles from patch.com. This actor is designed to crawl patch.com and extract comprehensive article data including titles, authors, publish dates, content, and images. ## Features - Comprehensive Data Extraction: Extracts article titles, authors, publish dates, content, and images - Robust Error Handling: Continues scraping even if individual pages fail - Proxy Support: Built-in proxy configuration for reliable scraping - Cloud Deployment Ready: Configured for Apify cloud platform - Flexible Input Configuration: Supports custom start URLs ## ⚠️ Important Notes 1. Respect Patch.com's Terms of Service - Use this Actor responsibly and in accordance with Patch.com's policies 2. Rate Limiting - The Actor includes built-in delays to avoid overwhelming Patch.com's servers 3. Proxy Usage - For large-scale scraping, always use residential proxies 4. Data Usage - Ensure you have permission to use scraped data for your intended purpose 5. Public Articles Only - The Actor can only scrape publicly accessible Patch.com articles ## Extracted Data Fields - url: The source URL of the article - title: Article headline - author: Article author name - publishDate: Publication date (ISO format when available) - content: Article content (truncated to 2000 characters) - imageUrl: Featured image URL - isArticle: Boolean indicating if the page is a news article - scrapedAt: Timestamp when the article was scraped ## Input Configuration The actor accepts the following input parameters: json { "startUrls": [ { "url": "https://patch.com/new-york/across-ny" } ] } ### Input Parameters - startUrls (array, optional): Array of objects with a url property to start crawling from. Default: [{"url": "https://patch.com/new-york/across-ny"}] ## Output Schema The actor outputs data in the following JSON format: json { "url": "https://patch.com/new-york/across-ny/article-slug", "title": "Article Title", "author": "Author Name", "publishDate": "2025-07-14T10:30:00.000Z", "content": "Article content text (truncated to 2000 characters)...", "imageUrl": "https://patch.com/img/cdn20/.../image.jpg", "isArticle": true, "scrapedAt": "2025-07-14T17:46:49.097Z" } ### Output Fields - url (string): The source URL of the article - title (string): Article headline/title - author (string): Article author name (may be empty if not found) - publishDate (string): Publication date in ISO format (may be empty if not found) - content (string): Article content text, truncated to 2000 characters - imageUrl (string): Featured image URL (may be empty if not found) - isArticle (boolean): Indicates if the page is a valid news article - scrapedAt (string): Timestamp when the article was scraped (ISO format) ## Usage ### Local Development 1. Install dependencies: bash npm install 2. Run locally: bash npm start 3. Format code: bash npm run format 4. Lint code: bash npm run lint npm run lint:fix ### Apify Cloud Deployment 1. Push to Apify: bash npm run push 2. Run on Apify Cloud: bash npm run agent:run 3. Check logs: bash npm run agent:log 4. Pull latest changes: bash npm run pull ## Development Workflow 1. Local Testing: Test changes locally with npm start 2. Code Quality: Run npm run lint and npm run format before committing 3. Cloud Testing: Push changes with npm run push and test on Apify 4. Monitor Logs: Use npm run agent:log to check for errors 5. Iterate: Fix issues and repeat the cycle ## Troubleshooting ### Common Issues 1. Rate Limiting: If you encounter rate limiting, ensure proxy is properly configured 2. Page Load Failures: The scraper waits for network idle state, but some pages may still fail 3. Data Extraction Issues: Check the page structure if data extraction is incomplete ### Debugging - Check logs with npm run agent:log - Run locally with npm start for detailed console output - Review the extracted dataset in Apify console ## License ISC License ## Author It's not you it's me