Google Review Scraper A highly optimized Node.js application designed to scrape Google reviews with Playwright, built for compatibility with the Apify platform and Docker deployment. Input-driven with no database dependencies. ## Features - 🚀 High Performance: Batch processing with single browser instance - ⚡ Speed Optimized: Minimal delays (1-2s navigation, 0.1-0.8s scrolling) - 🕸️ Stealth Mode: Advanced anti-detection techniques - 🐳 Docker Ready: Full containerization support - 🔄 Apify Compatible: Ready for Apify platform deployment - 📥 Input Driven: Simply provide place IDs as input - 📄 JSON Output: Results saved as structured JSON files - 🎯 Precise Parsing: Extracts author, rating, date, and review text - 🔀 Human-like Behavior: Randomized actions with minimal delays ## Performance Optimizations This scraper is highly optimized for speed while maintaining reliability: - Reduced Delays: Navigation delays reduced to 1-2 seconds, scroll delays to 0.3-0.8 seconds - Configurable Scrolling: Default 5 scrolls (vs 8-15), customizable via maxScrolls input - Single Browser Instance: Reuses browser context across multiple place IDs - Minimal Anti-Detection: Just enough randomization to avoid detection without sacrificing speed - Fast Parsing: Uses Cheerio for rapid HTML parsing - Batch Processing: Processes multiple place IDs in a single session ## Prerequisites - Node.js (version 18.0.0 or higher) - Chrome browser (for local development) - Docker & Docker Compose (for containerized deployment) ## Quick Start ### Local Development 1. Clone and setup: bash git clone https://github.com/MuhammadShahzeb123/google-review-scraper-2.git cd google-review-scraper npm install 2. Install Playwright browsers: bash npx playwright install chromium 3. Run with place IDs: bash # Method 1: Command line arguments node src/main.js ChIJN1t_tDeuEmsRUsoyG83frY4 ChIJGVtI4by3t4kRr51d_Qm_x58 # Method 2: Environment variable PLACE_IDS="ChIJN1t_tDeuEmsRUsoyG83frY4,ChIJGVtI4by3t4kRr51d_Qm_x58" npm start # Method 3: Run example node example.js ### Docker Deployment 1. Using Docker Compose: bash PLACE_IDS="ChIJN1t_tDeuEmsRUsoyG83frY4,ChIJGVtI4by3t4kRr51d_Qm_x58" docker-compose up 2. Using Docker only: bash docker build -t google-review-scraper . docker run --rm \ -e PLACE_IDS="ChIJN1t_tDeuEmsRUsoyG83frY4,ChIJGVtI4by3t4kRr51d_Qm_x58" \ -v $(pwd)/output:/app/output \ google-review-scraper ### Apify Platform 1. Configure via Apify Input UI: All settings are now configurable through the Apify input interface: - Place IDs: List of Google Place IDs to scrape - Max Scrolls: Number of scroll actions (1-20, default: 5) - Headless Mode: Run browser in background (default: true) - Browser Timeout: Page load timeout in ms (default: 60000) - Cleanup HTML: Delete temp files after processing (default: true) - Log Level: Logging verbosity (error/warn/info/debug) 2. Push to Apify: bash apify login apify push 3. Run on Apify: bash apify run ## Configuration ### Apify Platform (Recommended) All configuration is done through the Apify Input UI - no environment variables needed! ### Local/Standalone Environment Variables (Legacy) | Variable | Description | Default | |----------|-------------|---------| | PLACE_IDS | Comma-separated place IDs | Example IDs | | HEADLESS_MODE | Browser headless mode | false | | BROWSER_TIMEOUT | Page timeout (ms) | 60000 | | SCROLL_COUNT_MIN | Min scroll actions | 8 | | SCROLL_COUNT_MAX | Max scroll actions | 15 | ### Input Methods 1. Command Line Arguments: bash node src/main.js PLACE_ID_1 PLACE_ID_2 PLACE_ID_3 2. Environment Variable: bash export PLACE_IDS="ChIJN1t_tDeuEmsRUsoyG83frY4,ChIJGVtI4by3t4kRr51d_Qm_x58" npm start 3. Programmatic Usage: javascript const { scrapeReviews } = require('./src/main'); const placeIds = ['ChIJN1t_tDeuEmsRUsoyG83frY4']; const reviews = await scrapeReviews(placeIds); 4. Apify Input: json { "placeIds": [ "ChIJN1t_tDeuEmsRUsoyG83frY4", "ChIJGVtI4by3t4kRr51d_Qm_x58" ], "headlessMode": true, "scrollCount": 10 } ## Output Format ### For Apify Platform: Each review is stored as a separate dataset item for optimal Apify integration: json { "reviewerName": "John Doe", "rating": 5, "reviewText": "Great place, highly recommended!", "reviewDate": "2 months ago", "placeId": "ChIJN1t_tDeuEmsRUsoyG83frY4", "scrapedAt": "2025-06-16T12:00:00.000Z", "success": true } Summary statistics are saved in the OUTPUT key-value store: json { "totalPlaceIds": 2, "totalReviews": 45, "successfulScrapes": 2, "failedScrapes": 0, "maxScrollsUsed": 5, "processedAt": "2025-06-16T12:00:00.000Z", "placeIds": ["ChIJN1t_tDeuEmsRUsoyG83frY4"], "scrapingResults": {"ChIJN1t_tDeuEmsRUsoyG83frY4": true} } ### For Local/Standalone Usage: Results are saved as JSON files with nested structure: json { "summary": { "totalPlaceIds": 2, "totalReviews": 45, "successfulScrapes": 2, "failedScrapes": 0, "processedAt": "2025-06-16T12:00:00.000Z", "placeIds": ["ChIJN1t_tDeuEmsRUsoyG83frY4"] }, "reviews": [ { "author": "John Doe", "stars": 5, "date": "2 months ago", "text": "Great place, highly recommended!", "placeId": "ChIJN1t_tDeuEmsRUsoyG83frY4", "scrapedAt": "2025-06-16T12:00:00.000Z", "success": true } ] } ## Project Structure ├── src/ │ ├── main.js # Main application logic │ ├── scraper.js # Playwright browser automation │ ├── google_reviews_parser.js # HTML parsing with Cheerio │ ├── config.js # Configuration management │ └── apify-main.js # Apify-compatible version ├── example.js # Usage example ├── Dockerfile # Docker container configuration ├── docker-compose.yml # Container orchestration ├── package.json # Node.js dependencies ├── INPUT_SCHEMA.json # Apify input schema └── .env.example # Environment template ## How It Works 1. Input Processing: Accepts place IDs from various sources 2. Browser Automation: Launches Chrome with stealth settings 3. Navigation: Visits Google Maps place pages for each place ID 4. Review Loading: Clicks Reviews tab, sorts by newest, scrolls to load more 5. HTML Extraction: Saves review container HTML to temporary files 6. Parsing: Uses Cheerio to extract structured review data 7. Output: Saves results as JSON and cleans up temporary files ## Advanced Features ### Stealth Mode - Custom user agent and viewport - Disabled automation flags - Random delays between actions - Human-like scrolling patterns ### Error Handling - Graceful failure handling per place ID - Screenshot capture on errors - Comprehensive logging - Automatic retry mechanisms ### Performance Optimization - Single browser instance for all place IDs - Parallel file processing - Efficient memory management - Automatic cleanup of temporary files ## API Reference ### scrapeReviews(placeIds) Programmatically scrape reviews for given place IDs. Parameters: - placeIds (Array): Array of Google Place IDs Returns: - Promise<Array>: Array of review objects with metadata Example: javascript const { scrapeReviews } = require('./src/main'); const reviews = await scrapeReviews([ 'ChIJN1t_tDeuEmsRUsoyG83frY4' ]); console.log(`Found ${reviews.length} reviews`); ## Troubleshooting ### Common Issues 1. Browser not found: bash npx playwright install chromium 2. No place IDs provided: - Ensure place IDs are valid Google Place IDs - Check input format (comma-separated for env vars) 3. Permission errors in Docker: - Application runs as non-root user for security - Check file permissions when mounting volumes ### Finding Google Place IDs 1. Search for a business on Google Maps 2. Look at the URL: ...place/.../@... or use the share link 3. Extract the place ID from the URL or use Google Places API ## Contributing 1. Fork the repository 2. Create a feature branch 3. Make your changes 4. Test thoroughly 5. Submit a pull request ## License ISC License - see LICENSE file for details.