JSON & Visual Content Monitor Monitor JSON endpoints and web pages for changes with automatic snapshots, AI-powered summaries, email notifications, and multi-format exports. Perfect for tracking API changes, pricing updates, and visual website modifications. ## Why Use This Actor? Stop manually checking websites and APIs for changes. This Actor automatically monitors your endpoints and pages, detects any modifications, and notifies you instantly. Whether you're tracking competitor prices, monitoring API schemas, or watching for policy changes, this Actor has you covered. ## Features - 🔍 Dual Monitoring Modes - Monitor JSON data, visual screenshots, or both simultaneously - 📊 Detailed Diffs - Get path-level changes with old and new values for precise tracking - 📸 Visual Screenshots - Capture before/after screenshots automatically with hash-based comparison - 📄 HTML Diff Reports - Beautiful HTML reports showing before/after JSON with highlighted changes - 🤖 AI-Powered Summaries - Use OpenAI or Anthropic to generate human-readable change summaries - 📧 Multiple Notification Channels - Email (Apify/SendGrid/Mailgun), Slack, and custom webhooks - 💾 Smart Snapshots - Automatically stores and compares against previous versions - 📤 Multi-Format Export - Export to Dataset, CSV, JSON, JSONL, SQLite, PostgreSQL, MySQL, MongoDB, Google Sheets - 🔑 Entity Tracking - Track changes in arrays using primary keys (e.g., product IDs) - 🚨 Error Handling - Graceful handling of network errors with automatic retries - ⚙️ Highly Configurable - Control depth, ignore paths, viewport size, headers, and more ## Use Cases ### E-commerce & Retail - Price Monitoring - Track competitor pricing across multiple stores - Inventory Tracking - Monitor stock levels and availability - Product Changes - Detect when product descriptions or specifications change ### SaaS & Technology - API Monitoring - Track REST API changes and schema updates - Feature Flags - Monitor configuration and feature flag changes - Breaking Changes - Get alerted to API breaking changes before they affect your app ### Compliance & Legal - Terms of Service - Monitor changes to terms, policies, and legal documents - Regulatory Compliance - Track updates to compliance requirements - Privacy Policies - Stay informed about privacy policy modifications ### Finance & Trading - Exchange Rates - Monitor currency exchange rate changes - Stock Prices - Track stock price movements - Crypto Prices - Monitor cryptocurrency price changes ## Input ### Required Fields - targetUrls (array) - List of URLs to monitor (JSON APIs or web pages) json ["https://api.example.com/products", "https://example.com/pricing"] ### Optional Fields - monitoringMode (string) - Choose monitoring type: - json - Monitor JSON data only (default) - visual - Capture visual screenshots only - both - Monitor both JSON and visual changes - requestHeaders (object) - HTTP headers for API requests json { "Authorization": "Bearer YOUR_TOKEN", "Accept": "application/json" } - primaryKeyPath (string) - JSONPath for primary key in array responses json "$[*].id" - maxDepth (integer) - Maximum depth for JSON tree traversal (default: 5) - ignorePaths (array) - JSON paths to ignore in diffs json ["$.timestamp", "$.lastModified", "$[*].updatedAt"] - notificationEmail (string) - Simple email notification address (uses Apify's built-in email) json "admin@example.com" - emailNotifications (object) - Advanced email configuration (SendGrid/Mailgun) json { "enabled": true, "recipients": ["admin@example.com"], "provider": "sendgrid", "apiKey": "SG.xxx", "fromEmail": "alerts@example.com" } - slackWebhookUrl (string) - Slack Incoming Webhook URL for notifications json "https://hooks.slack.com/services/YOUR/WEBHOOK/URL" - webhookUrl (string) - Generic webhook URL to POST change data json "https://your-server.com/webhook" - enableAiSummary (boolean) - Generate AI summaries of changes (default: false) - aiApiKey (string, secret) - API key for OpenAI or Anthropic - aiProvider (string) - AI provider: "openai" or "anthropic" (default: "openai") - exportFormats (array) - Export destinations json [ {"type": "dataset"}, {"type": "csv"}, {"type": "json"}, {"type": "jsonl"}, {"type": "sqlite"} ] - screenshotOptions (object) - Screenshot configuration json { "fullPage": true, "viewportWidth": 1920, "viewportHeight": 1080, "waitForSelector": "#content" } ## Output ### Dataset Records Each detected change is saved as a structured record: json { "url": "https://api.example.com/products", "detectedAt": "2025-01-15T10:30:00.000Z", "changeType": "modified", "path": "$.items[123].price", "oldValue": 29.99, "newValue": 24.99, "meta": { "reason": "field_changed", "primaryKey": "123" } } Change Types: - added - New field or entity added - removed - Field or entity removed - modified - Existing value changed - error - Fetch or parsing error occurred ### Key-Value Store For each monitored URL, the Actor stores: 1. Snapshots (SNAPSHOT-<url-slug>) - Latest JSON response for comparison 2. Screenshots (SCREENSHOT-<url-slug>) - Latest visual screenshot (PNG) 3. Screenshot Hashes (SCREENSHOT-HASH-<url-slug>) - MD5 hash for comparison 4. Summaries (SUMMARY-<url-slug>) - Human-readable change summary 5. HTML Diff Reports (DIFF-REPORT-<url-slug>-<timestamp>.html) - Beautiful HTML diff visualization 6. Report Screenshots (DIFF-REPORT-<url-slug>-<timestamp>.png) - Screenshot of HTML report 7. Exports (changes-<timestamp>.csv/json/jsonl) - Exported change data 8. SQLite Database (changes-<runId>.sqlite) - SQLite database with all changes (if enabled) ### Dataset Views The Actor provides multiple views of your data: - All Changes - Complete change history - Added Items - Only new additions - Removed Items - Only deletions - Modified Items - Only modifications - Errors - Failed requests and errors ## Examples ### Example 1: Monitor JSON API json { "targetUrls": [ "https://api.github.com/repos/apify/apify-sdk-js" ], "monitoringMode": "json", "maxDepth": 5 } ### Example 2: Track Product Prices json { "targetUrls": [ "https://api.example.com/products" ], "primaryKeyPath": "$[*].id", "ignorePaths": ["$[*].lastModified"], "notificationEmail": "team@example.com" } ### Example 3: Visual Website Monitoring json { "targetUrls": [ "https://competitor.com/pricing" ], "monitoringMode": "visual", "screenshotOptions": { "fullPage": true, "viewportWidth": 1920, "viewportHeight": 1080 } } ### Example 4: Monitor with AI Summary and Notifications json { "targetUrls": [ "https://api.example.com/v1/products" ], "enableAiSummary": true, "aiApiKey": "sk-...", "aiProvider": "openai", "notificationEmail": "admin@example.com", "slackWebhookUrl": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL", "exportFormats": [ {"type": "dataset"}, {"type": "jsonl"}, {"type": "sqlite"} ] } ### Example 5: Track Array Changes json { "targetUrls": [ "https://jsonplaceholder.typicode.com/users" ], "primaryKeyPath": "$[*].id", "maxDepth": 3, "ignorePaths": ["$[*].company.catchPhrase"] } ## How It Works 1. First Run - Fetches data/screenshots and stores as baseline snapshot 2. Subsequent Runs - Compares new data against previous snapshot 3. Diff Generation - Produces detailed change records with paths and values 4. Notification - Sends email alerts if changes detected (optional) 5. Export - Saves changes to configured formats (Dataset, CSV, JSON, etc.) 6. Storage Update - Updates snapshot for next comparison ## Scheduling For continuous monitoring, schedule the Actor to run periodically: - Every hour - For frequently changing data - Every 6 hours - For moderate monitoring - Daily - For slow-changing content - Weekly - For rarely updated pages Set up schedules in the Apify Console under the "Schedules" tab. ## Integration ### Webhooks Connect to other services when changes are detected: - Slack notifications - Discord alerts - Custom webhooks - Zapier/Make integrations ### API Access Access your data programmatically: javascript const client = new ApifyClient({ token: 'YOUR_TOKEN' }); const dataset = await client.dataset('DATASET_ID').listItems(); ### Database Export Export directly to databases: json { "exportFormats": [ { "type": "postgresql", "config": { "connectionString": "postgresql://user:pass@host:5432/db" } } ] } ## Running Locally 1. Install dependencies: bash npm install 2. Create .actor/INPUT.json with your configuration 3. Run the actor: bash apify run 4. View results in ./storage/datasets/default/ ## Deploying to Apify 1. Login to Apify CLI: bash apify login 2. Push to Apify platform: bash apify push ## Limitations - Maximum JSON depth is configurable (default: 5 levels) - Very large JSON responses (>10MB) may impact performance - Screenshot comparison uses MD5 hashing (pixel-perfect comparison not available) - AI summaries require OpenAI or Anthropic API key (paid service) - SQLite export requires better-sqlite3 package (framework ready) - Database exports (PostgreSQL, MySQL, MongoDB) require additional packages - Rate limiting applies based on your Apify plan ## Error Handling - 5xx Server Errors - Automatic retry with exponential backoff (3 attempts) - Invalid JSON - Logged as error record, previous snapshot preserved - Network Failures - Retried up to 3 times before marking as error - Non-JSON Content - Detected and logged with content-type information - Screenshot Failures - Logged as error, JSON monitoring continues (if in "both" mode) ## Performance Tips 1. Use ignorePaths - Skip dynamic fields like timestamps to reduce noise 2. Set appropriate maxDepth - Limit depth for large nested objects 3. Use primaryKeyPath - For efficient array diffing with large datasets 4. Schedule wisely - Don't over-monitor; match frequency to change rate 5. Filter exports - Only export to formats you need ## Privacy & Security - API keys and tokens are stored securely - Screenshots are stored in your private Key-Value Store - Data is never shared with third parties - Use Apify's secret input fields for sensitive data - All data is encrypted at rest and in transit ## Version History ### 0.0 (Current) - Initial release - JSON monitoring with detailed diffs - Visual screenshot monitoring - Email notifications (Apify built-in, SendGrid, Mailgun) - Slack and webhook notifications - AI-powered summaries (OpenAI, Anthropic) - HTML diff reports with screenshots - Multi-format export (Dataset, CSV, JSON, JSONL, SQLite) - Database export support (PostgreSQL, MySQL, MongoDB, Google Sheets) - Entity tracking with primary keys - Configurable depth and ignore paths - Automatic retries and error handling ## License Apache-2.0 ## Author Built with ❤️ for the Apify community --- Ready to start monitoring? Click "Try for free" to run your first monitoring job!