Scraper Results Checker

Overview

This actor automatically checks the results from any Apify actor (or other process) that stores data to a dataset. It validates the output for errors and can send notifications based on the findings. It's designed to be triggered via webhook after a scraper run completes.

Key Features

• Error Validation: Checks dataset results for common issues.
• Webhook Integration: Primarily designed to run automatically from an actor/task webhook.
• Flexible Checks: Supports validation against a JSON schema, comparison with a previous run, and minimum item count thresholds.
• Notifications: Can send email alerts when errors are found (or when a run is successful).
• Chained Actions: Can trigger other actors (e.g., send an email) automatically based on the check result (success or error).

How to Use

1. Actor/Task Webhook (Recommended)

Set up a webhook for your actor or task with the following URL and payload template.

Webhook URL:

https://api.apify.com/v2/acts/drobnikj~check-crawler-results/runs?token=APIFY_API_TOKEN

Payload Template:

{
  "actId": "{{resource.actId}}",
  "runId": "{{resource.id}}",
  "options": {
    "notifyTo": "your-email@example.com",
    "minOutputtedPages": 10
  }
}

2. Direct API Call

You can also call it programmatically from another actor using Apify.call().

await Apify.call('drobnikj/check-crawler-results', {
  actId: 'TARGET_ACTOR_ID',
  runId: 'TARGET_RUN_ID',
  options: {
    minOutputtedPages: 1000,
  }
});

Input

The actor requires an input object with the following fields:

• actId (String): The ID of the actor whose results you want to check.
• runId (String): The specific run ID of the actor to check.
• datasetId (String, Optional): A specific dataset ID to check. If not provided, the actor uses the dataset from the specified run.
• options (Object, Optional): Configuration for the check.
  • sampleCount (Number): Number of dataset items to sample (default: 100000).
  • minOutputtedPages (Number): Minimum number of items required; triggers an error if not met.
  • jsonSchema (Object): A JSON schema to validate all sampled results against.
  • compareWithPreviousExecution (Boolean): Compares results with the previous execution (legacy crawler only).
  • notifyTo (String): Email address to send error notifications to.
  • runActOnSuccess (Object): Defines an actor to run if no errors are found (e.g., a success notification).
    json { "id": "apify/send-mail", "input": { ... } }
  • runActOnError (Object): Defines an actor to run if errors are found (e.g., an alert notification). Uses the same format as runActOnSuccess.

Output

The actor's output contains an errors array. If the array is empty, the check passed. Any validation failures are listed as strings in this array.

Example Output:

{
  "errors": [
    "Run is not in SUCCEEDED status, act status: ABORTED",
    "Crawler returns only 0 outputted pages and minimum is 100"
  ]
}