Apify Smart Dataset Comparator

Compare 2-10 Apify datasets to detect changes, find new or removed records, deduplicate data, and merge results using custom rules. Useful for price monitoring, lead list deduplication, SEO tracking, and data quality validation.

Facing an issue, unexpected error, edge case, or have a feature suggestion? Post it here and we'll address it within 24 hours.

Overview

This actor analyzes multiple datasets, identifies differences, and provides a structured output of changes. It works by matching records across datasets using a primary key you define.

Key Features

• Change Detection: Compares records by primary key and shows field-level before/after diffs.
• Record Tracking: Identifies records that are new, removed, or unchanged.
• Data Merging: Combines unique records using a chosen merge strategy (e.g., newest dataset wins, average numbers).
• Duplicate Detection: Finds exact duplicates and can use fuzzy matching (Levenshtein distance) for similar records.
• Schema Validation: Checks for field consistency, type conflicts, and missing fields across datasets.
• Smart Presets: Pre-configured settings for common use cases:
  • price_monitoring: For e-commerce; includes price tolerance and ignores timestamps.
  • lead_lists: For CRM; normalizes contact info and enables fuzzy deduplication.
  • seo: For content tracking; uses strict comparison and URL normalization.
  • real_estate: For property listings; applies tight price tolerance and phone normalization.
• Data Cleaning: Automatically normalizes emails, phone numbers, URLs (removes tracking parameters), and currency values before comparison.

How to Use

Provide the dataset IDs and a primary key field. The actor will process and return the comparison results.

Basic Input Example:

{
  "datasetIds": ["DATASET_ID_1", "DATASET_ID_2"],
  "primaryKey": "url"
}

Merge Strategies (when the same record exists in multiple datasets):
* left_priority: First dataset wins (default).
* right_priority: Last/newest dataset wins.
* most_recent: Record with the newest timestamp wins.
* most_complete: Record with the most filled fields wins.
* combine_arrays: Merge array fields from all records.
* average_numbers: Average numeric fields.

Input Parameters

Required

Optional

Output

The actor generates a dataset containing the following views:

• Changes: Records that were modified, with a detailed diff of altered fields.
• New Records: Records present only in the newer dataset(s).
• Removed Records: Records present only in the older dataset(s).
• Merged Data: A consolidated dataset of all unique records, merged according to your chosen strategy.
• Duplicates: A report of exact and fuzzy duplicate records found within the input datasets.
• Schema Analysis: A summary of field types, conflicts, and consistency across datasets.
• Anomalies: Flags significant changes, like price swings >50% or stock depletions.