Diff Datasets

Overview

An Apify actor that compares two datasets on the Apify platform. It identifies items present in one dataset but missing from another, and can also detect changed items using a compound key. Useful for data synchronization, change detection, and deduplication workflows.

Key Features

• Dataset Comparison: Compares a base dataset against an other dataset.
• Flexible Key Definition: Uses uniqueFields to create a compound key for comparison. Supports:
  • Simple primitive fields (string, number, boolean).
  • Nested object paths using lodash.get notation (e.g., sub.fields.0).
  • Array or string lengths (e.g., sub.fields.length).
• Handles Nested Objects: Automatically hashes nested object values for comparison using JSON.stringify.
• Output: Returns only the items from the other dataset that are not found in the base dataset.

How to Use

Call the actor via the Apify API, providing the IDs of your datasets and the fields to use for comparison.

await Apify.call('pocesar/diff-datasets', {
    baseDatasetId: 'LdNAlaOY1aKGhwAah', // The reference dataset
    otherDatasetId: 'Bzu1pgOjenN43VhPY', // The dataset to check for new/changed items
    uniqueFields: [ // Defines the compound key
        "pageUrl",
        "sub.fields.0",
        "sub.fields.length"
    ],
});

Important: The order of baseDatasetId and otherDatasetId matters. Items already present in the base dataset will not appear in the output; only unique items from the other dataset are returned.

Input/Output

Input Parameters:
* baseDatasetId (String, required): The ID of the reference dataset.
* otherDatasetId (String, required): The ID of the dataset to compare against the base.
* uniqueFields (Array, required): An array of field paths that together form a unique key for each item.

Output:
The actor outputs a new dataset containing items from the other dataset that were not found in the base dataset, based on the defined uniqueFields.

Limitations

• Memory Usage: All unique key values from the base dataset are held in memory. Larger datasets require more memory.
• Storage Limits: Saving the internal data structure to the key-value store may fail if the set of unique keys becomes extremely large.

License

Apache 2.0