Dataset Toolbox ## Input | Field | Type | Optional | Description | | ----- | ---- | -------- | ----------- | |Actor IDs|Array|depends|Load latest default datasets |Dataset IDs|Array|depends|Load the specified datasets Access 3rd party datasets Define secret environment variable to access other users' datasets: CUSTOM_SOURCE_APIFY_TOKEN ## Features - Dataset unification - merge datasets in uniform output - limit number of items extracted - Latest dataset detection - detect and use the latest datasets - Output fields management - filter out fields from source data - output data in custom field order - Dataset post-processing - apply custom function to each item - Output schema validation - validate records against schema - capture invalid items for recycling ### Dataset unification Use features described below to produce a single uniform dataset from datasets sharing a single common output schema and expected output structure ### Latest dataset detection Automatically detects and uses default datasets of the latest actor runs when: - Actor ID(s) are specified AND - Dataset ID(s) are not specified ### Output fields management Produces a download link for obtaining the resulting dataset with top level fields sorted and filtered based on the list of fields provided on input. This link is stored in default KV store: DATASET_DOWNLOAD-CUSTOM_FIELD_ORDER-{selected file type} Filter fields Filter and pick only certain fields from source dataset(s) Order fields Apply custom order to top level fields in custom order instead of alphabetical (default) ### Dataset post-processing Apply custom javascript function to every item from source dataset(s) before saving to result dataset ### Output schema validation Validate schema Validate every item against JSON schema specified on input and filter out invalid items before saving to result dataset Reuse invalid items Invalid items are captured in separate requestListSources saved in KV store.