Firestore Import Actor

This Apify actor imports data from an Apify dataset into a Firebase Firestore database. It provides flexible configuration for handling data conflicts, transforming records, and controlling how documents are created or updated.

Overview

The actor takes structured data from an Apify dataset and writes it to a specified collection in your Firestore database. You can control the target collection, generate or specify document IDs, and define how to handle existing documents. It's useful for automating data pipelines between Apify's scraping platform and Google's Firestore.

Key Features

• Flexible Data Import: Imports data from any Apify dataset into a Firestore collection.
• Configurable Document IDs: Use a field from your dataset as the document ID, or let Firestore auto-generate them.
• Conflict Resolution: Choose to overwrite, merge, or skip existing documents when IDs clash.
• Data Transformation: Apply a custom JavaScript function to modify dataset items before import. This function can also specify different collections or IDs for individual records.
• Batch Operations: Uses Firestore's batch writes for efficient performance (except when using the skip conflict resolution).

How to Use

1. Set up Firestore: In the Firebase Console, generate a service account key (JSON) for your project.
2. Prepare Data: Ensure your data is in an Apify dataset.
3. Configure the Actor: In the actor's input, provide your service account key, dataset ID, and target collection.
4. Run: Start the actor. It will process your dataset and import the records into Firestore.

Input

The actor requires the following configuration via its input fields:

• serviceAccountKey (string, secret, required)
  The full JSON service account key from Firebase. Get this from Firebase Console > Project Settings > Service accounts > Generate new private key. This value is stored encrypted.

• datasetId (string, required)
  The ID of the Apify dataset containing the data to import.

• collection (string, required)
  The name of the Firestore collection to import into. It will be created if it doesn't exist. You can override this per-document in the transformFunction.

• databaseName (string, optional)
  The specific Firestore database name. Uses the default "(default)" database if not provided.

• idField (string, optional)
  A field name from your dataset items (must be string or number) to use as the Firestore document ID. If not set, Firestore will auto-generate all IDs, and documentConflictResolution is ignored.

• documentConflictResolution (enum, required)
  Defines how to handle documents that already exist in Firestore:

  • overwrite: Completely replaces the existing document.
  • merge: Merges the new data with the existing document fields.
  • skip: Skips the import for documents with an existing ID. Note: This option performs individual document checks and cannot use batch writes, which significantly reduces performance for large datasets.

• transformFunction (string, optional)
  A JavaScript function that receives a dataset item and returns an object (or array of objects) to be written to Firestore. Use this to clean, reshape, or route your data.
  javascript // Example: Transform item and assign to a sub-collection. function transform(item) { // Return a single document configuration. return { collection: `users/${item.userId}/posts`, // Write to a sub-collection id: item.slug, // Custom ID data: { // The document data title: item.title, published: new Date(item.timestamp) } }; // Or return an array to create multiple documents from one item. }

Output

The actor outputs a summary of the import operation to the Apify dataset named RESULTS. This dataset contains a single record with execution details:

• importSummary: An object with counts like totalProcessedItems, successfulWrites, and skippedItems.
• errorSummary: An array listing any errors that occurred during the import, if applicable.

You can access this dataset via the Apify API or through the run console to verify the results.