openGauss Integration

This Apify Actor transfers data from Apify datasets into an openGauss database. It processes text, optionally splits it into chunks, computes vector embeddings, and stores them in openGauss. It's designed for search and Retrieval-Augmented Generation (RAG) applications, with support for incremental updates to save on compute and storage.

Overview

The Actor takes a dataset from another Apify Actor (like the Website Content Crawler), processes the text, generates embeddings using providers like OpenAI or Cohere via LangChain, and saves the vectors to your openGauss database. You control chunking, embedding models, and update strategies.

Key Features

• Vector Storage: Computes and stores text embeddings in openGauss for semantic search.
• Incremental Updates: Updates only changed data, reducing unnecessary embedding recomputation.
• Configurable Chunking: Uses LangChain's RecursiveCharacterTextSplitter to handle long documents (configurable chunk size/overlap).
• Embedding Provider Flexibility: Supports multiple providers (e.g., OpenAI, Cohere).
• Metadata Handling: Lets you specify which dataset fields to store as metadata alongside vectors.

How to Use

This Actor is typically configured via the "Integrations" section of another Actor's run (e.g., Website Content Crawler).

Prerequisites:
* A running openGauss database (you'll need host, port, user, password, DB name, and table name).
* An API key for your chosen embeddings provider (e.g., from OpenAI).

Basic Flow:
1. An upstream Actor (like a crawler) runs and produces a dataset.
2. This integration Actor is triggered, fetching that dataset.
3. (Optional) Text is split into chunks.
4. Embeddings are computed for the text/chunks.
5. Data and embeddings are saved to your specified openGauss table.

Input / Output

Input Configuration:

Your configuration needs three main parts: database connection, embeddings provider, and data mapping.

1. Database (openGauss):

{
  "opengaussHost": "your-host",
  "opengaussPort": "your-port",
  "opengaussUser": "your-user",
  "opengaussPassword": "your-password",
  "opengaussDBname": "your-database-name",
  "opengaussTableName": "apify_collection"
}

2. Embeddings Provider (Example: OpenAI):

{
  "embeddingsProvider": "OpenAIEmbeddings",
  "embeddingsApiKey": "your-openai-api-key",
  "embeddingsConfig": {"model": "text-embedding-3-large"}
}

Important: Ensure your openGauss table's vector column size matches the output dimensions of your chosen embedding model (e.g., 1536 for text-embedding-3-small).

3. Data Mapping (Example using Website Content Crawler output):
The crawler's dataset contains fields like url, text, and metadata. You specify which fields to embed and which to store as metadata.

{
  "datasetFields": ["text"],
  "metadataDatasetFields": {"title": "metadata.title"}
}

4. Optional Chunking:
To split long text before embedding, enable and configure chunking:

{
  "performChunking": true,
  "chunkSize": 1000,
  "chunkOverlap": 200
}

Output:
The Actor does not produce a separate dataset. Its output is the processed data (text, embeddings, and metadata) written directly to your specified openGauss table. Success or errors are reported in the Actor run log.