Apify to Kafka Integration ## Overview This integration facilitates the processing of data from an Apify dataset and its delivery to a Kafka topic. It is configured via a schema that includes details about the dataset, Kafka configuration, and batch processing size. ## Input Schema The input schema defines the configuration required to run the integration effectively. Below are the details of the schema and its properties: ### Schema Details - Title: Apify to Kafka Input Schema - Type: Object - Schema Version: 1 ### Properties #### 1. Default Dataset ID - Title: Default Dataset ID - Type: String - Description: The ID of the Apify dataset to process. - Default: default - Editor: Textfield #### 2. Kafka Configuration - Title: Kafka Configuration - Type: Object - Description: Configuration settings for Kafka connection. - Editor: JSON - Default: json { "clientId": "apify-kafka-producer", "brokers": ["localhost:9092"], "topic": "test-topic", "ssl": false } - Properties: - Client ID: - Title: Client ID - Type: String - Description: Kafka client identifier. - Default: apify-kafka-producer - Brokers: - Title: Brokers - Type: Array of strings - Description: Array of Kafka broker addresses. - Default: ["localhost:9092"] - Topic: - Title: Topic - Type: String - Description: Kafka topic name. - Default: test-topic - SSL: - Title: SSL - Type: Boolean - Description: Enable/disable SSL connection. - Default: false - SASL (Optional): - Title: SASL - Type: Object - Description: SASL configuration for Kafka connection. - Properties: - Username: - Title: Username - Type: String - Description: Kafka SASL username. - Password: - Title: Password - Type: String - Description: Kafka SASL password. - Mechanism: - Title: Mechanism - Type: String - Description: Kafka SASL mechanism. - Default: plain #### 3. Batch Size - Title: Batch Size - Type: Integer - Description: Number of messages to process in each batch. - Default: 2 - Minimum: 1 ### Required Properties - defaultDatasetId - kafkaConfig ## Example Configuration json { "defaultDatasetId": "my-dataset-id", "kafkaConfig": { "clientId": "my-kafka-client", "brokers": ["kafka-broker1:9092", "kafka-broker2:9092"], "topic": "my-kafka-topic", "ssl": true, "sasl": { "username": "my-username", "password": "my-password", "mechanism": "plain" } }, "batchSize": 5 } ## How to Use 1. Define the Input: Provide the necessary configurations in JSON format as per the schema. 2. Run the Integration: Pass the configuration to the Apify actor or script responsible for processing and delivering the dataset to Kafka. 3. Monitor Outputs: Check your Kafka topic for incoming messages based on the processed dataset. ## Notes - Ensure the Kafka broker addresses are reachable from the environment where the integration is executed. - SASL configuration is optional but required for secure Kafka connections. - Modify the batch size according to the volume of data and processing capacity.