Kafka

​

Overview

The Kafka source streams data from an Apache Kafka topic into Reelevant. Use it for real-time or near-real-time data ingestion from event-driven architectures, streaming platforms, or message brokers that expose a Kafka-compatible interface.

​

Configuration

​

Required Fields

​

Authentication

​

Optional Fields

​

Schema Registry

If your Kafka messages use Avro serialization with a Confluent Schema Registry, configure the registry object: When configured, messages are automatically deserialized using the schema from the registry.

​

Supported Message Formats

​

Consumer Behavior

​

Worker Mode (Batch)

When used with a worker-mode datasource (periodic batch sync), the Kafka source:

1. Connects to the topic and reads all available messages up to the current latest offset.
2. Disconnects once all messages have been consumed.
3. On subsequent syncs, reading resumes from where it left off (consumer group offsets are preserved).

​

Ingester Mode (Continuous)

When used with an ingester-mode datasource (continuous streaming), the Kafka source:

1. Connects to the topic and continuously consumes new messages as they arrive.
2. Messages are processed and stored in real-time.

​

How It Works

1. Reelevant connects to the Kafka cluster using the provided broker addresses.
2. A consumer group is created (or resumed) for the datasource.
3. Messages are consumed from the configured topic.
4. If a Schema Registry is configured, messages are deserialized using the Avro schema.
5. Fields are extracted and made available for mapping.