Confluent Kafka messaging provider configuration

Learn about the configuration parameters required to use Confluent Kafka as a messaging provider in Reltio.

Confluent Kafka can be configured as a messaging provider for event streaming destinations. You can define the provider using either a provider alias configuration or a URI-based configuration.

Provider type values

The following provider type values define the authentication mechanism used to connect to Confluent Kafka:


Type	Credentials required	Description
kafka	Optional	Uses PLAINTEXT if no credentials are provided, or SASL/PLAIN if credentials are provided.
kafka-local	No	Uses PLAINTEXT for local Confluent Kafka environments.
kafka-plain	Yes	Uses SASL/PLAIN authentication.
kafka-scram-sha-256	Yes	Uses SASL/SCRAM-SHA-256 authentication.
kafka-scram-sha-512	Yes	Uses SASL/SCRAM-SHA-512 authentication.

Provider configuration parameters

You can also configure Confluent Kafka providers using provider properties with a provider alias.


Parameter	Required	Description
provider.<alias>.type	Yes	Specifies the Confluent Kafka provider type and authentication mechanism.
provider.<alias>.host	Yes	Confluent Kafka bootstrap server in `host:port` format.
provider.<alias>.username	Conditional	Kafka API key or username. Required for SASL authentication types.
provider.<alias>.password	Conditional	Kafka API secret or password. Required for SASL authentication types.

For authentication types that require credentials, both username and password must be provided together. Providing only one of these values is not supported.

URI-based configuration

Confluent Kafka providers are configured using a URI that defines the authentication type, credentials, and Kafka bootstrap server.

The URI format is:


kafka-plain://{username}:{password}@{host}:{port}

The username and password values must be URL-encoded if they contain special characters.

Here's an example for uri:


kafka-plain://my-api-key:my-api-secret@pkc-12345.us-east-1.aws.confluent.cloud:9092

The Confluent Kafka provider URI contains the following components:

Scheme: Defines the authentication type, for example kafka-plain or kafka-scram-sha-256.
Username: Kafka API key or SASL username.
Password: Kafka API secret or SASL password.
Host and port: Confluent Kafka bootstrap server in host:port format.

Tenant configuration example

The Confluent Kafka provider is configured in the tenant messaging destinations using the provider URI.


{
    "messaging": {
        "destinations": [
            {
                "type": "queue",
                "provider": "kafka-plain://<username>:<password>@<host>:<port>",
                "name": "example",
                "enabled": true
            }
        ]
    }
}

Optional configuration properties

Additional Confluent Kafka configuration properties can be defined using the provider.<alias>.properties.* prefix.

These properties control connection behavior, topic configuration, and producer settings. Examples include:

Connection settings such as timeouts and in-flight message limits
Topic settings such as partition count and retention policies
Producer settings such as batching, compression, and retries

If not specified, default values defined by the Confluent Kafka client or environment configuration are used.

Unify and manage your data