这是用户在 2024-12-2 23:03 为 https://kafka.apache.org/documentation/#gettingStarted 保存的双语快照页面，由 沉浸式翻译 提供双语支持。了解如何保存？
<

Documentation

Kafka 3.9 Documentation

Prior releases: 0.7.x, 0.8.0, 0.8.1.X, 0.8.2.X, 0.9.0.X, 0.10.0.X, 0.10.1.X, 0.10.2.X, 0.11.0.X, 1.0.X, 1.1.X, 2.0.X, 2.1.X, 2.2.X, 2.3.X, 2.4.X, 2.5.X, 2.6.X, 2.7.X, 2.8.X, 3.0.X, 3.1.X, 3.2.X, 3.3.X, 3.4.X, 3.5.X, 3.6.X, 3.7.X, 3.8.X.

1. Getting Started

1.1 Introduction

What is event streaming?

Event streaming is the digital equivalent of the human body's central nervous system. It is the technological foundation for the 'always-on' world where businesses are increasingly software-defined and automated, and where the user of software is more software.

Technically speaking, event streaming is the practice of capturing data in real-time from event sources like databases, sensors, mobile devices, cloud services, and software applications in the form of streams of events; storing these event streams durably for later retrieval; manipulating, processing, and reacting to the event streams in real-time as well as retrospectively; and routing the event streams to different destination technologies as needed. Event streaming thus ensures a continuous flow and interpretation of data so that the right information is at the right place, at the right time.

What can I use event streaming for?

Event streaming is applied to a wide variety of use cases across a plethora of industries and organizations. Its many examples include:

  • To process payments and financial transactions in real-time, such as in stock exchanges, banks, and insurances.
  • To track and monitor cars, trucks, fleets, and shipments in real-time, such as in logistics and the automotive industry.
  • To continuously capture and analyze sensor data from IoT devices or other equipment, such as in factories and wind parks.
  • To collect and immediately react to customer interactions and orders, such as in retail, the hotel and travel industry, and mobile applications.
  • To monitor patients in hospital care and predict changes in condition to ensure timely treatment in emergencies.
  • To connect, store, and make available data produced by different divisions of a company.
  • To serve as the foundation for data platforms, event-driven architectures, and microservices.

Apache Kafka® is an event streaming platform. What does that mean?

Kafka combines three key capabilities so you can implement your use cases for event streaming end-to-end with a single battle-tested solution:

  1. To publish (write) and subscribe to (read) streams of events, including continuous import/export of your data from other systems.
  2. To store streams of events durably and reliably for as long as you want.
  3. To process streams of events as they occur or retrospectively.

And all this functionality is provided in a distributed, highly scalable, elastic, fault-tolerant, and secure manner. Kafka can be deployed on bare-metal hardware, virtual machines, and containers, and on-premises as well as in the cloud. You can choose between self-managing your Kafka environments and using fully managed services offered by a variety of vendors.

How does Kafka work in a nutshell?

Kafka is a distributed system consisting of servers and clients that communicate via a high-performance TCP network protocol. It can be deployed on bare-metal hardware, virtual machines, and containers in on-premise as well as cloud environments.

Servers: Kafka is run as a cluster of one or more servers that can span multiple datacenters or cloud regions. Some of these servers form the storage layer, called the brokers. Other servers run Kafka Connect to continuously import and export data as event streams to integrate Kafka with your existing systems such as relational databases as well as other Kafka clusters. To let you implement mission-critical use cases, a Kafka cluster is highly scalable and fault-tolerant: if any of its servers fails, the other servers will take over their work to ensure continuous operations without any data loss.

Clients: They allow you to write distributed applications and microservices that read, write, and process streams of events in parallel, at scale, and in a fault-tolerant manner even in the case of network problems or machine failures. Kafka ships with some such clients included, which are augmented by dozens of clients provided by the Kafka community: clients are available for Java and Scala including the higher-level Kafka Streams library, for Go, Python, C/C++, and many other programming languages as well as REST APIs.

Main Concepts and Terminology

An event records the fact that "something happened" in the world or in your business. It is also called record or message in the documentation. When you read or write data to Kafka, you do this in the form of events. Conceptually, an event has a key, value, timestamp, and optional metadata headers. Here's an example event:

  • Event key: "Alice"
  • Event value: "Made a payment of $200 to Bob"
  • Event timestamp: "Jun. 25, 2020 at 2:06 p.m."

Producers are those client applications that publish (write) events to Kafka, and consumers are those that subscribe to (read and process) these events. In Kafka, producers and consumers are fully decoupled and agnostic of each other, which is a key design element to achieve the high scalability that Kafka is known for. For example, producers never need to wait for consumers. Kafka provides various guarantees such as the ability to process events exactly-once.

Events are organized and durably stored in topics. Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder. An example topic name could be "payments". Topics in Kafka are always multi-producer and multi-subscriber: a topic can have zero, one, or many producers that write events to it, as well as zero, one, or many consumers that subscribe to these events. Events in a topic can be read as often as needed—unlike traditional messaging systems, events are not deleted after consumption. Instead, you define for how long Kafka should retain your events through a per-topic configuration setting, after which old events will be discarded. Kafka's performance is effectively constant with respect to data size, so storing data for a long time is perfectly fine.

Topics are partitioned, meaning a topic is spread over a number of "buckets" located on different Kafka brokers. This distributed placement of your data is very important for scalability because it allows client applications to both read and write the data from/to many brokers at the same time. When a new event is published to a topic, it is actually appended to one of the topic's partitions. Events with the same event key (e.g., a customer or vehicle ID) are written to the same partition, and Kafka guarantees that any consumer of a given topic-partition will always read that partition's events in exactly the same order as they were written.

Figure: This example topic has four partitions P1–P4. Two different producer clients are publishing, independently from each other, new events to the topic by writing events over the network to the topic's partitions. Events with the same key (denoted by their color in the figure) are written to the same partition. Note that both producers can write to the same partition if appropriate.

To make your data fault-tolerant and highly-available, every topic can be replicated, even across geo-regions or datacenters, so that there are always multiple brokers that have a copy of the data just in case things go wrong, you want to do maintenance on the brokers, and so on. A common production setting is a replication factor of 3, i.e., there will always be three copies of your data. This replication is performed at the level of topic-partitions.

This primer should be sufficient for an introduction. The Design section of the documentation explains Kafka's various concepts in full detail, if you are interested.

Kafka APIs

In addition to command line tooling for management and administration tasks, Kafka has five core APIs for Java and Scala:

  • The Admin API to manage and inspect topics, brokers, and other Kafka objects.
  • The Producer API to publish (write) a stream of events to one or more Kafka topics.
  • The Consumer API to subscribe to (read) one or more topics and to process the stream of events produced to them.
  • The Kafka Streams API to implement stream processing applications and microservices. It provides higher-level functions to process event streams, including transformations, stateful operations like aggregations and joins, windowing, processing based on event-time, and more. Input is read from one or more topics in order to generate output to one or more topics, effectively transforming the input streams to output streams.
  • The Kafka Connect API to build and run reusable data import/export connectors that consume (read) or produce (write) streams of events from and to external systems and applications so they can integrate with Kafka. For example, a connector to a relational database like PostgreSQL might capture every change to a set of tables. However, in practice, you typically don't need to implement your own connectors because the Kafka community already provides hundreds of ready-to-use connectors.

Where to go from here

1.2 Use Cases

Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.

Messaging

Kafka works well as a replacement for a more traditional message broker. Message brokers are used for a variety of reasons (to decouple processing from data producers, to buffer unprocessed messages, etc). In comparison to most messaging systems Kafka has better throughput, built-in partitioning, replication, and fault-tolerance which makes it a good solution for large scale message processing applications.

In our experience messaging uses are often comparatively low-throughput, but may require low end-to-end latency and often depend on the strong durability guarantees Kafka provides.

In this domain Kafka is comparable to traditional messaging systems such as ActiveMQ or RabbitMQ.

Website Activity Tracking

The original use case for Kafka was to be able to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds. This means site activity (page views, searches, or other actions users may take) is published to central topics with one topic per activity type. These feeds are available for subscription for a range of use cases including real-time processing, real-time monitoring, and loading into Hadoop or offline data warehousing systems for offline processing and reporting.

Activity tracking is often very high volume as many activity messages are generated for each user page view.

Metrics

Kafka is often used for operational monitoring data. This involves aggregating statistics from distributed applications to produce centralized feeds of operational data.

Log Aggregation

Many people use Kafka as a replacement for a log aggregation solution. Log aggregation typically collects physical log files off servers and puts them in a central place (a file server or HDFS perhaps) for processing. Kafka abstracts away the details of files and gives a cleaner abstraction of log or event data as a stream of messages. This allows for lower-latency processing and easier support for multiple data sources and distributed data consumption. In comparison to log-centric systems like Scribe or Flume, Kafka offers equally good performance, stronger durability guarantees due to replication, and much lower end-to-end latency.

Stream Processing

Many users of Kafka process data in processing pipelines consisting of multiple stages, where raw input data is consumed from Kafka topics and then aggregated, enriched, or otherwise transformed into new topics for further consumption or follow-up processing. For example, a processing pipeline for recommending news articles might crawl article content from RSS feeds and publish it to an "articles" topic; further processing might normalize or deduplicate this content and publish the cleansed article content to a new topic; a final processing stage might attempt to recommend this content to users. Such processing pipelines create graphs of real-time data flows based on the individual topics. Starting in 0.10.0.0, a light-weight but powerful stream processing library called Kafka Streams is available in Apache Kafka to perform such data processing as described above. Apart from Kafka Streams, alternative open source stream processing tools include Apache Storm and Apache Samza.

Event Sourcing

Event sourcing is a style of application design where state changes are logged as a time-ordered sequence of records. Kafka's support for very large stored log data makes it an excellent backend for an application built in this style.

Commit Log

Kafka can serve as a kind of external commit-log for a distributed system. The log helps replicate data between nodes and acts as a re-syncing mechanism for failed nodes to restore their data. The log compaction feature in Kafka helps support this usage. In this usage Kafka is similar to Apache BookKeeper project.

1.3 Quick Start

Step 1: Get Kafka

Download the latest Kafka release and extract it: 

$ tar -xzf kafka_2.13-3.9.0.tgz
$ cd kafka_2.13-3.9.0

Step 2: Start the Kafka environment

NOTE: Your local environment must have Java 8+ installed.

Apache Kafka can be started using KRaft or ZooKeeper. To get started with either configuration follow one of the sections below but not both.

Kafka with KRaft

Kafka can be run using KRaft mode using local scripts and downloaded files or the docker image. Follow one of the sections below but not both to start the kafka server.

Using downloaded files

Generate a Cluster UUID

$ KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

Format Log Directories

$ bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/kraft/reconfig-server.properties

Start the Kafka Server

$ bin/kafka-server-start.sh config/kraft/reconfig-server.properties

Once the Kafka server has successfully launched, you will have a basic Kafka environment running and ready to use.

Using JVM Based Apache Kafka Docker Image

Get the Docker image:

$ docker pull apache/kafka:3.9.0

Start the Kafka Docker container: 

$ docker run -p 9092:9092 apache/kafka:3.9.0
Using GraalVM Based Native Apache Kafka Docker Image

Get the Docker image:

$ docker pull apache/kafka-native:3.9.0

Start the Kafka Docker container:

$ docker run -p 9092:9092 apache/kafka-native:3.9.0
Kafka with ZooKeeper

Run the following commands in order to start all services in the correct order:

# Start the ZooKeeper service
$ bin/zookeeper-server-start.sh config/zookeeper.properties

Open another terminal session and run:

# Start the Kafka broker service
$ bin/kafka-server-start.sh config/server.properties

Once all services have successfully launched, you will have a basic Kafka environment running and ready to use.

Step 3: Create a topic to store your events

Kafka is a distributed event streaming platform that lets you read, write, store, and process events (also called records or messages in the documentation) across many machines.

Example events are payment transactions, geolocation updates from mobile phones, shipping orders, sensor measurements from IoT devices or medical equipment, and much more. These events are organized and stored in topics. Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder.

So before you can write your first events, you must create a topic. Open another terminal session and run: 

$ bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092

All of Kafka's command line tools have additional options: run the kafka-topics.sh command without any arguments to display usage information. For example, it can also show you details such as the partition count of the new topic: 

$ bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
Topic: quickstart-events        TopicId: NPmZHyhbR9y00wMglMH2sg PartitionCount: 1       ReplicationFactor: 1	Configs:
Topic: quickstart-events Partition: 0    Leader: 0   Replicas: 0 Isr: 0

Step 4: Write some events into the topic

A Kafka client communicates with the Kafka brokers via the network for writing (or reading) events. Once received, the brokers will store the events in a durable and fault-tolerant manner for as long as you need—even forever.

Run the console producer client to write a few events into your topic. By default, each line you enter will result in a separate event being written to the topic. 

$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
>This is my first event
>This is my second event

You can stop the producer client with Ctrl-C at any time.

Step 5: Read the events

Open another terminal session and run the console consumer client to read the events you just created:

$ bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
This is my first event
This is my second event

You can stop the consumer client with Ctrl-C at any time.

Feel free to experiment: for example, switch back to your producer terminal (previous step) to write additional events, and see how the events immediately show up in your consumer terminal.

Because events are durably stored in Kafka, they can be read as many times and by as many consumers as you want. You can easily verify this by opening yet another terminal session and re-running the previous command again.

Step 6: Import/export your data as streams of events with Kafka Connect

You probably have lots of data in existing systems like relational databases or traditional messaging systems, along with many applications that already use these systems. Kafka Connect allows you to continuously ingest data from external systems into Kafka, and vice versa. It is an extensible tool that runs connectors, which implement the custom logic for interacting with an external system. It is thus very easy to integrate existing systems with Kafka. To make this process even easier, there are hundreds of such connectors readily available.

In this quickstart we'll see how to run Kafka Connect with simple connectors that import data from a file to a Kafka topic and export data from a Kafka topic to a file.

First, make sure to add connect-file-3.9.0.jar to the plugin.path property in the Connect worker's configuration. For the purpose of this quickstart we'll use a relative path and consider the connectors' package as an uber jar, which works when the quickstart commands are run from the installation directory. However, it's worth noting that for production deployments using absolute paths is always preferable. See plugin.path for a detailed description of how to set this config.

Edit the config/connect-standalone.properties file, add or change the plugin.path configuration property match the following, and save the file: 

$ echo "plugin.path=libs/connect-file-3.9.0.jar" >> config/connect-standalone.properties

Then, start by creating some seed data to test with: 

$ echo -e "foo\nbar" > test.txt
Or on Windows: 
$ echo foo > test.txt
$ echo bar >> test.txt

Next, we'll start two connectors running in standalone mode, which means they run in a single, local, dedicated process. We provide three configuration files as parameters. The first is always the configuration for the Kafka Connect process, containing common configuration such as the Kafka brokers to connect to and the serialization format for data. The remaining configuration files each specify a connector to create. These files include a unique connector name, the connector class to instantiate, and any other configuration required by the connector. 

$ bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties

These sample configuration files, included with Kafka, use the default local cluster configuration you started earlier and create two connectors: the first is a source connector that reads lines from an input file and produces each to a Kafka topic and the second is a sink connector that reads messages from a Kafka topic and produces each as a line in an output file.

During startup you'll see a number of log messages, including some indicating that the connectors are being instantiated. Once the Kafka Connect process has started, the source connector should start reading lines from test.txt and producing them to the topic connect-test, and the sink connector should start reading messages from the topic connect-test and write them to the file test.sink.txt. We can verify the data has been delivered through the entire pipeline by examining the contents of the output file: 

$ more test.sink.txt
foo
bar

Note that the data is being stored in the Kafka topic connect-test, so we can also run a console consumer to see the data in the topic (or use custom consumer code to process it): 

$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic connect-test --from-beginning
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
…

The connectors continue to process data, so we can add data to the file and see it move through the pipeline:

$ echo "Another line" >> test.txt

You should see the line appear in the console consumer output and in the sink file.

Step 7: Process your events with Kafka Streams

Once your data is stored in Kafka as events, you can process the data with the Kafka Streams client library for Java/Scala. It allows you to implement mission-critical real-time applications and microservices, where the input and/or output data is stored in Kafka topics. Kafka Streams combines the simplicity of writing and deploying standard Java and Scala applications on the client side with the benefits of Kafka's server-side cluster technology to make these applications highly scalable, elastic, fault-tolerant, and distributed. The library supports exactly-once processing, stateful operations and aggregations, windowing, joins, processing based on event-time, and much more.

To give you a first taste, here's how one would implement the popular WordCount algorithm:

KStream<String, String> textLines = builder.stream("quickstart-events");

KTable<String, Long> wordCounts = textLines
            .flatMapValues(line -> Arrays.asList(line.toLowerCase().split(" ")))
            .groupBy((keyIgnored, word) -> word)
            .count();

wordCounts.toStream().to("output-topic", Produced.with(Serdes.String(), Serdes.Long()));

The Kafka Streams demo and the app development tutorial demonstrate how to code and run such a streaming application from start to finish.

Step 8: Terminate the Kafka environment

Now that you reached the end of the quickstart, feel free to tear down the Kafka environment—or continue playing around.

  1. Stop the producer and consumer clients with Ctrl-C, if you haven't done so already.
  2. Stop the Kafka broker with Ctrl-C.
  3. Lastly, if the Kafka with ZooKeeper section was followed, stop the ZooKeeper server with Ctrl-C.

If you also want to delete any data of your local Kafka environment including any events you have created along the way, run the command: 

$ rm -rf /tmp/kafka-logs /tmp/zookeeper /tmp/kraft-combined-logs

Congratulations!

You have successfully finished the Apache Kafka quickstart.

To learn more, we suggest the following next steps:

1.4 Ecosystem

There are a plethora of tools that integrate with Kafka outside the main distribution. The ecosystem page lists many of these, including stream processing systems, Hadoop integration, monitoring, and deployment tools.

1.5 Upgrading From Previous Versions

Upgrading to 3.9.0 from any version 0.8.x through 3.8.x

Upgrading ZooKeeper-based clusters

If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 2.4.0 or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version. However, before doing this, make sure ZooKeeper is upgraded to version 3.8.3 or higher.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 3.8, 3.7, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 3.9.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.8 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Upgrading KRaft-based clusters

If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.

For a rolling upgrade:

  1. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
  2. Once the cluster's behavior and performance has been verified, bump the metadata.version by running bin/kafka-features.sh upgrade --metadata 3.9
  3. Note that cluster metadata downgrade is not supported in this software version. Every MetadataVersion after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. IBP_3_3_IV3(7, "3.3", "IV3", true) means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.9.0
  • In case you run your Kafka clusters with no execution permission for the /tmp partition, Kafka will not work properly. It might either refuse to start or fail when producing and consuming messages. This is due to the compression libraries zstd-jni and snappy. To remediate this problem you need to pass the following JVM flags to Kafka ZstdTempFolder and org.xerial.snappy.tempdir pointing to a directory with execution permissions. For example, this could be done via the KAFKA_OPTS environment variable like follows: export KAFKA_OPTS="-DZstdTempFolder=/opt/kafka/tmp -Dorg.xerial.snappy.tempdir=/opt/kafka/tmp". This is a known issue for version 3.8.0 as well.
  • unclean.leader.election.enable config is supported in KRaft. Compared with ZK mode, there is one behavior change in KRaft mode when dynamically enabling unclean.leader.election.enable config. Please check here for more details.
  • Tiered storage is now a production ready feature. You can check Kafka Tiered Storage GA Release Notes for more details. The below enhancements are added in this release.
    • In KRaft mode, the tiered storage feature can be dynamically disabled and then re-enabled on topic level. See KIP-950 for more details.
    • With the tiered storage quota feature, users can define a maximum limit on the rate at which log segments are transferred to or retrieved from the remote storage. See KIP-956 for more details.
  • Controller membership change in KRaft is now supported when formatting with `--standalone` or `--initial-controllers` option. See here for more details.

Upgrading to 3.8.1 from any version 0.8.x through 3.7.x

Upgrading ZooKeeper-based clusters

If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 3.7, 3.6, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 3.8.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.8 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Upgrading KRaft-based clusters

If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.

For a rolling upgrade:

  1. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
  2. Once the cluster's behavior and performance has been verified, bump the metadata.version by running bin/kafka-features.sh upgrade --metadata 3.8
  3. Note that cluster metadata downgrade is not supported in this version since it has metadata changes. Every MetadataVersion after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. IBP_3_3_IV3(7, "3.3", "IV3", true) means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.8.1
  • In case you run your Kafka clusters with no execution permission for the /tmp partition, Kafka will not work properly. It might either refuse to start or fail when producing and consuming messages. This is due to the compression libraries zstd-jni and snappy. To remediate this problem you need to pass the following JVM flags to Kafka ZstdTempFolder and org.xerial.snappy.tempdir pointing to a directory with execution permissions. For example, this could be done via the KAFKA_OPTS environment variable like follows: export KAFKA_OPTS="-DZstdTempFolder=/opt/kafka/tmp -Dorg.xerial.snappy.tempdir=/opt/kafka/tmp". This is a known issue for version 3.8.0.
  • In 3.8.0 the kafka.utils.Thottler metric was accidentally renamed to org.apache.kafka.storage.internals.utils.Throttler. This change has been reverted and the metric is now named kafka.utils.Thottler again.

Upgrading to 3.8.0 from any version 0.8.x through 3.7.x

Notable changes in 3.8.0
  • MirrorMaker 2 can now emit checkpoints for offsets mirrored before the start of the Checkpoint task for improved offset translation. This requires MirrorMaker 2 to have READ authorization to the Checkpoint topic. If READ is not authorized, checkpointing is limited to offsets mirrorred after the start of the task. See KAFKA-15905 for more details.
  • JBOD in KRaft is no longer in early access.
  • Tiered Storage, which is still in early access, now supports clusters configured with multiple log directories (i.e. JBOD feature).

Upgrading to 3.7.1 from any version 0.8.x through 3.6.x

Upgrading ZooKeeper-based clusters

If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 3.6, 3.5, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 3.7.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.7 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Upgrading KRaft-based clusters

If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.

For a rolling upgrade:

  1. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
  2. Once the cluster's behavior and performance has been verified, bump the metadata.version by running bin/kafka-features.sh upgrade --metadata 3.7
  3. Note that cluster metadata downgrade is not supported in this version since it has metadata changes. Every MetadataVersion after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. IBP_3_3_IV3(7, "3.3", "IV3", true) means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.7.1
  • MirrorMaker 2 can now emit checkpoints for offsets mirrored before the start of the Checkpoint task for improved offset translation. This requires MirrorMaker 2 to have READ authorization to the Checkpoint topic. If READ is not authorized, checkpointing is limited to offsets mirrorred after the start of the task. See KAFKA-15905 for more details.
  • JBOD support in KRaft was introduced from Metadata Version (MV) 3.7-IV2. Configuring Brokers with multiple log directories can lead to indefinite unavailability. Brokers will now detect this situation and log an error. See KAFKA-16606 for more details.
Notable changes in 3.7.0
  • Java 11 support for the broker and tools has been deprecated and will be removed in Apache Kafka 4.0. This complements the previous deprecation of Java 8 for all components. Please refer to KIP-1013 for more details.
  • Client APIs released prior to Apache Kafka 2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. See KIP-896 for details and RPC versions that are now deprecated.
  • Early access of the new simplified Consumer Rebalance Protocol is available, and it is not recommended for use in production environments. You are encouraged to test it and provide feedback! For more information about the early access feature, please check KIP-848 and the Early Access Release Notes.
  • More metrics related to Tiered Storage have been introduced. They should improve the operational experience of running Tiered Storage in production. For more detailed information, please refer to KIP-963.
  • Kafka Streams ships multiple KIPs for IQv2 support. See the Kafka Streams upgrade section for more details.
  • In versions 3.5.0, 3.5.1, 3.5.2, 3.6.0, and 3.6.1, MirrorMaker 2 offset translation may not reach the end of a replicated topic after the upstream consumers commit at the end of the source topic. This was addressed in KAFKA-15906.
  • All the notable changes are present in the blog post announcing the 3.7.0 release.

Upgrading to 3.6.2 from any version 0.8.x through 3.5.x

Upgrading ZooKeeper-based clusters

If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 3.5, 3.4, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 3.6.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.6 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Upgrading KRaft-based clusters

If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.

For a rolling upgrade:

  1. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
  2. Once the cluster's behavior and performance has been verified, bump the metadata.version by running bin/kafka-features.sh upgrade --metadata 3.6
  3. Note that cluster metadata downgrade is not supported in this version since it has metadata changes. Every MetadataVersion after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. IBP_3_3_IV3(7, "3.3", "IV3", true) means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.6.0
  • Apache Kafka now supports having both an IPv4 and an IPv6 listener on the same port. This change only applies to non advertised listeners (advertised listeners already have this feature)
  • The Apache Zookeeper dependency has been upgraded to 3.8.1 due to 3.6 reaching end-of-life. To bring both your Kafka and Zookeeper clusters to the latest versions:
    • >=2.4 Kafka clusters can be updated directly. Zookeeper clusters which are running binaries bundled with Kafka versions 2.4 or above can be updated directly.
    • <2.4 Kafka clusters first need to be updated to a version greater than 2.4 and smaller than 3.6. Zookeeper clusters which are running binaries bundled with Kafka versions below 2.4 need to be updated to any binaries bundled with Kafka versions greater than 2.4 and smaller than 3.6. You can then follow the first bullet-point.
    For more detailed information please refer to the Compatibility, Deprecation, and Migration Plan section in KIP-902.
  • The configuration log.message.timestamp.difference.max.ms is deprecated. Two new configurations, log.message.timestamp.before.max.ms and log.message.timestamp.after.max.ms, have been added. For more detailed information, please refer to KIP-937.
  • Kafka Streams has introduced a new task assignor, RackAwareTaskAssignor, for computing task assignments which can minimize cross rack traffic under certain conditions. It works with existing StickyTaskAssignor and HighAvailabilityTaskAssignor. See KIP-925 and Kafka Streams Developer Guide for more details.
  • To account for a break in compatibility introduced in version 3.1.0, MirrorMaker 2 has added a new replication.policy.internal.topic.separator.enabled property. If upgrading from 3.0.x or earlier, it may be necessary to set this property to false; see the property's documentation for more details.
  • Early access of tiered storage feature is available, and it is not recommended for use in production environments. Welcome to test it and provide any feedback to us. For more information about the early access tiered storage feature, please check KIP-405 and Tiered Storage Early Access Release Note.
  • Transaction partition verification (KIP-890) has been added to data partitions to prevent hanging transactions. This feature is enabled by default and can be disabled by setting transaction.partition.verification.enable to false. The configuration can also be updated dynamically and is applied to the broker. Workloads running on version 3.6.0 with compression can experience InvalidRecordExceptions and UnknownServerExceptions. Upgrading to 3.6.1 or newer or disabling the feature fixes the issue.

Upgrading to 3.5.2 from any version 0.8.x through 3.4.x

All upgrade steps remain same as upgrading to 3.5.0
Notable changes in 3.5.2
  • When migrating producer ID blocks from ZK to KRaft, there could be duplicate producer IDs being given to transactional or idempotent producers. This can cause long term problems since the producer IDs are persisted and reused for a long time. See KAFKA-15552 for more details.
  • In 3.5.0 and 3.5.1, there could be an issue that the empty ISR is returned from controller after AlterPartition request during rolling upgrade. This issue will impact the availability of the topic partition. See KAFKA-15353 for more details.
  • In 3.5.0 and 3.5.1, there was an issue where MirrorMaker 2 offset translation produced an earlier offset than needed, substantially increasing the re-delivery of data when starting a consumer from the downstream consumer offsets. See KAFKA-15202 for more details.

Upgrading to 3.5.1 from any version 0.8.x through 3.4.x

All upgrade steps remain same as upgrading to 3.5.0
Notable changes in 3.5.1
  • Upgraded the dependency, snappy-java, to a version which is not vulnerable to CVE-2023-34455. You can find more information about the CVE at Kafka CVE list.
  • Fixed a regression introduced in 3.3.0, which caused security.protocol configuration values to be restricted to upper case only. After the fix, security.protocol values are case insensitive. See KAFKA-15053 for details.

Upgrading to 3.5.0 from any version 0.8.x through 3.4.x

Upgrading ZooKeeper-based clusters

If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 3.4, 3.3, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 3.5.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.5 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Upgrading KRaft-based clusters

If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.

For a rolling upgrade:

  1. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
  2. Once the cluster's behavior and performance has been verified, bump the metadata.version by running bin/kafka-features.sh upgrade --metadata 3.5
  3. Note that cluster metadata downgrade is not supported in this version since it has metadata changes. Every MetadataVersion after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. IBP_3_3_IV3(7, "3.3", "IV3", true) means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.5.0
  • Kafka Streams has introduced a new state store type, versioned key-value stores, for storing multiple record versions per key, thereby enabling timestamped retrieval operations to return the latest record (per key) as of a specified timestamp. See KIP-889 and KIP-914 for more details. If the new store typed is used in the DSL, improved processing semantics are applied as described in KIP-914.
  • KTable aggregation semantics got further improved via KIP-904, now avoiding spurious intermediate results.
  • Kafka Streams' ProductionExceptionHandler is improved via KIP-399, now also covering serialization errors.
  • MirrorMaker now uses incrementalAlterConfigs API by default to synchronize topic configurations instead of the deprecated alterConfigs API. A new settings called use.incremental.alter.configs is introduced to allow users to control which API to use. This new setting is marked deprecated and will be removed in the next major release when incrementalAlterConfigs API is always used. See KIP-894 for more details.
  • The JmxTool, EndToEndLatency, StreamsResetter, ConsumerPerformance and ClusterTool have been migrated to the tools module. The 'kafka.tools' package is deprecated and will change to 'org.apache.kafka.tools' in the next major release. See KAFKA-14525 for more details.
  • In versions earlier than 3.5.0 and 3.4.1, MirrorMaker 2 offset translation could incorrectly translate offsets for topics using compaction, transactional producers, and filter SMTs. In 3.5.0 and 3.4.1, offset translation has changed for all topics in order to ensure at-least-once delivery when a consumer is failed-over to the translated offsets. Translated offsets will be earlier than in previous versions, so consumers using downstream offsets may initially have more lag, and re-deliver more data after failing-over. See KAFKA-12468 and linked tickets, and PR #13178 for more details. Further improvements to the offset translation are included in later releases to reduce the lag introduced by this change, so consider upgrading MM2 to the latest version available.

Upgrading to 3.4.0 from any version 0.8.x through 3.3.x

If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 3.3, 3.2, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 3.4.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.4 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.

Upgrading a KRaft-based cluster to 3.4.0 from any version 3.0.x through 3.3.x

If you are upgrading from a version prior to 3.3.0, please see the note below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.

For a rolling upgrade:

  1. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
  2. Once the cluster's behavior and performance has been verified, bump the metadata.version by running bin/kafka-features.sh upgrade --metadata 3.4
  3. Note that cluster metadata downgrade is not supported in this version since it has metadata changes. Every MetadataVersion after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. IBP_3_3_IV3(7, "3.3", "IV3", true) means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.4.0
  • Since Apache Kafka 3.4.0, we have added a system property ("org.apache.kafka.disallowed.login.modules") to disable the problematic login modules usage in SASL JAAS configuration. Also by default "com.sun.security.auth.module.JndiLoginModule" is disabled from Apache Kafka 3.4.0.

Upgrading to 3.3.1 from any version 0.8.x through 3.2.x

If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 3.2, 3.1, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 3.3.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.3 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.

Upgrading a KRaft-based cluster to 3.3.1 from any version 3.0.x through 3.2.x

If you are upgrading from a version prior to 3.3.1, please see the note below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.

For a rolling upgrade:

  1. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
  2. Once the cluster's behavior and performance has been verified, bump the metadata.version by running bin/kafka-features.sh upgrade --metadata 3.3
  3. Note that cluster metadata downgrade is not supported in this version since it has metadata changes. Every MetadataVersion after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. IBP_3_3_IV3(7, "3.3", "IV3", true) means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.3.1
  • KRaft mode is production ready for new clusters. See KIP-833 for more details (including limitations).
  • The partitioner used by default for records with no keys has been improved to avoid pathological behavior when one or more brokers are slow. The new logic may affect the batching behavior, which can be tuned using the batch.size and/or linger.ms configuration settings. The previous behavior can be restored by setting partitioner.class=org.apache.kafka.clients.producer.internals.DefaultPartitioner. See KIP-794 for more details.
  • There is now a slightly different upgrade process for KRaft clusters than for ZK-based clusters, as described above.
  • Introduced a new API addMetricIfAbsent to Metrics which would create a new Metric if not existing or return the same metric if already registered. Note that this behaviour is different from addMetric API which throws an IllegalArgumentException when trying to create an already existing metric. (See KIP-843 for more details).

Upgrading to 3.2.0 from any version 0.8.x through 3.1.x

If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 3.1, 3.0, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 3.2.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.2 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 3.2.0
  • Idempotence for the producer is enabled by default if no conflicting configurations are set. When producing to brokers older than 2.8.0, the IDEMPOTENT_WRITE permission is required. Check the compatibility section of KIP-679 for details. In 3.0.0 and 3.1.0, a bug prevented this default from being applied, which meant that idempotence remained disabled unless the user had explicitly set enable.idempotence to true (See KAFKA-13598 for more details). This issue was fixed and the default is properly applied in 3.0.1, 3.1.1, and 3.2.0.
  • A notable exception is Connect that by default disables idempotent behavior for all of its producers in order to uniformly support using a wide range of Kafka broker versions. Users can change this behavior to enable idempotence for some or all producers via Connect worker and/or connector configuration. Connect may enable idempotent producers by default in a future major release.
  • Kafka has replaced log4j with reload4j due to security concerns. This only affects modules that specify a logging backend (connect-runtime and kafka-tools are two such examples). A number of modules, including kafka-clients, leave it to the application to specify the logging backend. More information can be found at reload4j. Projects that depend on the affected modules from the Kafka project should use slf4j-log4j12 version 1.7.35 or above or slf4j-reload4j to avoid possible compatibility issues originating from the logging framework.
  • The example connectors, FileStreamSourceConnector and FileStreamSinkConnector, have been removed from the default classpath. To use them in Kafka Connect standalone or distributed mode they need to be explicitly added, for example CLASSPATH=./libs/connect-file-3.2.0.jar bin/connect-distributed.sh.

Upgrading to 3.1.0 from any version 0.8.x through 3.0.x

If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 3.0, 2.8, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 3.1.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.1 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 3.1.1
  • Idempotence for the producer is enabled by default if no conflicting configurations are set. When producing to brokers older than 2.8.0, the IDEMPOTENT_WRITE permission is required. Check the compatibility section of KIP-679 for details. A bug prevented the producer idempotence default from being applied which meant that it remained disabled unless the user had explicitly set enable.idempotence to true. See KAFKA-13598 for more details. This issue was fixed and the default is properly applied.
  • A notable exception is Connect that by default disables idempotent behavior for all of its producers in order to uniformly support using a wide range of Kafka broker versions. Users can change this behavior to enable idempotence for some or all producers via Connect worker and/or connector configuration. Connect may enable idempotent producers by default in a future major release.
  • Kafka has replaced log4j with reload4j due to security concerns. This only affects modules that specify a logging backend (connect-runtime and kafka-tools are two such examples). A number of modules, including kafka-clients, leave it to the application to specify the logging backend. More information can be found at reload4j. Projects that depend on the affected modules from the Kafka project should use slf4j-log4j12 version 1.7.35 or above or slf4j-reload4j to avoid possible compatibility issues originating from the logging framework.
Notable changes in 3.1.0
  • Apache Kafka supports Java 17.
  • The following metrics have been deprecated: bufferpool-wait-time-total, io-waittime-total, and iotime-total. Please use bufferpool-wait-time-ns-total, io-wait-time-ns-total, and io-time-ns-total instead. See KIP-773 for more details.
  • IBP 3.1 introduces topic IDs to FetchRequest as a part of KIP-516.

Upgrading to 3.0.1 from any version 0.8.x through 2.8.x

If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 2.8, 2.7, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 3.0.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.0 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 3.0.1
  • Idempotence for the producer is enabled by default if no conflicting configurations are set. When producing to brokers older than 2.8.0, the IDEMPOTENT_WRITE permission is required. Check the compatibility section of KIP-679 for details. A bug prevented the producer idempotence default from being applied which meant that it remained disabled unless the user had explicitly set enable.idempotence to true. See KAFKA-13598 for more details. This issue was fixed and the default is properly applied.
Notable changes in 3.0.0
  • The producer has stronger delivery guarantees by default: idempotence is enabled and acks is set to all instead of 1. See KIP-679 for details. In 3.0.0 and 3.1.0, a bug prevented the idempotence default from being applied which meant that it remained disabled unless the user had explicitly set enable.idempotence to true. Note that the bug did not affect the acks=all change. See KAFKA-13598 for more details. This issue was fixed and the default is properly applied in 3.0.1, 3.1.1, and 3.2.0.
  • Java 8 and Scala 2.12 support have been deprecated since Apache Kafka 3.0 and will be removed in Apache Kafka 4.0. See KIP-750 and KIP-751 for more details.
  • ZooKeeper has been upgraded to version 3.6.3.
  • A preview of KRaft mode is available, though upgrading to it from the 2.8 Early Access release is not possible. See the KRaft section for details.
  • The release tarball no longer includes test, sources, javadoc and test sources jars. These are still published to the Maven Central repository.
  • A number of implementation dependency jars are now available in the runtime classpath instead of compile and runtime classpaths. Compilation errors after the upgrade can be fixed by adding the missing dependency jar(s) explicitly or updating the application not to use internal classes.
  • The default value for the consumer configuration session.timeout.ms was increased from 10s to 45s. See KIP-735 for more details.
  • The broker configuration log.message.format.version and topic configuration message.format.version have been deprecated. The value of both configurations is always assumed to be 3.0 if inter.broker.protocol.version is 3.0 or higher. If log.message.format.version or message.format.version are set, we recommend clearing them at the same time as the inter.broker.protocol.version upgrade to 3.0. This will avoid potential compatibility issues if the inter.broker.protocol.version is downgraded. See KIP-724 for more details.
  • The Streams API removed all deprecated APIs that were deprecated in version 2.5.0 or earlier. For a complete list of removed APIs compare the detailed Kafka Streams upgrade notes.
  • Kafka Streams no longer has a compile time dependency on "connect:json" module (KAFKA-5146). Projects that were relying on this transitive dependency will have to explicitly declare it.
  • Custom principal builder implementations specified through principal.builder.class must now implement the KafkaPrincipalSerde interface to allow for forwarding between brokers. See KIP-590 for more details about the usage of KafkaPrincipalSerde.
  • A number of deprecated classes, methods and tools have been removed from the clients, connect, core and tools modules:
    • The Scala Authorizer, SimpleAclAuthorizer and related classes have been removed. Please use the Java Authorizer and AclAuthorizer instead.
    • The Metric#value() method was removed (KAFKA-12573).
    • The Sum and Total classes were removed (KAFKA-12584). Please use WindowedSum and CumulativeSum instead.
    • The Count and SampledTotal classes were removed. Please use WindowedCount and WindowedSum respectively instead.
    • The PrincipalBuilder, DefaultPrincipalBuilder and ResourceFilter classes were removed.
    • Various constants and constructors were removed from SslConfigs, SaslConfigs, AclBinding and AclBindingFilter.
    • The Admin.electedPreferredLeaders() methods were removed. Please use Admin.electLeaders instead.
    • The kafka-preferred-replica-election command line tool was removed. Please use kafka-leader-election instead.
    • The --zookeeper option was removed from the kafka-topics and kafka-reassign-partitions command line tools. Please use --bootstrap-server instead.
    • In the kafka-configs command line tool, the --zookeeper option is only supported for updating SCRAM Credentials configuration and describing/updating dynamic broker configs when brokers are not running. Please use --bootstrap-server for other configuration operations.
    • The ConfigEntry constructor was removed (KAFKA-12577). Please use the remaining public constructor instead.
    • The config value default for the client config client.dns.lookup has been removed. In the unlikely event that you set this config explicitly, we recommend leaving the config unset (use_all_dns_ips is used by default).
    • The ExtendedDeserializer and ExtendedSerializer classes have been removed. Please use Deserializer and Serializer instead.
    • The close(long, TimeUnit) method was removed from the producer, consumer and admin client. Please use close(Duration).
    • The ConsumerConfig.addDeserializerToConfig and ProducerConfig.addSerializerToConfig methods were removed. These methods were not intended to be public API and there is no replacement.
    • The NoOffsetForPartitionException.partition() method was removed. Please use partitions() instead.
    • The default partition.assignment.strategy is changed to "[RangeAssignor, CooperativeStickyAssignor]", which will use the RangeAssignor by default, but allows upgrading to the CooperativeStickyAssignor with just a single rolling bounce that removes the RangeAssignor from the list. Please check the client upgrade path guide here for more detail.
    • The Scala kafka.common.MessageFormatter was removed. Please use the Java org.apache.kafka.common.MessageFormatter.
    • The MessageFormatter.init(Properties) method was removed. Please use configure(Map) instead.
    • The checksum() method has been removed from ConsumerRecord and RecordMetadata. The message format v2, which has been the default since 0.11, moved the checksum from the record to the record batch. As such, these methods don't make sense and no replacements exist.
    • The ChecksumMessageFormatter class was removed. It is not part of the public API, but it may have been used with kafka-console-consumer.sh. It reported the checksum of each record, which has not been supported since message format v2.
    • The org.apache.kafka.clients.consumer.internals.PartitionAssignor class has been removed. Please use org.apache.kafka.clients.consumer.ConsumerPartitionAssignor instead.
    • The quota.producer.default and quota.consumer.default configurations were removed (KAFKA-12591). Dynamic quota defaults must be used instead.
    • The port and host.name configurations were removed. Please use listeners instead.
    • The advertised.port and advertised.host.name configurations were removed. Please use advertised.listeners instead.
    • The deprecated worker configurations rest.host.name and rest.port were removed (KAFKA-12482) from the Kafka Connect worker configuration. Please use listeners instead.
  • The Producer#sendOffsetsToTransaction(Map offsets, String consumerGroupId) method has been deprecated. Please use Producer#sendOffsetsToTransaction(Map offsets, ConsumerGroupMetadata metadata) instead, where the ConsumerGroupMetadata can be retrieved via KafkaConsumer#groupMetadata() for stronger semantics. Note that the full set of consumer group metadata is only understood by brokers or version 2.5 or higher, so you must upgrade your kafka cluster to get the stronger semantics. Otherwise, you can just pass in new ConsumerGroupMetadata(consumerGroupId) to work with older brokers. See KIP-732 for more details.
  • The Connect internal.key.converter and internal.value.converter properties have been completely removed. The use of these Connect worker properties has been deprecated since version 2.0.0. Workers are now hardcoded to use the JSON converter with schemas.enable set to false. If your cluster has been using a different internal key or value converter, you can follow the migration steps outlined in KIP-738 to safely upgrade your Connect cluster to 3.0.
  • The Connect-based MirrorMaker (MM2) includes changes to support IdentityReplicationPolicy, enabling replication without renaming topics. The existing DefaultReplicationPolicy is still used by default, but identity replication can be enabled via the replication.policy configuration property. This is especially useful for users migrating from the older MirrorMaker (MM1), or for use-cases with simple one-way replication topologies where topic renaming is undesirable. Note that IdentityReplicationPolicy, unlike DefaultReplicationPolicy, cannot prevent replication cycles based on topic names, so take care to avoid cycles when constructing your replication topology.
  • The original MirrorMaker (MM1) and related classes have been deprecated. Please use the Connect-based MirrorMaker (MM2), as described in the Geo-Replication section.

Upgrading to 2.8.1 from any version 0.8.x through 2.7.x

If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 2.7, 2.6, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 2.8.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.8 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 2.8.0
  • The 2.8.0 release added a new method to the Authorizer Interface introduced in KIP-679. The motivation is to unblock our future plan to enable the strongest message delivery guarantee by default. Custom authorizer should consider providing a more efficient implementation that supports audit logging and any custom configs or access rules.
  • IBP 2.8 introduces topic IDs to topics as a part of KIP-516. When using ZooKeeper, this information is stored in the TopicZNode. If the cluster is downgraded to a previous IBP or version, future topics will not get topic IDs and it is not guaranteed that topics will retain their topic IDs in ZooKeeper. This means that upon upgrading again, some topics or all topics will be assigned new IDs.
  • Kafka Streams introduce a type-safe split() operator as a substitution for deprecated KStream#branch() method (cf. KIP-418).

Upgrading to 2.7.0 from any version 0.8.x through 2.6.x

If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 2.6, 2.5, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 2.7.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.7 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 2.7.0
  • The 2.7.0 release includes the core Raft implementation specified in KIP-595. There is a separate "raft" module containing most of the logic. Until integration with the controller is complete, there is a standalone server that users can use for testing the performance of the Raft implementation. See the README.md in the raft module for details
  • KIP-651 adds support for using PEM files for key and trust stores.
  • KIP-612 adds support for enforcing broker-wide and per-listener connection create rates. The 2.7.0 release contains the first part of KIP-612 with dynamic configuration coming in the 2.8.0 release.
  • The ability to throttle topic and partition creations or topics deletions to prevent a cluster from being harmed via KIP-599
  • When new features become available in Kafka there are two main issues:
    1. How do Kafka clients become aware of broker capabilities?
    2. How does the broker decide which features to enable?
    KIP-584 provides a flexible and operationally easy solution for client discovery, feature gating and rolling upgrades using a single restart.
  • The ability to print record offsets and headers with the ConsoleConsumer is now possible via KIP-431
  • The addition of KIP-554 continues progress towards the goal of Zookeeper removal from Kafka. The addition of KIP-554 means you don't have to connect directly to ZooKeeper anymore for managing SCRAM credentials.
  • Altering non-reconfigurable configs of existent listeners causes InvalidRequestException. By contrast, the previous (unintended) behavior would have caused the updated configuration to be persisted, but it wouldn't take effect until the broker was restarted. See KAFKA-10479 for more discussion. See DynamicBrokerConfig.DynamicSecurityConfigs and SocketServer.ListenerReconfigurableConfigs for the supported reconfigurable configs of existent listeners.
  • Kafka Streams adds support for Sliding Windows Aggregations in the KStreams DSL.
  • Reverse iteration over state stores enabling more efficient most recent update searches with KIP-617
  • End-to-End latency metrics in Kafka Steams see KIP-613 for more details
  • Kafka Streams added metrics reporting default RocksDB properties with KIP-607
  • Better Scala implicit Serdes support from KIP-616

Upgrading to 2.6.0 from any version 0.8.x through 2.5.x

If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 2.5, 2.4, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 2.6.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.6 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 2.6.0
  • Kafka Streams adds a new processing mode (requires broker 2.5 or newer) that improves application scalability using exactly-once guarantees (cf. KIP-447)
  • TLSv1.3 has been enabled by default for Java 11 or newer. The client and server will negotiate TLSv1.3 if both support it and fallback to TLSv1.2 otherwise. See KIP-573 for more details.
  • The default value for the client.dns.lookup configuration has been changed from default to use_all_dns_ips. If a hostname resolves to multiple IP addresses, clients and brokers will now attempt to connect to each IP in sequence until the connection is successfully established. See KIP-602 for more details.
  • NotLeaderForPartitionException has been deprecated and replaced with NotLeaderOrFollowerException. Fetch requests and other requests intended only for the leader or follower return NOT_LEADER_OR_FOLLOWER(6) instead of REPLICA_NOT_AVAILABLE(9) if the broker is not a replica, ensuring that this transient error during reassignments is handled by all clients as a retriable exception.

Upgrading to 2.5.0 from any version 0.8.x through 2.4.x

If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 2.4, 2.3, etc.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 2.5.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.5 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
  6. There are several notable changes to the reassignment tool kafka-reassign-partitions.sh following the completion of KIP-455. This tool now requires the --additional flag to be provided when changing the throttle of an active reassignment. Reassignment cancellation is now possible using the --cancel command. Finally, reassignment with --zookeeper has been deprecated in favor of --bootstrap-server. See the KIP for more detail.
Notable changes in 2.5.0
  • When RebalanceProtocol#COOPERATIVE is used, Consumer#poll can still return data while it is in the middle of a rebalance for those partitions still owned by the consumer; in addition Consumer#commitSync now may throw a non-fatal RebalanceInProgressException to notify users of such an event, in order to distinguish from the fatal CommitFailedException and allow users to complete the ongoing rebalance and then reattempt committing offsets for those still-owned partitions.
  • For improved resiliency in typical network environments, the default value of zookeeper.session.timeout.ms has been increased from 6s to 18s and replica.lag.time.max.ms from 10s to 30s.
  • New DSL operator cogroup() has been added for aggregating multiple streams together at once.
  • Added a new KStream.toTable() API to translate an input event stream into a KTable.
  • Added a new Serde type Void to represent null keys or null values from input topic.
  • Deprecated UsePreviousTimeOnInvalidTimestamp and replaced it with UsePartitionTimeOnInvalidTimeStamp.
  • Improved exactly-once semantics by adding a pending offset fencing mechanism and stronger transactional commit consistency check, which greatly simplifies the implementation of a scalable exactly-once application. We also added a new exactly-once semantics code example under examples folder. Check out KIP-447 for the full details.
  • Added a new public api KafkaStreams.queryMetadataForKey(String, K, Serializer) to get detailed information on the key being queried. It provides information about the partition number where the key resides in addition to hosts containing the active and standby partitions for the key.
  • Provided support to query stale stores (for high availability) and the stores belonging to a specific partition by deprecating KafkaStreams.store(String, QueryableStoreType) and replacing it with KafkaStreams.store(StoreQueryParameters).
  • Added a new public api to access lag information for stores local to an instance with KafkaStreams.allLocalStorePartitionLags().
  • Scala 2.11 is no longer supported. See KIP-531 for details.
  • All Scala classes from the package kafka.security.auth have been deprecated. See KIP-504 for details of the new Java authorizer API added in 2.4.0. Note that kafka.security.auth.Authorizer and kafka.security.auth.SimpleAclAuthorizer were deprecated in 2.4.0.
  • TLSv1 and TLSv1.1 have been disabled by default since these have known security vulnerabilities. Only TLSv1.2 is now enabled by default. You can continue to use TLSv1 and TLSv1.1 by explicitly enabling these in the configuration options ssl.protocol and ssl.enabled.protocols.
  • ZooKeeper has been upgraded to 3.5.7, and a ZooKeeper upgrade from 3.4.X to 3.5.7 can fail if there are no snapshot files in the 3.4 data directory. This usually happens in test upgrades where ZooKeeper 3.5.7 is trying to load an existing 3.4 data dir in which no snapshot file has been created. For more details about the issue please refer to ZOOKEEPER-3056. A fix is given in ZOOKEEPER-3056, which is to set snapshot.trust.empty=true config in zookeeper.properties before the upgrade.
  • ZooKeeper version 3.5.7 supports TLS-encrypted connectivity to ZooKeeper both with or without client certificates, and additional Kafka configurations are available to take advantage of this. See KIP-515 for details.

Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, 1.1.x, 2.0.x or 2.1.x or 2.2.x or 2.3.x to 2.4.0

If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.10.0, 0.11.0, 1.0, 2.0, 2.2).
    • log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
    If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0, 1.0, 1.1, 2.0, 2.1, 2.2, 2.3).
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 2.4.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.4 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.

Additional Upgrade Notes:

  1. ZooKeeper has been upgraded to 3.5.6. ZooKeeper upgrade from 3.4.X to 3.5.6 can fail if there are no snapshot files in 3.4 data directory. This usually happens in test upgrades where ZooKeeper 3.5.6 is trying to load an existing 3.4 data dir in which no snapshot file has been created. For more details about the issue please refer to ZOOKEEPER-3056. A fix is given in ZOOKEEPER-3056, which is to set snapshot.trust.empty=true config in zookeeper.properties before the upgrade. But we have observed data loss in standalone cluster upgrades when using snapshot.trust.empty=true config. For more details about the issue please refer to ZOOKEEPER-3644. So we recommend the safe workaround of copying empty snapshot file to the 3.4 data directory, if there are no snapshot files in 3.4 data directory. For more details about the workaround please refer to ZooKeeper Upgrade FAQ.
  2. An embedded Jetty based AdminServer added in ZooKeeper 3.5. AdminServer is enabled by default in ZooKeeper and is started on port 8080. AdminServer is disabled by default in the ZooKeeper config (zookeeper.properties) provided by the Apache Kafka distribution. Make sure to update your local zookeeper.properties file with admin.enableServer=false if you wish to disable the AdminServer. Please refer AdminServer config to configure the AdminServer.
Notable changes in 2.4.0
  • A new Admin API has been added for partition reassignments. Due to changing the way Kafka propagates reassignment information, it is possible to lose reassignment state in failure edge cases while upgrading to the new version. It is not recommended to start reassignments while upgrading.
  • ZooKeeper has been upgraded from 3.4.14 to 3.5.6. TLS and dynamic reconfiguration are supported by the new version.
  • The bin/kafka-preferred-replica-election.sh command line tool has been deprecated. It has been replaced by bin/kafka-leader-election.sh.
  • The methods electPreferredLeaders in the Java AdminClient class have been deprecated in favor of the methods electLeaders.
  • Scala code leveraging the NewTopic(String, int, short) constructor with literal values will need to explicitly call toShort on the second literal.
  • The argument in the constructor GroupAuthorizationException(String) is now used to specify an exception message. Previously it referred to the group that failed authorization. This was done for consistency with other exception types and to avoid potential misuse. The constructor TopicAuthorizationException(String) which was previously used for a single unauthorized topic was changed similarly.
  • The internal PartitionAssignor interface has been deprecated and replaced with a new ConsumerPartitionAssignor in the public API. Some methods/signatures are slightly different between the two interfaces. Users implementing a custom PartitionAssignor should migrate to the new interface as soon as possible.
  • The DefaultPartitioner now uses a sticky partitioning strategy. This means that records for specific topic with null keys and no assigned partition will be sent to the same partition until the batch is ready to be sent. When a new batch is created, a new partition is chosen. This decreases latency to produce, but it may result in uneven distribution of records across partitions in edge cases. Generally users will not be impacted, but this difference may be noticeable in tests and other situations producing records for a very short amount of time.
  • The blocking KafkaConsumer#committed methods have been extended to allow a list of partitions as input parameters rather than a single partition. It enables fewer request/response iterations between clients and brokers fetching for the committed offsets for the consumer group. The old overloaded functions are deprecated and we would recommend users to make their code changes to leverage the new methods (details can be found in KIP-520).
  • We've introduced a new INVALID_RECORD error in the produce response to distinguish from the CORRUPT_MESSAGE error. To be more concrete, previously when a batch of records was sent as part of a single request to the broker and one or more of the records failed the validation due to various causes (mismatch magic bytes, crc checksum errors, null key for log compacted topics, etc), the whole batch would be rejected with the same and misleading CORRUPT_MESSAGE, and the caller of the producer client would see the corresponding exception from either the future object of RecordMetadata returned from the send call as well as in the Callback#onCompletion(RecordMetadata metadata, Exception exception) Now with the new error code and improved error messages of the exception, producer callers would be better informed about the root cause why their sent records were failed.
  • We are introducing incremental cooperative rebalancing to the clients' group protocol, which allows consumers to keep all of their assigned partitions during a rebalance and at the end revoke only those which must be migrated to another consumer for overall cluster balance. The ConsumerCoordinator will choose the latest RebalanceProtocol that is commonly supported by all of the consumer's supported assignors. You can use the new built-in CooperativeStickyAssignor or plug in your own custom cooperative assignor. To do so you must implement the ConsumerPartitionAssignor interface and include RebalanceProtocol.COOPERATIVE in the list returned by ConsumerPartitionAssignor#supportedProtocols. Your custom assignor can then leverage the ownedPartitions field in each consumer's Subscription to give partitions back to their previous owners whenever possible. Note that when a partition is to be reassigned to another consumer, it must be removed from the new assignment until it has been revoked from its original owner. Any consumer that has to revoke a partition will trigger a followup rebalance to allow the revoked partition to safely be assigned to its new owner. See the ConsumerPartitionAssignor RebalanceProtocol javadocs for more information.
    To upgrade from the old (eager) protocol, which always revokes all partitions before rebalancing, to cooperative rebalancing, you must follow a specific upgrade path to get all clients on the same ConsumerPartitionAssignor that supports the cooperative protocol. This can be done with two rolling bounces, using the CooperativeStickyAssignor for the example: during the first one, add "cooperative-sticky" to the list of supported assignors for each member (without removing the previous assignor -- note that if previously using the default, you must include that explicitly as well). You then bounce and/or upgrade it. Once the entire group is on 2.4+ and all members have the "cooperative-sticky" among their supported assignors, remove the other assignor(s) and perform a second rolling bounce so that by the end all members support only the cooperative protocol. For further details on the cooperative rebalancing protocol and upgrade path, see KIP-429.
  • There are some behavioral changes to the ConsumerRebalanceListener, as well as a new API. Exceptions thrown during any of the listener's three callbacks will no longer be swallowed, and will instead be re-thrown all the way up to the Consumer.poll() call. The onPartitionsLost method has been added to allow users to react to abnormal circumstances where a consumer may have lost ownership of its partitions (such as a missed rebalance) and cannot commit offsets. By default, this will simply call the existing onPartitionsRevoked API to align with previous behavior. Note however that onPartitionsLost will not be called when the set of lost partitions is empty. This means that no callback will be invoked at the beginning of the first rebalance of a new consumer joining the group.
    The semantics of the ConsumerRebalanceListener's callbacks are further changed when following the cooperative rebalancing protocol described above. In addition to onPartitionsLost, onPartitionsRevoked will also never be called when the set of revoked partitions is empty. The callback will generally be invoked only at the end of a rebalance, and only on the set of partitions that are being moved to another consumer. The onPartitionsAssigned callback will however always be called, even with an empty set of partitions, as a way to notify users of a rebalance event (this is true for both cooperative and eager). For details on the new callback semantics, see the ConsumerRebalanceListener javadocs.
  • The Scala trait kafka.security.auth.Authorizer has been deprecated and replaced with a new Java API org.apache.kafka.server.authorizer.Authorizer. The authorizer implementation class kafka.security.auth.SimpleAclAuthorizer has also been deprecated and replaced with a new implementation kafka.security.authorizer.AclAuthorizer. AclAuthorizer uses features supported by the new API to improve authorization logging and is compatible with SimpleAclAuthorizer. For more details, see KIP-504.

Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, 1.1.x, 2.0.x or 2.1.x or 2.2.x to 2.3.0

If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0, 1.1).
    • log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
    If you are upgrading from 0.11.0.x, 1.0.x, 1.1.x, 2.0.x, or 2.1.x, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0, 1.0, 1.1, 2.0, 2.1, 2.2).
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 2.3.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.3 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 2.3.0
  • We are introducing a new rebalancing protocol for Kafka Connect based on incremental cooperative rebalancing. The new protocol does not require stopping all the tasks during a rebalancing phase between Connect workers. Instead, only the tasks that need to be exchanged between workers are stopped and they are started in a follow up rebalance. The new Connect protocol is enabled by default beginning with 2.3.0. For more details on how it works and how to enable the old behavior of eager rebalancing, checkout incremental cooperative rebalancing design.
  • We are introducing static membership towards consumer user. This feature reduces unnecessary rebalances during normal application upgrades or rolling bounces. For more details on how to use it, checkout static membership design.
  • Kafka Streams DSL switches its used store types. While this change is mainly transparent to users, there are some corner cases that may require code changes. See the Kafka Streams upgrade section for more details.
  • Kafka Streams 2.3.0 requires 0.11 message format or higher and does not work with older message format.

Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, 1.1.x, 2.0.x or 2.1.x to 2.2.0

If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0, 1.1).
    • log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
    If you are upgrading from 0.11.0.x, 1.0.x, 1.1.x, or 2.0.x and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0, 1.0, 1.1, 2.0).
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 2.2.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.2 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 2.2.1
  • Kafka Streams 2.2.1 requires 0.11 message format or higher and does not work with older message format.
Notable changes in 2.2.0
  • The default consumer group id has been changed from the empty string ("") to null. Consumers who use the new default group id will not be able to subscribe to topics, and fetch or commit offsets. The empty string as consumer group id is deprecated but will be supported until a future major release. Old clients that rely on the empty string group id will now have to explicitly provide it as part of their consumer config. For more information see KIP-289.
  • The bin/kafka-topics.sh command line tool is now able to connect directly to brokers with --bootstrap-server instead of zookeeper. The old --zookeeper option is still available for now. Please read KIP-377 for more information.
  • Kafka Streams depends on a newer version of RocksDBs that requires MacOS 10.13 or higher.

Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, 1.1.x, or 2.0.0 to 2.1.0

Note that 2.1.x contains a change to the internal schema used to store consumer offsets. Once the upgrade is complete, it will not be possible to downgrade to previous versions. See the rolling upgrade notes below for more detail.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0, 1.1).
    • log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
    If you are upgrading from 0.11.0.x, 1.0.x, 1.1.x, or 2.0.x and you have not overridden the message format, then you only need to override the inter-broker protocol version.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0, 1.0, 1.1, 2.0).
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
  3. Once the cluster's behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 2.1.
  4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.1 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.

Additional Upgrade Notes:

  1. Offset expiration semantics has slightly changed in this version. According to the new semantics, offsets of partitions in a group will not be removed while the group is subscribed to the corresponding topic and is still active (has active consumers). If group becomes empty all its offsets will be removed after default offset retention period (or the one set by broker) has passed (unless the group becomes active again). Offsets associated with standalone (simple) consumers, that do not use Kafka group management, will be removed after default offset retention period (or the one set by broker) has passed since their last commit.
  2. The default for console consumer's enable.auto.commit property when no group.id is provided is now set to false. This is to avoid polluting the consumer coordinator cache as the auto-generated group is not likely to be used by other consumers.
  3. The default value for the producer's retries config was changed to Integer.MAX_VALUE, as we introduced delivery.timeout.ms in KIP-91, which sets an upper bound on the total time between sending a record and receiving acknowledgement from the broker. By default, the delivery timeout is set to 2 minutes.
  4. By default, MirrorMaker now overrides delivery.timeout.ms to Integer.MAX_VALUE when configuring the producer. If you have overridden the value of retries in order to fail faster, you will instead need to override delivery.timeout.ms.
  5. The ListGroup API now expects, as a recommended alternative, Describe Group access to the groups a user should be able to list. Even though the old Describe Cluster access is still supported for backward compatibility, using it for this API is not advised.
  6. KIP-336 deprecates the ExtendedSerializer and ExtendedDeserializer interfaces and propagates the usage of Serializer and Deserializer. ExtendedSerializer and ExtendedDeserializer were introduced with KIP-82 to provide record headers for serializers and deserializers in a Java 7 compatible fashion. Now we consolidated these interfaces as Java 7 support has been dropped since.
Notable changes in 2.1.0
  • Jetty has been upgraded to 9.4.12, which excludes TLS_RSA_* ciphers by default because they do not support forward secrecy, see https://github.com/eclipse/jetty.project/issues/2807 for more information.
  • Unclean leader election is automatically enabled by the controller when unclean.leader.election.enable config is dynamically updated by using per-topic config override.
  • The AdminClient has added a method AdminClient#metrics(). Now any application using the AdminClient can gain more information and insight by viewing the metrics captured from the AdminClient. For more information see KIP-324
  • Kafka now supports Zstandard compression from KIP-110. You must upgrade the broker as well as clients to make use of it. Consumers prior to 2.1.0 will not be able to read from topics which use Zstandard compression, so you should not enable it for a topic until all downstream consumers are upgraded. See the KIP for more detail.

Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, or 1.1.x to 2.0.0

Kafka 2.0.0 introduces wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please review the notable changes in 2.0.0 before upgrading.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0, 1.1).
    • log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
    If you are upgrading from 0.11.0.x, 1.0.x, or 1.1.x and you have not overridden the message format, then you only need to override the inter-broker protocol format.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0, 1.0, 1.1).
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.
  3. Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 2.0.
  4. Restart the brokers one by one for the new protocol version to take effect.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.0 on each broker and restart them one by one. Note that the older Scala consumer does not support the new message format introduced in 0.11, so to avoid the performance cost of down-conversion (or to take advantage of exactly once semantics), the newer Java consumer must be used.

Additional Upgrade Notes:

  1. If you are willing to accept downtime, you can simply take all the brokers down, update the code and start them back up. They will start with the new protocol by default.
  2. Bumping the protocol version and restarting can be done any time after the brokers are upgraded. It does not have to be immediately after. Similarly for the message format version.
  3. If you are using Java8 method references in your Kafka Streams code you might need to update your code to resolve method ambiguities. Hot-swapping the jar-file only might not work.
  4. ACLs should not be added to prefixed resources, (added in KIP-290), until all brokers in the cluster have been updated.

    NOTE: any prefixed ACLs added to a cluster, even after the cluster is fully upgraded, will be ignored should the cluster be downgraded again.

Notable changes in 2.0.0
  • KIP-186 increases the default offset retention time from 1 day to 7 days. This makes it less likely to "lose" offsets in an application that commits infrequently. It also increases the active set of offsets and therefore can increase memory usage on the broker. Note that the console consumer currently enables offset commit by default and can be the source of a large number of offsets which this change will now preserve for 7 days instead of 1. You can preserve the existing behavior by setting the broker config offsets.retention.minutes to 1440.
  • Support for Java 7 has been dropped, Java 8 is now the minimum version required.
  • The default value for ssl.endpoint.identification.algorithm was changed to https, which performs hostname verification (man-in-the-middle attacks are possible otherwise). Set ssl.endpoint.identification.algorithm to an empty string to restore the previous behaviour.
  • KAFKA-5674 extends the lower interval of max.connections.per.ip minimum to zero and therefore allows IP-based filtering of inbound connections.
  • KIP-272 added API version tag to the metric kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower|...}. This metric now becomes kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower|...},version={0|1|2|3|...}. This will impact JMX monitoring tools that do not automatically aggregate. To get the total count for a specific request type, the tool needs to be updated to aggregate across different versions.
  • KIP-225 changed the metric "records.lag" to use tags for topic and partition. The original version with the name format "{topic}-{partition}.records-lag" has been removed.
  • The Scala consumers, which have been deprecated since 0.11.0.0, have been removed. The Java consumer has been the recommended option since 0.10.0.0. Note that the Scala consumers in 1.1.0 (and older) will continue to work even if the brokers are upgraded to 2.0.0.
  • The Scala producers, which have been deprecated since 0.10.0.0, have been removed. The Java producer has been the recommended option since 0.9.0.0. Note that the behaviour of the default partitioner in the Java producer differs from the default partitioner in the Scala producers. Users migrating should consider configuring a custom partitioner that retains the previous behaviour. Note that the Scala producers in 1.1.0 (and older) will continue to work even if the brokers are upgraded to 2.0.0.
  • MirrorMaker and ConsoleConsumer no longer support the Scala consumer, they always use the Java consumer.
  • The ConsoleProducer no longer supports the Scala producer, it always uses the Java producer.
  • A number of deprecated tools that rely on the Scala clients have been removed: ReplayLogProducer, SimpleConsumerPerformance, SimpleConsumerShell, ExportZkOffsets, ImportZkOffsets, UpdateOffsetsInZK, VerifyConsumerRebalance.
  • The deprecated kafka.tools.ProducerPerformance has been removed, please use org.apache.kafka.tools.ProducerPerformance.
  • New Kafka Streams configuration parameter upgrade.from added that allows rolling bounce upgrade from older version.
  • KIP-284 changed the retention time for Kafka Streams repartition topics by setting its default value to Long.MAX_VALUE.
  • Updated ProcessorStateManager APIs in Kafka Streams for registering state stores to the processor topology. For more details please read the Streams Upgrade Guide.
  • In earlier releases, Connect's worker configuration required the internal.key.converter and internal.value.converter properties. In 2.0, these are no longer required and default to the JSON converter. You may safely remove these properties from your Connect standalone and distributed worker configurations:
    internal.key.converter=org.apache.kafka.connect.json.JsonConverter internal.key.converter.schemas.enable=false internal.value.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter.schemas.enable=false
  • KIP-266 adds a new consumer configuration default.api.timeout.ms to specify the default timeout to use for KafkaConsumer APIs that could block. The KIP also adds overloads for such blocking APIs to support specifying a specific timeout to use for each of them instead of using the default timeout set by default.api.timeout.ms. In particular, a new poll(Duration) API has been added which does not block for dynamic partition assignment. The old poll(long) API has been deprecated and will be removed in a future version. Overloads have also been added for other KafkaConsumer methods like partitionsFor, listTopics, offsetsForTimes, beginningOffsets, endOffsets and close that take in a Duration.
  • Also as part of KIP-266, the default value of request.timeout.ms has been changed to 30 seconds. The previous value was a little higher than 5 minutes to account for maximum time that a rebalance would take. Now we treat the JoinGroup request in the rebalance as a special case and use a value derived from max.poll.interval.ms for the request timeout. All other request types use the timeout defined by request.timeout.ms
  • The internal method kafka.admin.AdminClient.deleteRecordsBefore has been removed. Users are encouraged to migrate to org.apache.kafka.clients.admin.AdminClient.deleteRecords.
  • The AclCommand tool --producer convenience option uses the KIP-277 finer grained ACL on the given topic.
  • KIP-176 removes the --new-consumer option for all consumer based tools. This option is redundant since the new consumer is automatically used if --bootstrap-server is defined.
  • KIP-290 adds the ability to define ACLs on prefixed resources, e.g. any topic starting with 'foo'.
  • KIP-283 improves message down-conversion handling on Kafka broker, which has typically been a memory-intensive operation. The KIP adds a mechanism by which the operation becomes less memory intensive by down-converting chunks of partition data at a time which helps put an upper bound on memory consumption. With this improvement, there is a change in FetchResponse protocol behavior where the broker could send an oversized message batch towards the end of the response with an invalid offset. Such oversized messages must be ignored by consumer clients, as is done by KafkaConsumer.

    KIP-283 also adds new topic and broker configurations message.downconversion.enable and log.message.downconversion.enable respectively to control whether down-conversion is enabled. When disabled, broker does not perform any down-conversion and instead sends an UNSUPPORTED_VERSION error to the client.

  • Dynamic broker configuration options can be stored in ZooKeeper using kafka-configs.sh before brokers are started. This option can be used to avoid storing clear passwords in server.properties as all password configs may be stored encrypted in ZooKeeper.
  • ZooKeeper hosts are now re-resolved if connection attempt fails. But if your ZooKeeper host names resolve to multiple addresses and some of them are not reachable, then you may need to increase the connection timeout zookeeper.connection.timeout.ms.
New Protocol Versions
  • KIP-279: OffsetsForLeaderEpochResponse v1 introduces a partition-level leader_epoch field.
  • KIP-219: Bump up the protocol versions of non-cluster action requests and responses that are throttled on quota violation.
  • KIP-290: Bump up the protocol versions ACL create, describe and delete requests and responses.
Upgrading a 1.1 Kafka Streams Application
  • Upgrading your Streams application from 1.1 to 2.0 does not require a broker upgrade. A Kafka Streams 2.0 application can connect to 2.0, 1.1, 1.0, 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
  • Note that in 2.0 we have removed the public APIs that are deprecated prior to 1.0; users leveraging on those deprecated APIs need to make code changes accordingly. See Streams API changes in 2.0.0 for more details.

Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, or 1.0.x to 1.1.x

Kafka 1.1.0 introduces wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please review the notable changes in 1.1.0 before upgrading.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0).
    • log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
    If you are upgrading from 0.11.0.x or 1.0.x and you have not overridden the message format, then you only need to override the inter-broker protocol format.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0 or 1.0).
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.
  3. Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 1.1.
  4. Restart the brokers one by one for the new protocol version to take effect.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 1.1 on each broker and restart them one by one. Note that the older Scala consumer does not support the new message format introduced in 0.11, so to avoid the performance cost of down-conversion (or to take advantage of exactly once semantics), the newer Java consumer must be used.

Additional Upgrade Notes:

  1. If you are willing to accept downtime, you can simply take all the brokers down, update the code and start them back up. They will start with the new protocol by default.
  2. Bumping the protocol version and restarting can be done any time after the brokers are upgraded. It does not have to be immediately after. Similarly for the message format version.
  3. If you are using Java8 method references in your Kafka Streams code you might need to update your code to resolve method ambiguties. Hot-swapping the jar-file only might not work.
Notable changes in 1.1.1
  • New Kafka Streams configuration parameter upgrade.from added that allows rolling bounce upgrade from version 0.10.0.x
  • See the Kafka Streams upgrade guide for details about this new config.
Notable changes in 1.1.0
  • The kafka artifact in Maven no longer depends on log4j or slf4j-log4j12. Similarly to the kafka-clients artifact, users can now choose the logging back-end by including the appropriate slf4j module (slf4j-log4j12, logback, etc.). The release tarball still includes log4j and slf4j-log4j12.
  • KIP-225 changed the metric "records.lag" to use tags for topic and partition. The original version with the name format "{topic}-{partition}.records-lag" is deprecated and will be removed in 2.0.0.
  • Kafka Streams is more robust against broker communication errors. Instead of stopping the Kafka Streams client with a fatal exception, Kafka Streams tries to self-heal and reconnect to the cluster. Using the new AdminClient you have better control of how often Kafka Streams retries and can configure fine-grained timeouts (instead of hard coded retries as in older version).
  • Kafka Streams rebalance time was reduced further making Kafka Streams more responsive.
  • Kafka Connect now supports message headers in both sink and source connectors, and to manipulate them via simple message transforms. Connectors must be changed to explicitly use them. A new HeaderConverter is introduced to control how headers are (de)serialized, and the new "SimpleHeaderConverter" is used by default to use string representations of values.
  • kafka.tools.DumpLogSegments now automatically sets deep-iteration option if print-data-log is enabled explicitly or implicitly due to any of the other options like decoder.
New Protocol Versions
  • KIP-226 introduced DescribeConfigs Request/Response v1.
  • KIP-227 introduced Fetch Request/Response v7.
Upgrading a 1.0 Kafka Streams Application
  • Upgrading your Streams application from 1.0 to 1.1 does not require a broker upgrade. A Kafka Streams 1.1 application can connect to 1.0, 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
  • See Streams API changes in 1.1.0 for more details.

Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x or 0.11.0.x to 1.0.0

Kafka 1.0.0 introduces wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please review the notable changes in 1.0.0 before upgrading.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0).
    • log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
    If you are upgrading from 0.11.0.x and you have not overridden the message format, you must set both the message format version and the inter-broker protocol version to 0.11.0.
    • inter.broker.protocol.version=0.11.0
    • log.message.format.version=0.11.0
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.
  3. Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 1.0.
  4. Restart the brokers one by one for the new protocol version to take effect.
  5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 1.0 on each broker and restart them one by one. If you are upgrading from 0.11.0 and log.message.format.version is set to 0.11.0, you can update the config and skip the rolling restart. Note that the older Scala consumer does not support the new message format introduced in 0.11, so to avoid the performance cost of down-conversion (or to take advantage of exactly once semantics), the newer Java consumer must be used.

Additional Upgrade Notes:

  1. If you are willing to accept downtime, you can simply take all the brokers down, update the code and start them back up. They will start with the new protocol by default.
  2. Bumping the protocol version and restarting can be done any time after the brokers are upgraded. It does not have to be immediately after. Similarly for the message format version.
Notable changes in 1.0.2
  • New Kafka Streams configuration parameter upgrade.from added that allows rolling bounce upgrade from version 0.10.0.x
  • See the Kafka Streams upgrade guide for details about this new config.
Notable changes in 1.0.1
  • Restored binary compatibility of AdminClient's Options classes (e.g. CreateTopicsOptions, DeleteTopicsOptions, etc.) with 0.11.0.x. Binary (but not source) compatibility had been broken inadvertently in 1.0.0.
Notable changes in 1.0.0
  • Topic deletion is now enabled by default, since the functionality is now stable. Users who wish to to retain the previous behavior should set the broker config delete.topic.enable to false. Keep in mind that topic deletion removes data and the operation is not reversible (i.e. there is no "undelete" operation)
  • For topics that support timestamp search if no offset can be found for a partition, that partition is now included in the search result with a null offset value. Previously, the partition was not included in the map. This change was made to make the search behavior consistent with the case of topics not supporting timestamp search.
  • If the inter.broker.protocol.version is 1.0 or later, a broker will now stay online to serve replicas on live log directories even if there are offline log directories. A log directory may become offline due to IOException caused by hardware failure. Users need to monitor the per-broker metric offlineLogDirectoryCount to check whether there is offline log directory.
  • Added KafkaStorageException which is a retriable exception. KafkaStorageException will be converted to NotLeaderForPartitionException in the response if the version of the client's FetchRequest or ProducerRequest does not support KafkaStorageException.
  • -XX:+DisableExplicitGC was replaced by -XX:+ExplicitGCInvokesConcurrent in the default JVM settings. This helps avoid out of memory exceptions during allocation of native memory by direct buffers in some cases.
  • The overridden handleError method implementations have been removed from the following deprecated classes in the kafka.api package: FetchRequest, GroupCoordinatorRequest, OffsetCommitRequest, OffsetFetchRequest, OffsetRequest, ProducerRequest, and TopicMetadataRequest. This was only intended for use on the broker, but it is no longer in use and the implementations have not been maintained. A stub implementation has been retained for binary compatibility.
  • The Java clients and tools now accept any string as a client-id.
  • The deprecated tool kafka-consumer-offset-checker.sh has been removed. Use kafka-consumer-groups.sh to get consumer group details.
  • SimpleAclAuthorizer now logs access denials to the authorizer log by default.
  • Authentication failures are now reported to clients as one of the subclasses of AuthenticationException. No retries will be performed if a client connection fails authentication.
  • Custom SaslServer implementations may throw SaslAuthenticationException to provide an error message to return to clients indicating the reason for authentication failure. Implementors should take care not to include any security-critical information in the exception message that should not be leaked to unauthenticated clients.
  • The app-info mbean registered with JMX to provide version and commit id will be deprecated and replaced with metrics providing these attributes.
  • Kafka metrics may now contain non-numeric values. org.apache.kafka.common.Metric#value() has been deprecated and will return 0.0 in such cases to minimise the probability of breaking users who read the value of every client metric (via a MetricsReporter implementation or by calling the metrics() method). org.apache.kafka.common.Metric#metricValue() can be used to retrieve numeric and non-numeric metric values.
  • Every Kafka rate metric now has a corresponding cumulative count metric with the suffix -total to simplify downstream processing. For example, records-consumed-rate has a corresponding metric named records-consumed-total.
  • Mx4j will only be enabled if the system property kafka_mx4jenable is set to true. Due to a logic inversion bug, it was previously enabled by default and disabled if kafka_mx4jenable was set to true.
  • The package org.apache.kafka.common.security.auth in the clients jar has been made public and added to the javadocs. Internal classes which had previously been located in this package have been moved elsewhere.
  • When using an Authorizer and a user doesn't have required permissions on a topic, the broker will return TOPIC_AUTHORIZATION_FAILED errors to requests irrespective of topic existence on broker. If the user have required permissions and the topic doesn't exists, then the UNKNOWN_TOPIC_OR_PARTITION error code will be returned.
  • config/consumer.properties file updated to use new consumer config properties.
New Protocol Versions
  • KIP-112: LeaderAndIsrRequest v1 introduces a partition-level is_new field.
  • KIP-112: UpdateMetadataRequest v4 introduces a partition-level offline_replicas field.
  • KIP-112: MetadataResponse v5 introduces a partition-level offline_replicas field.
  • KIP-112: ProduceResponse v4 introduces error code for KafkaStorageException.
  • KIP-112: FetchResponse v6 introduces error code for KafkaStorageException.
  • KIP-152: SaslAuthenticate request has been added to enable reporting of authentication failures. This request will be used if the SaslHandshake request version is greater than 0.
Upgrading a 0.11.0 Kafka Streams Application
  • Upgrading your Streams application from 0.11.0 to 1.0 does not require a broker upgrade. A Kafka Streams 1.0 application can connect to 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though). However, Kafka Streams 1.0 requires 0.10 message format or newer and does not work with older message formats.
  • If you are monitoring on streams metrics, you will need make some changes to the metrics names in your reporting and monitoring code, because the metrics sensor hierarchy was changed.
  • There are a few public APIs including ProcessorContext#schedule(), Processor#punctuate() and KStreamBuilder, TopologyBuilder are being deprecated by new APIs. We recommend making corresponding code changes, which should be very minor since the new APIs look quite similar, when you upgrade.
  • See Streams API changes in 1.0.0 for more details.
Upgrading a 0.10.2 Kafka Streams Application
  • Upgrading your Streams application from 0.10.2 to 1.0 does not require a broker upgrade. A Kafka Streams 1.0 application can connect to 1.0, 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
  • If you are monitoring on streams metrics, you will need make some changes to the metrics names in your reporting and monitoring code, because the metrics sensor hierarchy was changed.
  • There are a few public APIs including ProcessorContext#schedule(), Processor#punctuate() and KStreamBuilder, TopologyBuilder are being deprecated by new APIs. We recommend making corresponding code changes, which should be very minor since the new APIs look quite similar, when you upgrade.
  • If you specify customized key.serde, value.serde and timestamp.extractor in configs, it is recommended to use their replaced configure parameter as these configs are deprecated.
  • See Streams API changes in 0.11.0 for more details.
Upgrading a 0.10.1 Kafka Streams Application
  • Upgrading your Streams application from 0.10.1 to 1.0 does not require a broker upgrade. A Kafka Streams 1.0 application can connect to 1.0, 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
  • You need to recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
  • If you are monitoring on streams metrics, you will need make some changes to the metrics names in your reporting and monitoring code, because the metrics sensor hierarchy was changed.
  • There are a few public APIs including ProcessorContext#schedule(), Processor#punctuate() and KStreamBuilder, TopologyBuilder are being deprecated by new APIs. We recommend making corresponding code changes, which should be very minor since the new APIs look quite similar, when you upgrade.
  • If you specify customized key.serde, value.serde and timestamp.extractor in configs, it is recommended to use their replaced configure parameter as these configs are deprecated.
  • If you use a custom (i.e., user implemented) timestamp extractor, you will need to update this code, because the TimestampExtractor interface was changed.
  • If you register custom metrics, you will need to update this code, because the StreamsMetric interface was changed.
  • See Streams API changes in 1.0.0, Streams API changes in 0.11.0 and Streams API changes in 0.10.2 for more details.
Upgrading a 0.10.0 Kafka Streams Application
  • Upgrading your Streams application from 0.10.0 to 1.0 does require a broker upgrade because a Kafka Streams 1.0 application can only connect to 0.1, 0.11.0, 0.10.2, or 0.10.1 brokers.
  • There are couple of API changes, that are not backward compatible (cf. Streams API changes in 1.0.0, Streams API changes in 0.11.0, Streams API changes in 0.10.2, and Streams API changes in 0.10.1 for more details). Thus, you need to update and recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
  • Upgrading from 0.10.0.x to 1.0.2 requires two rolling bounces with config upgrade.from="0.10.0" set for first upgrade phase (cf. KIP-268). As an alternative, an offline upgrade is also possible.
    • prepare your application instances for a rolling bounce and make sure that config upgrade.from is set to "0.10.0" for new version 0.11.0.3
    • bounce each instance of your application once
    • prepare your newly deployed 1.0.2 application instances for a second round of rolling bounces; make sure to remove the value for config upgrade.from
    • bounce each instance of your application once more to complete the upgrade
  • Upgrading from 0.10.0.x to 1.0.0 or 1.0.1 requires an offline upgrade (rolling bounce upgrade is not supported)
    • stop all old (0.10.0.x) application instances
    • update your code and swap old code and jar file with new code and new jar file
    • restart all new (1.0.0 or 1.0.1) application instances

Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x or 0.10.2.x to 0.11.0.0

Kafka 0.11.0.0 introduces a new message format version as well as wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please review the notable changes in 0.11.0.0 before upgrading.

Starting with version 0.10.2, Java clients (producer and consumer) have acquired the ability to communicate with older brokers. Version 0.11.0 clients can talk to version 0.10.0 or newer brokers. However, if your brokers are older than 0.10.0, you must upgrade all the brokers in the Kafka cluster before upgrading your clients. Version 0.11.0 brokers support 0.8.x and newer clients.

For a rolling upgrade:

  1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the current message format version currently in use. If you have not overridden the message format previously, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
    • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1 or 0.10.2).
    • log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.
  3. Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 0.11.0, but do not change log.message.format.version yet.
  4. Restart the brokers one by one for the new protocol version to take effect.
  5. Once all (or most) consumers have been upgraded to 0.11.0 or later, then change log.message.format.version to 0.11.0 on each broker and restart them one by one. Note that the older Scala consumer does not support the new message format, so to avoid the performance cost of down-conversion (or to take advantage of exactly once semantics), the new Java consumer must be used.

Additional Upgrade Notes:

  1. If you are willing to accept downtime, you can simply take all the brokers down, update the code and start them back up. They will start with the new protocol by default.
  2. Bumping the protocol version and restarting can be done any time after the brokers are upgraded. It does not have to be immediately after. Similarly for the message format version.
  3. It is also possible to enable the 0.11.0 message format on individual topics using the topic admin tool (bin/kafka-topics.sh) prior to updating the global setting log.message.format.version.
  4. If you are upgrading from a version prior to 0.10.0, it is NOT necessary to first update the message format to 0.10.0 before you switch to 0.11.0.
Upgrading a 0.10.2 Kafka Streams Application
  • Upgrading your Streams application from 0.10.2 to 0.11.0 does not require a broker upgrade. A Kafka Streams 0.11.0 application can connect to 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
  • If you specify customized key.serde, value.serde and timestamp.extractor in configs, it is recommended to use their replaced configure parameter as these configs are deprecated.
  • See Streams API changes in 0.11.0 for more details.
Upgrading a 0.10.1 Kafka Streams Application
  • Upgrading your Streams application from 0.10.1 to 0.11.0 does not require a broker upgrade. A Kafka Streams 0.11.0 application can connect to 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
  • You need to recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
  • If you specify customized key.serde, value.serde and timestamp.extractor in configs, it is recommended to use their replaced configure parameter as these configs are deprecated.
  • If you use a custom (i.e., user implemented) timestamp extractor, you will need to update this code, because the TimestampExtractor interface was changed.
  • If you register custom metrics, you will need to update this code, because the StreamsMetric interface was changed.
  • See Streams API changes in 0.11.0 and Streams API changes in 0.10.2 for more details.
Upgrading a 0.10.0 Kafka Streams Application
  • Upgrading your Streams application from 0.10.0 to 0.11.0 does require a broker upgrade because a Kafka Streams 0.11.0 application can only connect to 0.11.0, 0.10.2, or 0.10.1 brokers.
  • There are couple of API changes, that are not backward compatible (cf. Streams API changes in 0.11.0, Streams API changes in 0.10.2, and Streams API changes in 0.10.1 for more details). Thus, you need to update and recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
  • Upgrading from 0.10.0.x to 0.11.0.3 requires two rolling bounces with config upgrade.from="0.10.0" set for first upgrade phase (cf. KIP-268). As an alternative, an offline upgrade is also possible.
    • prepare your application instances for a rolling bounce and make sure that config upgrade.from is set to "0.10.0" for new version 0.11.0.3
    • bounce each instance of your application once
    • prepare your newly deployed 0.11.0.3 application instances for a second round of rolling bounces; make sure to remove the value for config upgrade.from
    • bounce each instance of your application once more to complete the upgrade
  • Upgrading from 0.10.0.x to 0.11.0.0, 0.11.0.1, or 0.11.0.2 requires an offline upgrade (rolling bounce upgrade is not supported)
    • stop all old (0.10.0.x) application instances
    • update your code and swap old code and jar file with new code and new jar file
    • restart all new (0.11.0.0 , 0.11.0.1, or 0.11.0.2) application instances
Notable changes in 0.11.0.3
  • New Kafka Streams configuration parameter upgrade.from added that allows rolling bounce upgrade from version 0.10.0.x
  • See the Kafka Streams upgrade guide for details about this new config.
Notable changes in 0.11.0.0
  • Unclean leader election is now disabled by default. The new default favors durability over availability. Users who wish to to retain the previous behavior should set the broker config unclean.leader.election.enable to true.
  • Producer configs block.on.buffer.full, metadata.fetch.timeout.ms and timeout.ms have been removed. They were initially deprecated in Kafka 0.9.0.0.
  • The offsets.topic.replication.factor broker config is now enforced upon auto topic creation. Internal auto topic creation will fail with a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets this replication factor requirement.
  • When compressing data with snappy, the producer and broker will use the compression scheme's default block size (2 x 32 KB) instead of 1 KB in order to improve the compression ratio. There have been reports of data compressed with the smaller block size being 50% larger than when compressed with the larger block size. For the snappy case, a producer with 5000 partitions will require an additional 315 MB of JVM heap.
  • Similarly, when compressing data with gzip, the producer and broker will use 8 KB instead of 1 KB as the buffer size. The default for gzip is excessively low (512 bytes).
  • The broker configuration max.message.bytes now applies to the total size of a batch of messages. Previously the setting applied to batches of compressed messages, or to non-compressed messages individually. A message batch may consist of only a single message, so in most cases, the limitation on the size of individual messages is only reduced by the overhead of the batch format. However, there are some subtle implications for message format conversion (see below for more detail). Note also that while previously the broker would ensure that at least one message is returned in each fetch request (regardless of the total and partition-level fetch sizes), the same behavior now applies to one message batch.
  • GC log rotation is enabled by default, see KAFKA-3754 for details.
  • Deprecated constructors of RecordMetadata, MetricName and Cluster classes have been removed.
  • Added user headers support through a new Headers interface providing user headers read and write access.
  • ProducerRecord and ConsumerRecord expose the new Headers API via Headers headers() method call.
  • ExtendedSerializer and ExtendedDeserializer interfaces are introduced to support serialization and deserialization for headers. Headers will be ignored if the configured serializer and deserializer are not the above classes.
  • A new config, group.initial.rebalance.delay.ms, was introduced. This config specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance. The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms. The default value for this is 3 seconds. During development and testing it might be desirable to set this to 0 in order to not delay test execution time.
  • org.apache.kafka.common.Cluster#partitionsForTopic, partitionsForNode and availablePartitionsForTopic methods will return an empty list instead of null (which is considered a bad practice) in case the metadata for the required topic does not exist.
  • Streams API configuration parameters timestamp.extractor, key.serde, and value.serde were deprecated and replaced by default.timestamp.extractor, default.key.serde, and default.value.serde, respectively.
  • For offset commit failures in the Java consumer's commitAsync APIs, we no longer expose the underlying cause when instances of RetriableCommitFailedException are passed to the commit callback. See KAFKA-5052 for more detail.
New Protocol Versions
  • KIP-107: FetchRequest v5 introduces a partition-level log_start_offset field.
  • KIP-107: FetchResponse v5 introduces a partition-level log_start_offset field.
  • KIP-82: ProduceRequest v3 introduces an array of header in the message protocol, containing key field and value field.
  • KIP-82: FetchResponse v5 introduces an array of header in the message protocol, containing key field and value field.
Notes on Exactly Once Semantics

Kafka 0.11.0 includes support for idempotent and transactional capabilities in the producer. Idempotent delivery ensures that messages are delivered exactly once to a particular topic partition during the lifetime of a single producer. Transactional delivery allows producers to send data to multiple partitions such that either all messages are successfully delivered, or none of them are. Together, these capabilities enable "exactly once semantics" in Kafka. More details on these features are available in the user guide, but below we add a few specific notes on enabling them in an upgraded cluster. Note that enabling EoS is not required and there is no impact on the broker's behavior if unused.

  1. Only the new Java producer and consumer support exactly once semantics.
  2. These features depend crucially on the 0.11.0 message format. Attempting to use them on an older format will result in unsupported version errors.
  3. Transaction state is stored in a new internal topic __transaction_state. This topic is not created until the the first attempt to use a transactional request API. Similar to the consumer offsets topic, there are several settings to control the topic's configuration. For example, transaction.state.log.min.isr controls the minimum ISR for this topic. See the configuration section in the user guide for a full list of options.
  4. For secure clusters, the transactional APIs require new ACLs which can be turned on with the bin/kafka-acls.sh. tool.
  5. EoS in Kafka introduces new request APIs and modifies several existing ones. See KIP-98 for the full details
Notes on the new message format in 0.11.0

The 0.11.0 message format includes several major enhancements in order to support better delivery semantics for the producer (see KIP-98) and improved replication fault tolerance (see KIP-101). Although the new format contains more information to make these improvements possible, we have made the batch format much more efficient. As long as the number of messages per batch is more than 2, you can expect lower overall overhead. For smaller batches, however, there may be a small performance impact. See here for the results of our initial performance analysis of the new message format. You can also find more detail on the message format in the KIP-98 proposal.

One of the notable differences in the new message format is that even uncompressed messages are stored together as a single batch. This has a few implications for the broker configuration max.message.bytes, which limits the size of a single batch. First, if an older client produces messages to a topic partition using the old format, and the messages are individually smaller than max.message.bytes, the broker may still reject them after they are merged into a single batch during the up-conversion process. Generally this can happen when the aggregate size of the individual messages is larger than max.message.bytes. There is a similar effect for older consumers reading messages down-converted from the new format: if the fetch size is not set at least as large as max.message.bytes, the consumer may not be able to make progress even if the individual uncompressed messages are smaller than the configured fetch size. This behavior does not impact the Java client for 0.10.1.0 and later since it uses an updated fetch protocol which ensures that at least one message can be returned even if it exceeds the fetch size. To get around these problems, you should ensure 1) that the producer's batch size is not set larger than max.message.bytes, and 2) that the consumer's fetch size is set at least as large as max.message.bytes.

Most of the discussion on the performance impact of upgrading to the 0.10.0 message format remains pertinent to the 0.11.0 upgrade. This mainly affects clusters that are not secured with TLS since "zero-copy" transfer is already not possible in that case. In order to avoid the cost of down-conversion, you should ensure that consumer applications are upgraded to the latest 0.11.0 client. Significantly, since the old consumer has been deprecated in 0.11.0.0, it does not support the new message format. You must upgrade to use the new consumer to use the new message format without the cost of down-conversion. Note that 0.11.0 consumers support backwards compatibility with 0.10.0 brokers and upward, so it is possible to upgrade the clients first before the brokers.

Upgrading from 0.8.x, 0.9.x, 0.10.0.x or 0.10.1.x to 0.10.2.0

0.10.2.0 has wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please review the notable changes in 0.10.2.0 before upgrading.

Starting with version 0.10.2, Java clients (producer and consumer) have acquired the ability to communicate with older brokers. Version 0.10.2 clients can talk to version 0.10.0 or newer brokers. However, if your brokers are older than 0.10.0, you must upgrade all the brokers in the Kafka cluster before upgrading your clients. Version 0.10.2 brokers support 0.8.x and newer clients.

For a rolling upgrade:

  1. Update server.properties file on all brokers and add the following properties:
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.
  3. Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 0.10.2.
  4. If your previous message format is 0.10.0, change log.message.format.version to 0.10.2 (this is a no-op as the message format is the same for 0.10.0, 0.10.1 and 0.10.2). If your previous message format version is lower than 0.10.0, do not change log.message.format.version yet - this parameter should only change once all consumers have been upgraded to 0.10.0.0 or later.
  5. Restart the brokers one by one for the new protocol version to take effect.
  6. If log.message.format.version is still lower than 0.10.0 at this point, wait until all consumers have been upgraded to 0.10.0 or later, then change log.message.format.version to 0.10.2 on each broker and restart them one by one.

Note: If you are willing to accept downtime, you can simply take all the brokers down, update the code and start all of them. They will start with the new protocol by default.

Note: Bumping the protocol version and restarting can be done any time after the brokers were upgraded. It does not have to be immediately after.

Upgrading a 0.10.1 Kafka Streams Application
  • Upgrading your Streams application from 0.10.1 to 0.10.2 does not require a broker upgrade. A Kafka Streams 0.10.2 application can connect to 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
  • You need to recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
  • If you use a custom (i.e., user implemented) timestamp extractor, you will need to update this code, because the TimestampExtractor interface was changed.
  • If you register custom metrics, you will need to update this code, because the StreamsMetric interface was changed.
  • See Streams API changes in 0.10.2 for more details.
Upgrading a 0.10.0 Kafka Streams Application
  • Upgrading your Streams application from 0.10.0 to 0.10.2 does require a broker upgrade because a Kafka Streams 0.10.2 application can only connect to 0.10.2 or 0.10.1 brokers.
  • There are couple of API changes, that are not backward compatible (cf. Streams API changes in 0.10.2 for more details). Thus, you need to update and recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
  • Upgrading from 0.10.0.x to 0.10.2.2 requires two rolling bounces with config upgrade.from="0.10.0" set for first upgrade phase (cf. KIP-268). As an alternative, an offline upgrade is also possible.
    • prepare your application instances for a rolling bounce and make sure that config upgrade.from is set to "0.10.0" for new version 0.10.2.2
    • bounce each instance of your application once
    • prepare your newly deployed 0.10.2.2 application instances for a second round of rolling bounces; make sure to remove the value for config upgrade.from
    • bounce each instance of your application once more to complete the upgrade
  • Upgrading from 0.10.0.x to 0.10.2.0 or 0.10.2.1 requires an offline upgrade (rolling bounce upgrade is not supported)
    • stop all old (0.10.0.x) application instances
    • update your code and swap old code and jar file with new code and new jar file
    • restart all new (0.10.2.0 or 0.10.2.1) application instances
Notable changes in 0.10.2.2
  • New configuration parameter upgrade.from added that allows rolling bounce upgrade from version 0.10.0.x
Notable changes in 0.10.2.1
  • The default values for two configurations of the StreamsConfig class were changed to improve the resiliency of Kafka Streams applications. The internal Kafka Streams producer retries default value was changed from 0 to 10. The internal Kafka Streams consumer max.poll.interval.ms default value was changed from 300000 to Integer.MAX_VALUE.
Notable changes in 0.10.2.0
  • The Java clients (producer and consumer) have acquired the ability to communicate with older brokers. Version 0.10.2 clients can talk to version 0.10.0 or newer brokers. Note that some features are not available or are limited when older brokers are used.
  • Several methods on the Java consumer may now throw InterruptException if the calling thread is interrupted. Please refer to the KafkaConsumer Javadoc for a more in-depth explanation of this change.
  • Java consumer now shuts down gracefully. By default, the consumer waits up to 30 seconds to complete pending requests. A new close API with timeout has been added to KafkaConsumer to control the maximum wait time.
  • Multiple regular expressions separated by commas can be passed to MirrorMaker with the new Java consumer via the --whitelist option. This makes the behaviour consistent with MirrorMaker when used the old Scala consumer.
  • Upgrading your Streams application from 0.10.1 to 0.10.2 does not require a broker upgrade. A Kafka Streams 0.10.2 application can connect to 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
  • The Zookeeper dependency was removed from the Streams API. The Streams API now uses the Kafka protocol to manage internal topics instead of modifying Zookeeper directly. This eliminates the need for privileges to access Zookeeper directly and "StreamsConfig.ZOOKEEPER_CONFIG" should not be set in the Streams app any more. If the Kafka cluster is secured, Streams apps must have the required security privileges to create new topics.
  • Several new fields including "security.protocol", "connections.max.idle.ms", "retry.backoff.ms", "reconnect.backoff.ms" and "request.timeout.ms" were added to StreamsConfig class. User should pay attention to the default values and set these if needed. For more details please refer to 3.5 Kafka Streams Configs.
New Protocol Versions
  • KIP-88: OffsetFetchRequest v2 supports retrieval of offsets for all topics if the topics array is set to null.
  • KIP-88: OffsetFetchResponse v2 introduces a top-level error_code field.
  • KIP-103: UpdateMetadataRequest v3 introduces a listener_name field to the elements of the end_points array.
  • KIP-108: CreateTopicsRequest v1 introduces a validate_only field.
  • KIP-108: CreateTopicsResponse v1 introduces an error_message field to the elements of the topic_errors array.

Upgrading from 0.8.x, 0.9.x or 0.10.0.X to 0.10.1.0

0.10.1.0 has wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please notice the Potential breaking changes in 0.10.1.0 before upgrade.
Note: Because new protocols are introduced, it is important to upgrade your Kafka clusters before upgrading your clients (i.e. 0.10.1.x clients only support 0.10.1.x or later brokers while 0.10.1.x brokers also support older clients).

For a rolling upgrade:

  1. Update server.properties file on all brokers and add the following properties:
  2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.
  3. Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 0.10.1.0.
  4. If your previous message format is 0.10.0, change log.message.format.version to 0.10.1 (this is a no-op as the message format is the same for both 0.10.0 and 0.10.1). If your previous message format version is lower than 0.10.0, do not change log.message.format.version yet - this parameter should only change once all consumers have been upgraded to 0.10.0.0 or later.
  5. Restart the brokers one by one for the new protocol version to take effect.
  6. If log.message.format.version is still lower than 0.10.0 at this point, wait until all consumers have been upgraded to 0.10.0 or later, then change log.message.format.version to 0.10.1 on each broker and restart them one by one.

Note: If you are willing to accept downtime, you can simply take all the brokers down, update the code and start all of them. They will start with the new protocol by default.

Note: Bumping the protocol version and restarting can be done any time after the brokers were upgraded. It does not have to be immediately after.

Notable changes in 0.10.1.2
  • New configuration parameter upgrade.from added that allows rolling bounce upgrade from version 0.10.0.x
Potential breaking changes in 0.10.1.0
  • The log retention time is no longer based on last modified time of the log segments. Instead it will be based on the largest timestamp of the messages in a log segment.
  • The log rolling time is no longer depending on log segment create time. Instead it is now based on the timestamp in the messages. More specifically. if the timestamp of the first message in the segment is T, the log will be rolled out when a new message has a timestamp greater than or equal to T + log.roll.ms
  • The open file handlers of 0.10.0 will increase by ~33% because of the addition of time index files for each segment.
  • The time index and offset index share the same index size configuration. Since each time index entry is 1.5x the size of offset index entry. User may need to increase log.index.size.max.bytes to avoid potential frequent log rolling.
  • Due to the increased number of index files, on some brokers with large amount the log segments (e.g. >15K), the log loading process during the broker startup could be longer. Based on our experiment, setting the num.recovery.threads.per.data.dir to one may reduce the log loading time.
Upgrading a 0.10.0 Kafka Streams Application
  • Upgrading your Streams application from 0.10.0 to 0.10.1 does require a broker upgrade because a Kafka Streams 0.10.1 application can only connect to 0.10.1 brokers.
  • There are couple of API changes, that are not backward compatible (cf. Streams API changes in 0.10.1 for more details). Thus, you need to update and recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
  • Upgrading from 0.10.0.x to 0.10.1.2 requires two rolling bounces with config upgrade.from="0.10.0" set for first upgrade phase (cf. KIP-268). As an alternative, an offline upgrade is also possible.
    • prepare your application instances for a rolling bounce and make sure that config upgrade.from is set to "0.10.0" for new version 0.10.1.2
    • bounce each instance of your application once
    • prepare your newly deployed 0.10.1.2 application instances for a second round of rolling bounces; make sure to remove the value for config upgrade.from
    • bounce each instance of your application once more to complete the upgrade
  • Upgrading from 0.10.0.x to 0.10.1.0 or 0.10.1.1 requires an offline upgrade (rolling bounce upgrade is not supported)
    • stop all old (0.10.0.x) application instances
    • update your code and swap old code and jar file with new code and new jar file
    • restart all new (0.10.1.0 or 0.10.1.1) application instances
Notable changes in 0.10.1.0
  • The new Java consumer is no longer in beta and we recommend it for all new development. The old Scala consumers are still supported, but they will be deprecated in the next release and will be removed in a future major release.
  • The --new-consumer/--new.consumer switch is no longer required to use tools like MirrorMaker and the Console Consumer with the new consumer; one simply needs to pass a Kafka broker to connect to instead of the ZooKeeper ensemble. In addition, usage of the Console Consumer with the old consumer has been deprecated and it will be removed in a future major release.
  • Kafka clusters can now be uniquely identified by a cluster id. It will be automatically generated when a broker is upgraded to 0.10.1.0. The cluster id is available via the kafka.server:type=KafkaServer,name=ClusterId metric and it is part of the Metadata response. Serializers, client interceptors and metric reporters can receive the cluster id by implementing the ClusterResourceListener interface.
  • The BrokerState "RunningAsController" (value 4) has been removed. Due to a bug, a broker would only be in this state briefly before transitioning out of it and hence the impact of the removal should be minimal. The recommended way to detect if a given broker is the controller is via the kafka.controller:type=KafkaController,name=ActiveControllerCount metric.
  • The new Java Consumer now allows users to search offsets by timestamp on partitions.
  • The new Java Consumer now supports heartbeating from a background thread. There is a new configuration max.poll.interval.ms which controls the maximum time between poll invocations before the consumer will proactively leave the group (5 minutes by default). The value of the configuration request.timeout.ms (default to 30 seconds) must always be smaller than max.poll.interval.ms(default to 5 minutes), since that is the maximum time that a JoinGroup request can block on the server while the consumer is rebalance. Finally, the default value of session.timeout.ms has been adjusted down to 10 seconds, and the default value of max.poll.records has been changed to 500.
  • When using an Authorizer and a user doesn't have Describe authorization on a topic, the broker will no longer return TOPIC_AUTHORIZATION_FAILED errors to requests since this leaks topic names. Instead, the UNKNOWN_TOPIC_OR_PARTITION error code will be returned. This may cause unexpected timeouts or delays when using the producer and consumer since Kafka clients will typically retry automatically on unknown topic errors. You should consult the client logs if you suspect this could be happening.
  • Fetch responses have a size limit by default (50 MB for consumers and 10 MB for replication). The existing per partition limits also apply (1 MB for consumers and replication). Note that neither of these limits is an absolute maximum as explained in the next point.
  • Consumers and replicas can make progress if a message larger than the response/partition size limit is found. More concretely, if the first message in the first non-empty partition of the fetch is larger than either or both limits, the message will still be returned.
  • Overloaded constructors were added to kafka.api.FetchRequest and kafka.javaapi.FetchRequest to allow the caller to specify the order of the partitions (since order is significant in v3). The previously existing constructors were deprecated and the partitions are shuffled before the request is sent to avoid starvation issues.
New Protocol Versions
  • ListOffsetRequest v1 supports accurate offset search based on timestamps.
  • MetadataResponse v2 introduces a new field: "cluster_id".
  • FetchRequest v3 supports limiting the response size (in addition to the existing per partition limit), it returns messages bigger than the limits if required to make progress and the order of partitions in the request is now significant.
  • JoinGroup v1 introduces a new field: "rebalance_timeout".

Upgrading from 0.8.x or 0.9.x to 0.10.0.0

0.10.0.0 has potential breaking changes (please review before upgrading) and possible performance impact following the upgrade. By following the recommended rolling upgrade plan below, you guarantee no downtime and no performance impact during and following the upgrade.
Note: Because new protocols are introduced, it is important to upgrade your Kafka clusters before upgrading your clients.

Notes to clients with version 0.9.0.0: Due to a bug introduced in 0.9.0.0, clients that depend on ZooKeeper (old Scala high-level Consumer and MirrorMaker if used with the old consumer) will not work with 0.10.0.x brokers. Therefore, 0.9.0.0 clients should be upgraded to 0.9.0.1 before brokers are upgraded to 0.10.0.x. This step is not necessary for 0.8.X or 0.9.0.1 clients.

For a rolling upgrade:

  1. Update server.properties file on all brokers and add the following properties:
  2. Upgrade the brokers. This can be done a broker at a time by simply bringing it down, updating the code, and restarting it.
  3. Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 0.10.0.0. NOTE: You shouldn't touch log.message.format.version yet - this parameter should only change once all consumers have been upgraded to 0.10.0.0
  4. Restart the brokers one by one for the new protocol version to take effect.
  5. Once all consumers have been upgraded to 0.10.0, change log.message.format.version to 0.10.0 on each broker and restart them one by one.

Note: If you are willing to accept downtime, you can simply take all the brokers down, update the code and start all of them. They will start with the new protocol by default.

Note: Bumping the protocol version and restarting can be done any time after the brokers were upgraded. It does not have to be immediately after.

Potential performance impact following upgrade to 0.10.0.0

The message format in 0.10.0 includes a new timestamp field and uses relative offsets for compressed messages. The on disk message format can be configured through log.message.format.version in the server.properties file. The default on-disk message format is 0.10.0. If a consumer client is on a version before 0.10.0.0, it only understands message formats before 0.10.0. In this case, the broker is able to convert messages from the 0.10.0 format to an earlier format before sending the response to the consumer on an older version. However, the broker can't use zero-copy transfer in this case. Reports from the Kafka community on the performance impact have shown CPU utilization going from 20% before to 100% after an upgrade, which forced an immediate upgrade of all clients to bring performance back to normal. To avoid such message conversion before consumers are upgraded to 0.10.0.0, one can set log.message.format.version to 0.8.2 or 0.9.0 when upgrading the broker to 0.10.0.0. This way, the broker can still use zero-copy transfer to send the data to the old consumers. Once consumers are upgraded, one can change the message format to 0.10.0 on the broker and enjoy the new message format that includes new timestamp and improved compression. The conversion is supported to ensure compatibility and can be useful to support a few apps that have not updated to newer clients yet, but is impractical to support all consumer traffic on even an overprovisioned cluster. Therefore, it is critical to avoid the message conversion as much as possible when brokers have been upgraded but the majority of clients have not.

For clients that are upgraded to 0.10.0.0, there is no performance impact.

Note: By setting the message format version, one certifies that all existing messages are on or below that message format version. Otherwise consumers before 0.10.0.0 might break. In particular, after the message format is set to 0.10.0, one should not change it back to an earlier format as it may break consumers on versions before 0.10.0.0.

Note: Due to the additional timestamp introduced in each message, producers sending small messages may see a message throughput degradation because of the increased overhead. Likewise, replication now transmits an additional 8 bytes per message. If you're running close to the network capacity of your cluster, it's possible that you'll overwhelm the network cards and see failures and performance issues due to the overload.

Note: If you have enabled compression on producers, you may notice reduced producer throughput and/or lower compression rate on the broker in some cases. When receiving compressed messages, 0.10.0 brokers avoid recompressing the messages, which in general reduces the latency and improves the throughput. In certain cases, however, this may reduce the batching size on the producer, which could lead to worse throughput. If this happens, users can tune linger.ms and batch.size of the producer for better throughput. In addition, the producer buffer used for compressing messages with snappy is smaller than the one used by the broker, which may have a negative impact on the compression ratio for the messages on disk. We intend to make this configurable in a future Kafka release.

Potential breaking changes in 0.10.0.0
  • Starting from Kafka 0.10.0.0, the message format version in Kafka is represented as the Kafka version. For example, message format 0.9.0 refers to the highest message version supported by Kafka 0.9.0.
  • Message format 0.10.0 has been introduced and it is used by default. It includes a timestamp field in the messages and relative offsets are used for compressed messages.
  • ProduceRequest/Response v2 has been introduced and it is used by default to support message format 0.10.0
  • FetchRequest/Response v2 has been introduced and it is used by default to support message format 0.10.0
  • MessageFormatter interface was changed from def writeTo(key: Array[Byte], value: Array[Byte], output: PrintStream) to def writeTo(consumerRecord: ConsumerRecord[Array[Byte], Array[Byte]], output: PrintStream)
  • MessageReader interface was changed from def readMessage(): KeyedMessage[Array[Byte], Array[Byte]] to def readMessage(): ProducerRecord[Array[Byte], Array[Byte]]
  • MessageFormatter's package was changed from kafka.tools to kafka.common
  • MessageReader's package was changed from kafka.tools to kafka.common
  • MirrorMakerMessageHandler no longer exposes the handle(record: MessageAndMetadata[Array[Byte], Array[Byte]]) method as it was never called.
  • The 0.7 KafkaMigrationTool is no longer packaged with Kafka. If you need to migrate from 0.7 to 0.10.0, please migrate to 0.8 first and then follow the documented upgrade process to upgrade from 0.8 to 0.10.0.
  • The new consumer has standardized its APIs to accept java.util.Collection as the sequence type for method parameters. Existing code may have to be updated to work with the 0.10.0 client library.
  • LZ4-compressed message handling was changed to use an interoperable framing specification (LZ4f v1.5.1). To maintain compatibility with old clients, this change only applies to Message format 0.10.0 and later. Clients that Produce/Fetch LZ4-compressed messages using v0/v1 (Message format 0.9.0) should continue to use the 0.9.0 framing implementation. Clients that use Produce/Fetch protocols v2 or later should use interoperable LZ4f framing. A list of interoperable LZ4 libraries is available at https://www.lz4.org/
Notable changes in 0.10.0.0
  • Starting from Kafka 0.10.0.0, a new client library named Kafka Streams is available for stream processing on data stored in Kafka topics. This new client library only works with 0.10.x and upward versioned brokers due to message format changes mentioned above. For more information please read Streams documentation.
  • The default value of the configuration parameter receive.buffer.bytes is now 64K for the new consumer.
  • The new consumer now exposes the configuration parameter exclude.internal.topics to restrict internal topics (such as the consumer offsets topic) from accidentally being included in regular expression subscriptions. By default, it is enabled.
  • The old Scala producer has been deprecated. Users should migrate their code to the Java producer included in the kafka-clients JAR as soon as possible.
  • The new consumer API has been marked stable.

Upgrading from 0.8.0, 0.8.1.X, or 0.8.2.X to 0.9.0.0

0.9.0.0 has potential breaking changes (please review before upgrading) and an inter-broker protocol change from previous versions. This means that upgraded brokers and clients may not be compatible with older versions. It is important that you upgrade your Kafka cluster before upgrading your clients. If you are using MirrorMaker downstream clusters should be upgraded first as well.

For a rolling upgrade:

  1. Update server.properties file on all brokers and add the following property: inter.broker.protocol.version=0.8.2.X
  2. Upgrade the brokers. This can be done a broker at a time by simply bringing it down, updating the code, and restarting it.
  3. Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 0.9.0.0.
  4. Restart the brokers one by one for the new protocol version to take effect

Note: If you are willing to accept downtime, you can simply take all the brokers down, update the code and start all of them. They will start with the new protocol by default.

Note: Bumping the protocol version and restarting can be done any time after the brokers were upgraded. It does not have to be immediately after.

Potential breaking changes in 0.9.0.0
  • Java 1.6 is no longer supported.
  • Scala 2.9 is no longer supported.
  • Broker IDs above 1000 are now reserved by default to automatically assigned broker IDs. If your cluster has existing broker IDs above that threshold make sure to increase the reserved.broker.max.id broker configuration property accordingly.
  • Configuration parameter replica.lag.max.messages was removed. Partition leaders will no longer consider the number of lagging messages when deciding which replicas are in sync.
  • Configuration parameter replica.lag.time.max.ms now refers not just to the time passed since last fetch request from replica, but also to time since the replica last caught up. Replicas that are still fetching messages from leaders but did not catch up to the latest messages in replica.lag.time.max.ms will be considered out of sync.
  • Compacted topics no longer accept messages without key and an exception is thrown by the producer if this is attempted. In 0.8.x, a message without key would cause the log compaction thread to subsequently complain and quit (and stop compacting all compacted topics).
  • MirrorMaker no longer supports multiple target clusters. As a result it will only accept a single --consumer.config parameter. To mirror multiple source clusters, you will need at least one MirrorMaker instance per source cluster, each with its own consumer configuration.
  • Tools packaged under org.apache.kafka.clients.tools.* have been moved to org.apache.kafka.tools.*. All included scripts will still function as usual, only custom code directly importing these classes will be affected.
  • The default Kafka JVM performance options (KAFKA_JVM_PERFORMANCE_OPTS) have been changed in kafka-run-class.sh.
  • The kafka-topics.sh script (kafka.admin.TopicCommand) now exits with non-zero exit code on failure.
  • The kafka-topics.sh script (kafka.admin.TopicCommand) will now print a warning when topic names risk metric collisions due to the use of a '.' or '_' in the topic name, and error in the case of an actual collision.
  • The kafka-console-producer.sh script (kafka.tools.ConsoleProducer) will use the Java producer instead of the old Scala producer be default, and users have to specify 'old-producer' to use the old producer.
  • By default, all command line tools will print all logging messages to stderr instead of stdout.
Notable changes in 0.9.0.1
  • The new broker id generation feature can be disabled by setting broker.id.generation.enable to false.
  • Configuration parameter log.cleaner.enable is now true by default. This means topics with a cleanup.policy=compact will now be compacted by default, and 128 MB of heap will be allocated to the cleaner process via log.cleaner.dedupe.buffer.size. You may want to review log.cleaner.dedupe.buffer.size and the other log.cleaner configuration values based on your usage of compacted topics.
  • Default value of configuration parameter fetch.min.bytes for the new consumer is now 1 by default.
Deprecations in 0.9.0.0
  • Altering topic configuration from the kafka-topics.sh script (kafka.admin.TopicCommand) has been deprecated. Going forward, please use the kafka-configs.sh script (kafka.admin.ConfigCommand) for this functionality.
  • The kafka-consumer-offset-checker.sh (kafka.tools.ConsumerOffsetChecker) has been deprecated. Going forward, please use kafka-consumer-groups.sh (kafka.admin.ConsumerGroupCommand) for this functionality.
  • The kafka.tools.ProducerPerformance class has been deprecated. Going forward, please use org.apache.kafka.tools.ProducerPerformance for this functionality (kafka-producer-perf-test.sh will also be changed to use the new class).
  • The producer config block.on.buffer.full has been deprecated and will be removed in future release. Currently its default value has been changed to false. The KafkaProducer will no longer throw BufferExhaustedException but instead will use max.block.ms value to block, after which it will throw a TimeoutException. If block.on.buffer.full property is set to true explicitly, it will set the max.block.ms to Long.MAX_VALUE and metadata.fetch.timeout.ms will not be honoured

Upgrading from 0.8.1 to 0.8.2

0.8.2 is fully compatible with 0.8.1. The upgrade can be done one broker at a time by simply bringing it down, updating the code, and restarting it.

Upgrading from 0.8.0 to 0.8.1

0.8.1 is fully compatible with 0.8. The upgrade can be done one broker at a time by simply bringing it down, updating the code, and restarting it.

Upgrading from 0.7

Release 0.7 is incompatible with newer releases. Major changes were made to the API, ZooKeeper data structures, and protocol, and configuration in order to add replication (Which was missing in 0.7). The upgrade from 0.7 to later versions requires a special tool for migration. This migration can be done without downtime.

1.6 Docker

JVM Based Apache Kafka Docker Image

Docker is a popular container runtime. Docker images for the JVM based Apache Kafka can be found on Docker Hub and are available from version 3.7.0.

Docker image can be pulled from Docker Hub using the following command: 

$ docker pull apache/kafka:3.9.0

If you want to fetch the latest version of the Docker image use following command: 

$ docker pull apache/kafka:latest

To start the Kafka container using this Docker image with default configs and on default port 9092: 

$ docker run -p 9092:9092 apache/kafka:3.9.0

GraalVM Based Native Apache Kafka Docker Image

Docker images for the GraalVM Based Native Apache Kafka can be found on Docker Hub and are available from version 3.8.0.
NOTE: This image is experimental and intended for local development and testing purposes only; it is not recommended for production use.

Docker image can be pulled from Docker Hub using the following command: 

$ docker pull apache/kafka-native:3.9.0

If you want to fetch the latest version of the Docker image use following command: 

$ docker pull apache/kafka-native:latest

To start the Kafka container using this Docker image with default configs and on default port 9092: 

$ docker run -p 9092:9092 apache/kafka-native:3.9.0

Usage guide

Detailed instructions for using the Docker image are mentioned here.

2. APIs

Kafka includes five core apis:
  1. The Producer API allows applications to send streams of data to topics in the Kafka cluster.
  2. The Consumer API allows applications to read streams of data from topics in the Kafka cluster.
  3. The Streams API allows transforming streams of data from input topics to output topics.
  4. The Connect API allows implementing connectors that continually pull from some source system or application into Kafka or push from Kafka into some sink system or application.
  5. The Admin API allows managing and inspecting topics, brokers, and other Kafka objects.
Kafka exposes all its functionality over a language independent protocol which has clients available in many programming languages. However only the Java clients are maintained as part of the main Kafka project, the others are available as independent open source projects. A list of non-Java clients is available here.

2.1 Producer API

The Producer API allows applications to send streams of data to topics in the Kafka cluster.

Examples showing how to use the producer are given in the javadocs.

To use the producer, you can use the following maven dependency: 

<dependency>
	<groupId>org.apache.kafka</groupId>
	<artifactId>kafka-clients</artifactId>
	<version>3.9.0</version>
</dependency>

2.2 Consumer API

The Consumer API allows applications to read streams of data from topics in the Kafka cluster.

Examples showing how to use the consumer are given in the javadocs.

To use the consumer, you can use the following maven dependency: 

<dependency>
	<groupId>org.apache.kafka</groupId>
	<artifactId>kafka-clients</artifactId>
	<version>3.9.0</version>
</dependency>

2.3 Streams API

The Streams API allows transforming streams of data from input topics to output topics.

Examples showing how to use this library are given in the javadocs

Additional documentation on using the Streams API is available here.

To use Kafka Streams you can use the following maven dependency: 

<dependency>
	<groupId>org.apache.kafka</groupId>
	<artifactId>kafka-streams</artifactId>
	<version>3.9.0</version>
</dependency>

When using Scala you may optionally include the kafka-streams-scala library. Additional documentation on using the Kafka Streams DSL for Scala is available in the developer guide.

To use Kafka Streams DSL for Scala for Scala 2.13 you can use the following maven dependency: 

<dependency>
	<groupId>org.apache.kafka</groupId>
	<artifactId>kafka-streams-scala_2.13</artifactId>
	<version>3.9.0</version>
</dependency>

2.4 Connect API

The Connect API allows implementing connectors that continually pull from some source data system into Kafka or push from Kafka into some sink data system.

Many users of Connect won't need to use this API directly, though, they can use pre-built connectors without needing to write any code. Additional information on using Connect is available here.

Those who want to implement custom connectors can see the javadoc.

2.5 Admin API

The Admin API supports managing and inspecting topics, brokers, acls, and other Kafka objects.

To use the Admin API, add the following Maven dependency: 

<dependency>
	<groupId>org.apache.kafka</groupId>
	<artifactId>kafka-clients</artifactId>
	<version>3.9.0</version>
</dependency>
For more information about the Admin APIs, see the javadoc.

3. Configuration

Kafka uses key-value pairs in the property file format for configuration. These values can be supplied either from a file or programmatically.

3.1 Broker Configs

The essential configurations are the following:
  • broker.id
  • log.dirs
  • zookeeper.connect
Topic-level configurations and defaults are discussed in more detail below.

  • advertised.listeners

    Listeners to publish to ZooKeeper for clients to use, if different than the listeners config property. In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, the value for listeners will be used. Unlike listeners, it is not valid to advertise the 0.0.0.0 meta-address.
    Also unlike listeners, there can be duplicated ports in this property, so that one listener can be configured to advertise another listener's address. This can be useful in some cases where external load balancers are used.

    Type:string
    Default:null
    Valid Values:
    Importance:high
    Update Mode:per-broker

  • auto.create.topics.enable

    Enable auto creation of topic on the server.

    Type:boolean
    Default:true
    Valid Values:
    Importance:high
    Update Mode:read-only

  • auto.leader.rebalance.enable

    Enables auto leader balancing. A background thread checks the distribution of partition leaders at regular intervals, configurable by leader.imbalance.check.interval.seconds. If the leader imbalance exceeds leader.imbalance.per.broker.percentage, leader rebalance to the preferred leader for partitions is triggered.

    Type:boolean
    Default:true
    Valid Values:
    Importance:high
    Update Mode:read-only

  • background.threads

    The number of threads to use for various background processing tasks

    Type:int
    Default:10
    Valid Values:[1,...]
    Importance:high
    Update Mode:cluster-wide

  • broker.id

    The broker id for this server. If unset, a unique broker id will be generated.To avoid conflicts between ZooKeeper generated broker id's and user configured broker id's, generated broker ids start from reserved.broker.max.id + 1.

    Type:int
    Default:-1
    Valid Values:
    Importance:high
    Update Mode:read-only

  • compression.type

    Specify the final compression type for a given topic. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'uncompressed' which is equivalent to no compression; and 'producer' which means retain the original compression codec set by the producer.

    Type:string
    Default:producer
    Valid Values:[uncompressed, zstd, lz4, snappy, gzip, producer]
    Importance:high
    Update Mode:cluster-wide

  • control.plane.listener.name

    Name of listener used for communication between controller and brokers. A broker will use the control.plane.listener.name to locate the endpoint in listeners list, to listen for connections from the controller. For example, if a broker's config is:
    listeners = INTERNAL://192.1.1.8:9092, EXTERNAL://10.1.1.5:9093, CONTROLLER://192.1.1.8:9094listener.security.protocol.map = INTERNAL:PLAINTEXT, EXTERNAL:SSL, CONTROLLER:SSLcontrol.plane.listener.name = CONTROLLER
    On startup, the broker will start listening on "192.1.1.8:9094" with security protocol "SSL".
    On the controller side, when it discovers a broker's published endpoints through ZooKeeper, it will use the control.plane.listener.name to find the endpoint, which it will use to establish connection to the broker.
    For example, if the broker's published endpoints on ZooKeeper are:
    "endpoints" : ["INTERNAL://broker1.example.com:9092","EXTERNAL://broker1.example.com:9093","CONTROLLER://broker1.example.com:9094"]
    and the controller's config is:
    listener.security.protocol.map = INTERNAL:PLAINTEXT, EXTERNAL:SSL, CONTROLLER:SSLcontrol.plane.listener.name = CONTROLLER
    then the controller will use "broker1.example.com:9094" with security protocol "SSL" to connect to the broker.
    If not explicitly configured, the default value will be null and there will be no dedicated endpoints for controller connections.
    If explicitly configured, the value cannot be the same as the value of inter.broker.listener.name.

    Type:string
    Default:null
    Valid Values:
    Importance:high
    Update Mode:read-only

  • controller.listener.names

    A comma-separated list of the names of the listeners used by the controller. This is required if running in KRaft mode. When communicating with the controller quorum, the broker will always use the first listener in this list.
    Note: The ZooKeeper-based controller should not set this configuration.

    Type:string
    Default:null
    Valid Values:
    Importance:high
    Update Mode:read-only

  • controller.quorum.bootstrap.servers

    List of endpoints to use for bootstrapping the cluster metadata. The endpoints are specified in comma-separated list of {host}:{port} entries. For example: localhost:9092,localhost:9093,localhost:9094.

    Type:list
    Default:""
    Valid Values:non-empty list
    Importance:high
    Update Mode:read-only

  • controller.quorum.election.backoff.max.ms

    Maximum time in milliseconds before starting new elections. This is used in the binary exponential backoff mechanism that helps prevent gridlocked elections

    Type:int
    Default:1000 (1 second)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • controller.quorum.election.timeout.ms

    Maximum time in milliseconds to wait without being able to fetch from the leader before triggering a new election

    Type:int
    Default:1000 (1 second)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • controller.quorum.fetch.timeout.ms

    Maximum time without a successful fetch from the current leader before becoming a candidate and triggering an election for voters; Maximum time a leader can go without receiving valid fetch or fetchSnapshot request from a majority of the quorum before resigning.

    Type:int
    Default:2000 (2 seconds)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • controller.quorum.voters

    Map of id/endpoint information for the set of voters in a comma-separated list of {id}@{host}:{port} entries. For example: 1@localhost:9092,2@localhost:9093,3@localhost:9094

    Type:list
    Default:""
    Valid Values:non-empty list
    Importance:high
    Update Mode:read-only

  • delete.topic.enable

    Enables delete topic. Delete topic through the admin tool will have no effect if this config is turned off

    Type:boolean
    Default:true
    Valid Values:
    Importance:high
    Update Mode:read-only

  • early.start.listeners

    A comma-separated list of listener names which may be started before the authorizer has finished initialization. This is useful when the authorizer is dependent on the cluster itself for bootstrapping, as is the case for the StandardAuthorizer (which stores ACLs in the metadata log.) By default, all listeners included in controller.listener.names will also be early start listeners. A listener should not appear in this list if it accepts external traffic.

    Type:string
    Default:null
    Valid Values:
    Importance:high
    Update Mode:read-only

  • eligible.leader.replicas.enable

    Enable the Eligible leader replicas

    Type:boolean
    Default:false
    Valid Values:
    Importance:high
    Update Mode:read-only

  • leader.imbalance.check.interval.seconds

    The frequency with which the partition rebalance check is triggered by the controller

    Type:long
    Default:300
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • leader.imbalance.per.broker.percentage

    The ratio of leader imbalance allowed per broker. The controller would trigger a leader balance if it goes above this value per broker. The value is specified in percentage.

    Type:int
    Default:10
    Valid Values:
    Importance:high
    Update Mode:read-only

  • listeners

    Listener List - Comma-separated list of URIs we will listen on and the listener names. If the listener name is not a security protocol, listener.security.protocol.map must also be set.
    Listener names and port numbers must be unique unless %n one listener is an IPv4 address and the other listener is %n an IPv6 address (for the same port).%n Specify hostname as 0.0.0.0 to bind to all interfaces.%n Leave hostname empty to bind to default interface.%n Examples of legal listener lists:%n PLAINTEXT://myhost:9092,SSL://:9091%n CLIENT://0.0.0.0:9092,REPLICATION://localhost:9093%n PLAINTEXT://127.0.0.1:9092,SSL://[::1]:9092%n

    Type:string
    Default:PLAINTEXT://:9092
    Valid Values:
    Importance:high
    Update Mode:per-broker

  • log.dir

    The directory in which the log data is kept (supplemental for log.dirs property)

    Type:string
    Default:/tmp/kafka-logs
    Valid Values:
    Importance:high
    Update Mode:read-only

  • log.dirs

    A comma-separated list of the directories where the log data is stored. If not set, the value in log.dir is used.

    Type:string
    Default:null
    Valid Values:
    Importance:high
    Update Mode:read-only

  • log.flush.interval.messages

    The number of messages accumulated on a log partition before messages are flushed to disk.

    Type:long
    Default:9223372036854775807
    Valid Values:[1,...]
    Importance:high
    Update Mode:cluster-wide

  • log.flush.interval.ms

    The maximum time in ms that a message in any topic is kept in memory before flushed to disk. If not set, the value in log.flush.scheduler.interval.ms is used

    Type:long
    Default:null
    Valid Values:
    Importance:high
    Update Mode:cluster-wide

  • log.flush.offset.checkpoint.interval.ms

    The frequency with which we update the persistent record of the last flush which acts as the log recovery point.

    Type:int
    Default:60000 (1 minute)
    Valid Values:[0,...]
    Importance:high
    Update Mode:read-only

  • log.flush.scheduler.interval.ms

    The frequency in ms that the log flusher checks whether any log needs to be flushed to disk

    Type:long
    Default:9223372036854775807
    Valid Values:
    Importance:high
    Update Mode:read-only

  • log.flush.start.offset.checkpoint.interval.ms

    The frequency with which we update the persistent record of log start offset

    Type:int
    Default:60000 (1 minute)
    Valid Values:[0,...]
    Importance:high
    Update Mode:read-only

  • log.retention.bytes

    The maximum size of the log before deleting it

    Type:long
    Default:-1
    Valid Values:
    Importance:high
    Update Mode:cluster-wide

  • log.retention.hours

    The number of hours to keep a log file before deleting it (in hours), tertiary to log.retention.ms property

    Type:int
    Default:168
    Valid Values:
    Importance:high
    Update Mode:read-only

  • log.retention.minutes

    The number of minutes to keep a log file before deleting it (in minutes), secondary to log.retention.ms property. If not set, the value in log.retention.hours is used

    Type:int
    Default:null
    Valid Values:
    Importance:high
    Update Mode:read-only

  • log.retention.ms

    The number of milliseconds to keep a log file before deleting it (in milliseconds), If not set, the value in log.retention.minutes is used. If set to -1, no time limit is applied.

    Type:long
    Default:null
    Valid Values:
    Importance:high
    Update Mode:cluster-wide

  • log.roll.hours

    The maximum time before a new log segment is rolled out (in hours), secondary to log.roll.ms property

    Type:int
    Default:168
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • log.roll.jitter.hours

    The maximum jitter to subtract from logRollTimeMillis (in hours), secondary to log.roll.jitter.ms property

    Type:int
    Default:0
    Valid Values:[0,...]
    Importance:high
    Update Mode:read-only

  • log.roll.jitter.ms

    The maximum jitter to subtract from logRollTimeMillis (in milliseconds). If not set, the value in log.roll.jitter.hours is used

    Type:long
    Default:null
    Valid Values:
    Importance:high
    Update Mode:cluster-wide

  • log.roll.ms

    The maximum time before a new log segment is rolled out (in milliseconds). If not set, the value in log.roll.hours is used

    Type:long
    Default:null
    Valid Values:
    Importance:high
    Update Mode:cluster-wide

  • log.segment.bytes

    The maximum size of a single log file

    Type:int
    Default:1073741824 (1 gibibyte)
    Valid Values:[14,...]
    Importance:high
    Update Mode:cluster-wide

  • log.segment.delete.delay.ms

    The amount of time to wait before deleting a file from the filesystem. If the value is 0 and there is no file to delete, the system will wait 1 millisecond. Low value will cause busy waiting

    Type:long
    Default:60000 (1 minute)
    Valid Values:[0,...]
    Importance:high
    Update Mode:cluster-wide

  • message.max.bytes

    The largest record batch size allowed by Kafka (after compression if compression is enabled). If this is increased and there are consumers older than 0.10.2, the consumers' fetch size must also be increased so that they can fetch record batches this large. In the latest message format version, records are always grouped into batches for efficiency. In previous message format versions, uncompressed records are not grouped into batches and this limit only applies to a single record in that case.This can be set per topic with the topic level max.message.bytes config.

    Type:int
    Default:1048588
    Valid Values:[0,...]
    Importance:high
    Update Mode:cluster-wide

  • metadata.log.dir

    This configuration determines where we put the metadata log for clusters in KRaft mode. If it is not set, the metadata log is placed in the first log directory from log.dirs.

    Type:string
    Default:null
    Valid Values:
    Importance:high
    Update Mode:read-only

  • metadata.log.max.record.bytes.between.snapshots

    This is the maximum number of bytes in the log between the latest snapshot and the high-watermark needed before generating a new snapshot. The default value is 20971520. To generate snapshots based on the time elapsed, see the metadata.log.max.snapshot.interval.ms configuration. The Kafka node will generate a snapshot when either the maximum time interval is reached or the maximum bytes limit is reached.

    Type:long
    Default:20971520
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • metadata.log.max.snapshot.interval.ms

    This is the maximum number of milliseconds to wait to generate a snapshot if there are committed records in the log that are not included in the latest snapshot. A value of zero disables time based snapshot generation. The default value is 3600000. To generate snapshots based on the number of metadata bytes, see the metadata.log.max.record.bytes.between.snapshots configuration. The Kafka node will generate a snapshot when either the maximum time interval is reached or the maximum bytes limit is reached.

    Type:long
    Default:3600000 (1 hour)
    Valid Values:[0,...]
    Importance:high
    Update Mode:read-only

  • metadata.log.segment.bytes

    The maximum size of a single metadata log file.

    Type:int
    Default:1073741824 (1 gibibyte)
    Valid Values:[12,...]
    Importance:high
    Update Mode:read-only

  • metadata.log.segment.ms

    The maximum time before a new metadata log file is rolled out (in milliseconds).

    Type:long
    Default:604800000 (7 days)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • metadata.max.retention.bytes

    The maximum combined size of the metadata log and snapshots before deleting old snapshots and log files. Since at least one snapshot must exist before any logs can be deleted, this is a soft limit.

    Type:long
    Default:104857600 (100 mebibytes)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • metadata.max.retention.ms

    The number of milliseconds to keep a metadata log file or snapshot before deleting it. Since at least one snapshot must exist before any logs can be deleted, this is a soft limit.

    Type:long
    Default:604800000 (7 days)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • min.insync.replicas

    When a producer sets acks to "all" (or "-1"), min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this minimum cannot be met, then the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend).
    When used together, min.insync.replicas and acks allow you to enforce greater durability guarantees. A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all". This will ensure that the producer raises an exception if a majority of replicas do not receive a write.

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:high
    Update Mode:cluster-wide

  • node.id

    The node ID associated with the roles this process is playing when process.roles is non-empty. This is required configuration when running in KRaft mode.

    Type:int
    Default:-1
    Valid Values:
    Importance:high
    Update Mode:read-only

  • num.io.threads

    The number of threads that the server uses for processing requests, which may include disk I/O

    Type:int
    Default:8
    Valid Values:[1,...]
    Importance:high
    Update Mode:cluster-wide

  • num.network.threads

    The number of threads that the server uses for receiving requests from the network and sending responses to the network. Noted: each listener (except for controller listener) creates its own thread pool.

    Type:int
    Default:3
    Valid Values:[1,...]
    Importance:high
    Update Mode:cluster-wide

  • num.recovery.threads.per.data.dir

    The number of threads per data directory to be used for log recovery at startup and flushing at shutdown

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:high
    Update Mode:cluster-wide

  • num.replica.alter.log.dirs.threads

    The number of threads that can move replicas between log directories, which may include disk I/O

    Type:int
    Default:null
    Valid Values:
    Importance:high
    Update Mode:read-only

  • num.replica.fetchers

    Number of fetcher threads used to replicate records from each source broker. The total number of fetchers on each broker is bound by num.replica.fetchers multiplied by the number of brokers in the cluster.Increasing this value can increase the degree of I/O parallelism in the follower and leader broker at the cost of higher CPU and memory utilization.

    Type:int
    Default:1
    Valid Values:
    Importance:high
    Update Mode:cluster-wide

  • offset.metadata.max.bytes

    The maximum size for a metadata entry associated with an offset commit.

    Type:int
    Default:4096 (4 kibibytes)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • offsets.commit.required.acks

    DEPRECATED: The required acks before the commit can be accepted. In general, the default (-1) should not be overridden.

    Type:short
    Default:-1
    Valid Values:
    Importance:high
    Update Mode:read-only

  • offsets.commit.timeout.ms

    Offset commit will be delayed until all replicas for the offsets topic receive the commit or this timeout is reached. This is similar to the producer request timeout.

    Type:int
    Default:5000 (5 seconds)
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • offsets.load.buffer.size

    Batch size for reading from the offsets segments when loading offsets into the cache (soft-limit, overridden if records are too large).

    Type:int
    Default:5242880
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • offsets.retention.check.interval.ms

    Frequency at which to check for stale offsets

    Type:long
    Default:600000 (10 minutes)
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • offsets.retention.minutes

    For subscribed consumers, committed offset of a specific partition will be expired and discarded when 1) this retention period has elapsed after the consumer group loses all its consumers (i.e. becomes empty); 2) this retention period has elapsed since the last time an offset is committed for the partition and the group is no longer subscribed to the corresponding topic. For standalone consumers (using manual assignment), offsets will be expired after this retention period has elapsed since the time of last commit. Note that when a group is deleted via the delete-group request, its committed offsets will also be deleted without extra retention period; also when a topic is deleted via the delete-topic request, upon propagated metadata update any group's committed offsets for that topic will also be deleted without extra retention period.

    Type:int
    Default:10080
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • offsets.topic.compression.codec

    Compression codec for the offsets topic - compression may be used to achieve "atomic" commits.

    Type:int
    Default:0
    Valid Values:
    Importance:high
    Update Mode:read-only

  • offsets.topic.num.partitions

    The number of partitions for the offset commit topic (should not change after deployment).

    Type:int
    Default:50
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • offsets.topic.replication.factor

    The replication factor for the offsets topic (set higher to ensure availability). Internal topic creation will fail until the cluster size meets this replication factor requirement.

    Type:short
    Default:3
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • offsets.topic.segment.bytes

    The offsets topic segment bytes should be kept relatively small in order to facilitate faster log compaction and cache loads.

    Type:int
    Default:104857600 (100 mebibytes)
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • process.roles

    The roles that this process plays: 'broker', 'controller', or 'broker,controller' if it is both. This configuration is only applicable for clusters in KRaft (Kafka Raft) mode (instead of ZooKeeper). Leave this config undefined or empty for ZooKeeper clusters.

    Type:list
    Default:""
    Valid Values:[broker, controller]
    Importance:high
    Update Mode:read-only

  • queued.max.requests

    The number of queued requests allowed for data-plane, before blocking the network threads

    Type:int
    Default:500
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • replica.fetch.min.bytes

    Minimum bytes expected for each fetch response. If not enough bytes, wait up to replica.fetch.wait.max.ms (broker config).

    Type:int
    Default:1
    Valid Values:
    Importance:high
    Update Mode:read-only

  • replica.fetch.wait.max.ms

    The maximum wait time for each fetcher request issued by follower replicas. This value should always be less than the replica.lag.time.max.ms at all times to prevent frequent shrinking of ISR for low throughput topics

    Type:int
    Default:500
    Valid Values:
    Importance:high
    Update Mode:read-only

  • replica.high.watermark.checkpoint.interval.ms

    The frequency with which the high watermark is saved out to disk

    Type:long
    Default:5000 (5 seconds)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • replica.lag.time.max.ms

    If a follower hasn't sent any fetch requests or hasn't consumed up to the leaders log end offset for at least this time, the leader will remove the follower from isr

    Type:long
    Default:30000 (30 seconds)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • replica.socket.receive.buffer.bytes

    The socket receive buffer for network requests to the leader for replicating data

    Type:int
    Default:65536 (64 kibibytes)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • replica.socket.timeout.ms

    The socket timeout for network requests. Its value should be at least replica.fetch.wait.max.ms

    Type:int
    Default:30000 (30 seconds)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • request.timeout.ms

    The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

    Type:int
    Default:30000 (30 seconds)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • sasl.mechanism.controller.protocol

    SASL mechanism used for communication with controllers. Default is GSSAPI.

    Type:string
    Default:GSSAPI
    Valid Values:
    Importance:high
    Update Mode:read-only

  • socket.receive.buffer.bytes

    The SO_RCVBUF buffer of the socket server sockets. If the value is -1, the OS default will be used.

    Type:int
    Default:102400 (100 kibibytes)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • socket.request.max.bytes

    The maximum number of bytes in a socket request

    Type:int
    Default:104857600 (100 mebibytes)
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • socket.send.buffer.bytes

    The SO_SNDBUF buffer of the socket server sockets. If the value is -1, the OS default will be used.

    Type:int
    Default:102400 (100 kibibytes)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • transaction.max.timeout.ms

    The maximum allowed timeout for transactions. If a client’s requested transaction time exceed this, then the broker will return an error in InitProducerIdRequest. This prevents a client from too large of a timeout, which can stall consumers reading from topics included in the transaction.

    Type:int
    Default:900000 (15 minutes)
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • transaction.state.log.load.buffer.size

    Batch size for reading from the transaction log segments when loading producer ids and transactions into the cache (soft-limit, overridden if records are too large).

    Type:int
    Default:5242880
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • transaction.state.log.min.isr

    The minimum number of replicas that must acknowledge a write to transaction topic in order to be considered successful.

    Type:int
    Default:2
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • transaction.state.log.num.partitions

    The number of partitions for the transaction topic (should not change after deployment).

    Type:int
    Default:50
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • transaction.state.log.replication.factor

    The replication factor for the transaction topic (set higher to ensure availability). Internal topic creation will fail until the cluster size meets this replication factor requirement.

    Type:short
    Default:3
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • transaction.state.log.segment.bytes

    The transaction topic segment bytes should be kept relatively small in order to facilitate faster log compaction and cache loads

    Type:int
    Default:104857600 (100 mebibytes)
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • transactional.id.expiration.ms

    The time in ms that the transaction coordinator will wait without receiving any transaction status updates for the current transaction before expiring its transactional id. Transactional IDs will not expire while a the transaction is still ongoing.

    Type:int
    Default:604800000 (7 days)
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • unclean.leader.election.enable

    Indicates whether to enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss

    Note: In KRaft mode, when enabling this config dynamically, it needs to wait for the unclean leader election thread to trigger election periodically (default is 5 minutes). Please run `kafka-leader-election.sh` with `unclean` option to trigger the unclean leader election immediately if needed.

    Type:boolean
    Default:false
    Valid Values:
    Importance:high
    Update Mode:cluster-wide

  • zookeeper.connect

    Specifies the ZooKeeper connection string in the form hostname:port where host and port are the host and port of a ZooKeeper server. To allow connecting through other ZooKeeper nodes when that ZooKeeper machine is down you can also specify multiple hosts in the form hostname1:port1,hostname2:port2,hostname3:port3.
    The server can also have a ZooKeeper chroot path as part of its ZooKeeper connection string which puts its data under some path in the global ZooKeeper namespace. For example to give a chroot path of /chroot/path you would give the connection string as hostname1:port1,hostname2:port2,hostname3:port3/chroot/path.

    Type:string
    Default:null
    Valid Values:
    Importance:high
    Update Mode:read-only

  • zookeeper.connection.timeout.ms

    The max time that the client waits to establish a connection to ZooKeeper. If not set, the value in zookeeper.session.timeout.ms is used

    Type:int
    Default:null
    Valid Values:
    Importance:high
    Update Mode:read-only

  • zookeeper.max.in.flight.requests

    The maximum number of unacknowledged requests the client will send to ZooKeeper before blocking.

    Type:int
    Default:10
    Valid Values:[1,...]
    Importance:high
    Update Mode:read-only

  • zookeeper.metadata.migration.enable

    Enable ZK to KRaft migration

    Type:boolean
    Default:false
    Valid Values:
    Importance:high
    Update Mode:read-only

  • zookeeper.session.timeout.ms

    Zookeeper session timeout

    Type:int
    Default:18000 (18 seconds)
    Valid Values:
    Importance:high
    Update Mode:read-only

  • zookeeper.set.acl

    Set client to use secure ACLs

    Type:boolean
    Default:false
    Valid Values:
    Importance:high
    Update Mode:read-only

  • broker.heartbeat.interval.ms

    The length of time in milliseconds between broker heartbeats. Used when running in KRaft mode.

    Type:int
    Default:2000 (2 seconds)
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • broker.id.generation.enable

    Enable automatic broker id generation on the server. When enabled the value configured for reserved.broker.max.id should be reviewed.

    Type:boolean
    Default:true
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • broker.rack

    Rack of the broker. This will be used in rack aware replication assignment for fault tolerance. Examples: RACK1, us-east-1d

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • broker.session.timeout.ms

    The length of time in milliseconds that a broker lease lasts if no heartbeats are made. Used when running in KRaft mode.

    Type:int
    Default:9000 (9 seconds)
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • compression.gzip.level

    The compression level to use if compression.type is set to 'gzip'.

    Type:int
    Default:-1
    Valid Values:[1,...,9] or -1
    Importance:medium
    Update Mode:cluster-wide

  • compression.lz4.level

    The compression level to use if compression.type is set to 'lz4'.

    Type:int
    Default:9
    Valid Values:[1,...,17]
    Importance:medium
    Update Mode:cluster-wide

  • compression.zstd.level

    The compression level to use if compression.type is set to 'zstd'.

    Type:int
    Default:3
    Valid Values:[-131072,...,22]
    Importance:medium
    Update Mode:cluster-wide

  • connections.max.idle.ms

    Idle connections timeout: the server socket processor threads close the connections that idle more than this

    Type:long
    Default:600000 (10 minutes)
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • connections.max.reauth.ms

    When explicitly set to a positive number (the default is 0, not a positive number), a session lifetime that will not exceed the configured value will be communicated to v2.2.0 or later clients when they authenticate. The broker will disconnect any such connection that is not re-authenticated within the session lifetime and that is then subsequently used for any purpose other than re-authentication. Configuration names can optionally be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.oauthbearer.connections.max.reauth.ms=3600000

    Type:long
    Default:0
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • controlled.shutdown.enable

    Enable controlled shutdown of the server.

    Type:boolean
    Default:true
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • controlled.shutdown.max.retries

    Controlled shutdown can fail for multiple reasons. This determines the number of retries when such failure happens

    Type:int
    Default:3
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • controlled.shutdown.retry.backoff.ms

    Before each retry, the system needs time to recover from the state that caused the previous failure (Controller fail over, replica lag etc). This config determines the amount of time to wait before retrying.

    Type:long
    Default:5000 (5 seconds)
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • controller.quorum.append.linger.ms

    The duration in milliseconds that the leader will wait for writes to accumulate before flushing them to disk.

    Type:int
    Default:25
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • controller.quorum.request.timeout.ms

    The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

    Type:int
    Default:2000 (2 seconds)
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • controller.socket.timeout.ms

    The socket timeout for controller-to-broker channels.

    Type:int
    Default:30000 (30 seconds)
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • default.replication.factor

    The replication factor for automatically created topics, and for topics created with -1 as the replication factor

    Type:int
    Default:1
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • delegation.token.expiry.time.ms

    The token validity time in milliseconds before the token needs to be renewed. Default value 1 day.

    Type:long
    Default:86400000 (1 day)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • delegation.token.master.key

    DEPRECATED: An alias for delegation.token.secret.key, which should be used instead of this config.

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • delegation.token.max.lifetime.ms

    The token has a maximum lifetime beyond which it cannot be renewed anymore. Default value 7 days.

    Type:long
    Default:604800000 (7 days)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • delegation.token.secret.key

    Secret key to generate and verify delegation tokens. The same key must be configured across all the brokers. If using Kafka with KRaft, the key must also be set across all controllers. If the key is not set or set to empty string, brokers will disable the delegation token support.

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • delete.records.purgatory.purge.interval.requests

    The purge interval (in number of requests) of the delete records request purgatory

    Type:int
    Default:1
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • fetch.max.bytes

    The maximum number of bytes we will return for a fetch request. Must be at least 1024.

    Type:int
    Default:57671680 (55 mebibytes)
    Valid Values:[1024,...]
    Importance:medium
    Update Mode:read-only

  • fetch.purgatory.purge.interval.requests

    The purge interval (in number of requests) of the fetch request purgatory

    Type:int
    Default:1000
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • group.consumer.assignors

    The server side assignors as a list of full class names. The first one in the list is considered as the default assignor to be used in the case where the consumer does not specify an assignor.

    Type:list
    Default:org.apache.kafka.coordinator.group.assignor.UniformAssignor,org.apache.kafka.coordinator.group.assignor.RangeAssignor
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • group.consumer.heartbeat.interval.ms

    The heartbeat interval given to the members of a consumer group.

    Type:int
    Default:5000 (5 seconds)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.consumer.max.heartbeat.interval.ms

    The maximum heartbeat interval for registered consumers.

    Type:int
    Default:15000 (15 seconds)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.consumer.max.session.timeout.ms

    The maximum allowed session timeout for registered consumers.

    Type:int
    Default:60000 (1 minute)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.consumer.max.size

    The maximum number of consumers that a single consumer group can accommodate. This value will only impact the new consumer coordinator. To configure the classic consumer coordinator check group.max.size instead.

    Type:int
    Default:2147483647
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.consumer.min.heartbeat.interval.ms

    The minimum heartbeat interval for registered consumers.

    Type:int
    Default:5000 (5 seconds)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.consumer.min.session.timeout.ms

    The minimum allowed session timeout for registered consumers.

    Type:int
    Default:45000 (45 seconds)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.consumer.session.timeout.ms

    The timeout to detect client failures when using the consumer group protocol.

    Type:int
    Default:45000 (45 seconds)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.coordinator.append.linger.ms

    The duration in milliseconds that the coordinator will wait for writes to accumulate before flushing them to disk. Transactional writes are not accumulated.

    Type:int
    Default:10
    Valid Values:[0,...]
    Importance:medium
    Update Mode:read-only

  • group.coordinator.rebalance.protocols

    The list of enabled rebalance protocols. The consumer and share rebalance protocol are in early access and therefore must not be used in production.

    Type:list
    Default:classic
    Valid Values:[consumer, classic, share]
    Importance:medium
    Update Mode:read-only

  • group.coordinator.threads

    The number of threads used by the group coordinator.

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.initial.rebalance.delay.ms

    The amount of time the group coordinator will wait for more consumers to join a new group before performing the first rebalance. A longer delay means potentially fewer rebalances, but increases the time until processing begins.

    Type:int
    Default:3000 (3 seconds)
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • group.max.session.timeout.ms

    The maximum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures.

    Type:int
    Default:1800000 (30 minutes)
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • group.max.size

    The maximum number of consumers that a single consumer group can accommodate.

    Type:int
    Default:2147483647
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.min.session.timeout.ms

    The minimum allowed session timeout for registered consumers. Shorter timeouts result in quicker failure detection at the cost of more frequent consumer heartbeating, which can overwhelm broker resources.

    Type:int
    Default:6000 (6 seconds)
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • group.share.delivery.count.limit

    The maximum number of delivery attempts for a record delivered to a share group.

    Type:int
    Default:5
    Valid Values:[2,...,10]
    Importance:medium
    Update Mode:read-only

  • group.share.heartbeat.interval.ms

    The heartbeat interval given to the members of a share group.

    Type:int
    Default:5000 (5 seconds)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.share.max.groups

    The maximum number of share groups.

    Type:short
    Default:10
    Valid Values:[1,...,100]
    Importance:medium
    Update Mode:read-only

  • group.share.max.heartbeat.interval.ms

    The maximum heartbeat interval for share group members.

    Type:int
    Default:15000 (15 seconds)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.share.max.record.lock.duration.ms

    The record acquisition lock maximum duration in milliseconds for share groups.

    Type:int
    Default:60000 (1 minute)
    Valid Values:[30000,...,3600000]
    Importance:medium
    Update Mode:read-only

  • group.share.max.session.timeout.ms

    The maximum allowed session timeout for share group members.

    Type:int
    Default:60000 (1 minute)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.share.max.size

    The maximum number of members that a single share group can accommodate.

    Type:short
    Default:200
    Valid Values:[10,...,1000]
    Importance:medium
    Update Mode:read-only

  • group.share.min.heartbeat.interval.ms

    The minimum heartbeat interval for share group members.

    Type:int
    Default:5000 (5 seconds)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.share.min.record.lock.duration.ms

    The record acquisition lock minimum duration in milliseconds for share groups.

    Type:int
    Default:15000 (15 seconds)
    Valid Values:[1000,...,30000]
    Importance:medium
    Update Mode:read-only

  • group.share.min.session.timeout.ms

    The minimum allowed session timeout for share group members.

    Type:int
    Default:45000 (45 seconds)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • group.share.partition.max.record.locks

    Share-group record lock limit per share-partition.

    Type:int
    Default:200
    Valid Values:[100,...,10000]
    Importance:medium
    Update Mode:read-only

  • group.share.record.lock.duration.ms

    The record acquisition lock duration in milliseconds for share groups.

    Type:int
    Default:30000 (30 seconds)
    Valid Values:[1000,...,60000]
    Importance:medium
    Update Mode:read-only

  • group.share.session.timeout.ms

    The timeout to detect client failures when using the share group protocol.

    Type:int
    Default:45000 (45 seconds)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • initial.broker.registration.timeout.ms

    When initially registering with the controller quorum, the number of milliseconds to wait before declaring failure and exiting the broker process.

    Type:int
    Default:60000 (1 minute)
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • inter.broker.listener.name

    Name of listener used for communication between brokers. If this is unset, the listener name is defined by security.inter.broker.protocolIt is an error to set this and security.inter.broker.protocol properties at the same time.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • inter.broker.protocol.version

    Specify which version of the inter-broker protocol will be used.
    . This is typically bumped after all brokers were upgraded to a new version.
    Example of some valid values are: 0.8.0, 0.8.1, 0.8.1.1, 0.8.2, 0.8.2.0, 0.8.2.1, 0.9.0.0, 0.9.0.1 Check MetadataVersion for the full list.

    Type:string
    Default:3.9-IV0
    Valid Values:[0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.10.0-IV0, 0.10.0-IV1, 0.10.1-IV0, 0.10.1-IV1, 0.10.1-IV2, 0.10.2-IV0, 0.11.0-IV0, 0.11.0-IV1, 0.11.0-IV2, 1.0-IV0, 1.1-IV0, 2.0-IV0, 2.0-IV1, 2.1-IV0, 2.1-IV1, 2.1-IV2, 2.2-IV0, 2.2-IV1, 2.3-IV0, 2.3-IV1, 2.4-IV0, 2.4-IV1, 2.5-IV0, 2.6-IV0, 2.7-IV0, 2.7-IV1, 2.7-IV2, 2.8-IV0, 2.8-IV1, 3.0-IV0, 3.0-IV1, 3.1-IV0, 3.2-IV0, 3.3-IV0, 3.3-IV1, 3.3-IV2, 3.3-IV3, 3.4-IV0, 3.5-IV0, 3.5-IV1, 3.5-IV2, 3.6-IV0, 3.6-IV1, 3.6-IV2, 3.7-IV0, 3.7-IV1, 3.7-IV2, 3.7-IV3, 3.7-IV4, 3.8-IV0, 3.9-IV0, 4.0-IV0, 4.0-IV1]
    Importance:medium
    Update Mode:read-only

  • log.cleaner.backoff.ms

    The amount of time to sleep when there are no logs to clean

    Type:long
    Default:15000 (15 seconds)
    Valid Values:[0,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.cleaner.dedupe.buffer.size

    The total memory used for log deduplication across all cleaner threads

    Type:long
    Default:134217728
    Valid Values:
    Importance:medium
    Update Mode:cluster-wide

  • log.cleaner.delete.retention.ms

    The amount of time to retain tombstone message markers for log compacted topics. This setting also gives a bound on the time in which a consumer must complete a read if they begin from offset 0 to ensure that they get a valid snapshot of the final stage (otherwise tombstones messages may be collected before a consumer completes their scan).

    Type:long
    Default:86400000 (1 day)
    Valid Values:[0,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.cleaner.enable

    Enable the log cleaner process to run on the server. Should be enabled if using any topics with a cleanup.policy=compact including the internal offsets topic. If disabled those topics will not be compacted and continually grow in size.

    Type:boolean
    Default:true
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • log.cleaner.io.buffer.load.factor

    Log cleaner dedupe buffer load factor. The percentage full the dedupe buffer can become. A higher value will allow more log to be cleaned at once but will lead to more hash collisions

    Type:double
    Default:0.9
    Valid Values:
    Importance:medium
    Update Mode:cluster-wide

  • log.cleaner.io.buffer.size

    The total memory used for log cleaner I/O buffers across all cleaner threads

    Type:int
    Default:524288
    Valid Values:[0,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.cleaner.io.max.bytes.per.second

    The log cleaner will be throttled so that the sum of its read and write i/o will be less than this value on average

    Type:double
    Default:1.7976931348623157E308
    Valid Values:
    Importance:medium
    Update Mode:cluster-wide

  • log.cleaner.max.compaction.lag.ms

    The maximum time a message will remain ineligible for compaction in the log. Only applicable for logs that are being compacted.

    Type:long
    Default:9223372036854775807
    Valid Values:[1,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.cleaner.min.cleanable.ratio

    The minimum ratio of dirty log to total log for a log to eligible for cleaning. If the log.cleaner.max.compaction.lag.ms or the log.cleaner.min.compaction.lag.ms configurations are also specified, then the log compactor considers the log eligible for compaction as soon as either: (i) the dirty ratio threshold has been met and the log has had dirty (uncompacted) records for at least the log.cleaner.min.compaction.lag.ms duration, or (ii) if the log has had dirty (uncompacted) records for at most the log.cleaner.max.compaction.lag.ms period.

    Type:double
    Default:0.5
    Valid Values:[0,...,1]
    Importance:medium
    Update Mode:cluster-wide

  • log.cleaner.min.compaction.lag.ms

    The minimum time a message will remain uncompacted in the log. Only applicable for logs that are being compacted.

    Type:long
    Default:0
    Valid Values:[0,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.cleaner.threads

    The number of background threads to use for log cleaning

    Type:int
    Default:1
    Valid Values:[0,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.cleanup.policy

    The default cleanup policy for segments beyond the retention window. A comma separated list of valid policies. Valid policies are: "delete" and "compact"

    Type:list
    Default:delete
    Valid Values:[compact, delete]
    Importance:medium
    Update Mode:cluster-wide

  • log.index.interval.bytes

    The interval with which we add an entry to the offset index.

    Type:int
    Default:4096 (4 kibibytes)
    Valid Values:[0,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.index.size.max.bytes

    The maximum size in bytes of the offset index

    Type:int
    Default:10485760 (10 mebibytes)
    Valid Values:[4,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.local.retention.bytes

    The maximum size of local log segments that can grow for a partition before it gets eligible for deletion. Default value is -2, it represents `log.retention.bytes` value to be used. The effective value should always be less than or equal to `log.retention.bytes` value.

    Type:long
    Default:-2
    Valid Values:[-2,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.local.retention.ms

    The number of milliseconds to keep the local log segments before it gets eligible for deletion. Default value is -2, it represents `log.retention.ms` value is to be used. The effective value should always be less than or equal to `log.retention.ms` value.

    Type:long
    Default:-2
    Valid Values:[-2,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.message.format.version

    Specify the message format version the broker will use to append messages to the logs. The value should be a valid MetadataVersion. Some examples are: 0.8.2, 0.9.0.0, 0.10.0, check MetadataVersion for more details. By setting a particular message format version, the user is certifying that all the existing messages on disk are smaller or equal than the specified version. Setting this value incorrectly will cause consumers with older versions to break as they will receive messages with a format that they don't understand.

    Type:string
    Default:3.0-IV1
    Valid Values:[0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.10.0-IV0, 0.10.0-IV1, 0.10.1-IV0, 0.10.1-IV1, 0.10.1-IV2, 0.10.2-IV0, 0.11.0-IV0, 0.11.0-IV1, 0.11.0-IV2, 1.0-IV0, 1.1-IV0, 2.0-IV0, 2.0-IV1, 2.1-IV0, 2.1-IV1, 2.1-IV2, 2.2-IV0, 2.2-IV1, 2.3-IV0, 2.3-IV1, 2.4-IV0, 2.4-IV1, 2.5-IV0, 2.6-IV0, 2.7-IV0, 2.7-IV1, 2.7-IV2, 2.8-IV0, 2.8-IV1, 3.0-IV0, 3.0-IV1, 3.1-IV0, 3.2-IV0, 3.3-IV0, 3.3-IV1, 3.3-IV2, 3.3-IV3, 3.4-IV0, 3.5-IV0, 3.5-IV1, 3.5-IV2, 3.6-IV0, 3.6-IV1, 3.6-IV2, 3.7-IV0, 3.7-IV1, 3.7-IV2, 3.7-IV3, 3.7-IV4, 3.8-IV0, 3.9-IV0, 4.0-IV0, 4.0-IV1]
    Importance:medium
    Update Mode:read-only

  • log.message.timestamp.after.max.ms

    This configuration sets the allowable timestamp difference between the message timestamp and the broker's timestamp. The message timestamp can be later than or equal to the broker's timestamp, with the maximum allowable difference determined by the value set in this configuration. If log.message.timestamp.type=CreateTime, the message will be rejected if the difference in timestamps exceeds this specified threshold. This configuration is ignored if log.message.timestamp.type=LogAppendTime.

    Type:long
    Default:9223372036854775807
    Valid Values:[0,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.message.timestamp.before.max.ms

    This configuration sets the allowable timestamp difference between the broker's timestamp and the message timestamp. The message timestamp can be earlier than or equal to the broker's timestamp, with the maximum allowable difference determined by the value set in this configuration. If log.message.timestamp.type=CreateTime, the message will be rejected if the difference in timestamps exceeds this specified threshold. This configuration is ignored if log.message.timestamp.type=LogAppendTime.

    Type:long
    Default:9223372036854775807
    Valid Values:[0,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.message.timestamp.difference.max.ms

    [DEPRECATED] The maximum difference allowed between the timestamp when a broker receives a message and the timestamp specified in the message. If log.message.timestamp.type=CreateTime, a message will be rejected if the difference in timestamp exceeds this threshold. This configuration is ignored if log.message.timestamp.type=LogAppendTime.The maximum timestamp difference allowed should be no greater than log.retention.ms to avoid unnecessarily frequent log rolling.

    Type:long
    Default:9223372036854775807
    Valid Values:[0,...]
    Importance:medium
    Update Mode:cluster-wide

  • log.message.timestamp.type

    Define whether the timestamp in the message is message create time or log append time. The value should be either CreateTime or LogAppendTime.

    Type:string
    Default:CreateTime
    Valid Values:[CreateTime, LogAppendTime]
    Importance:medium
    Update Mode:cluster-wide

  • log.preallocate

    Should pre allocate file when create new segment? If you are using Kafka on Windows, you probably need to set it to true.

    Type:boolean
    Default:false
    Valid Values:
    Importance:medium
    Update Mode:cluster-wide

  • log.retention.check.interval.ms

    The frequency in milliseconds that the log cleaner checks whether any log is eligible for deletion

    Type:long
    Default:300000 (5 minutes)
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • max.connection.creation.rate

    The maximum connection creation rate we allow in the broker at any time. Listener-level limits may also be configured by prefixing the config name with the listener prefix, for example, listener.name.internal.max.connection.creation.rate.Broker-wide connection rate limit should be configured based on broker capacity while listener limits should be configured based on application requirements. New connections will be throttled if either the listener or the broker limit is reached, with the exception of inter-broker listener. Connections on the inter-broker listener will be throttled only when the listener-level rate limit is reached.

    Type:int
    Default:2147483647
    Valid Values:[0,...]
    Importance:medium
    Update Mode:cluster-wide

  • max.connections

    The maximum number of connections we allow in the broker at any time. This limit is applied in addition to any per-ip limits configured using max.connections.per.ip. Listener-level limits may also be configured by prefixing the config name with the listener prefix, for example, listener.name.internal.max.connections.per.ip. Broker-wide limit should be configured based on broker capacity while listener limits should be configured based on application requirements. New connections are blocked if either the listener or broker limit is reached. Connections on the inter-broker listener are permitted even if broker-wide limit is reached. The least recently used connection on another listener will be closed in this case.

    Type:int
    Default:2147483647
    Valid Values:[0,...]
    Importance:medium
    Update Mode:cluster-wide

  • max.connections.per.ip

    The maximum number of connections we allow from each ip address. This can be set to 0 if there are overrides configured using max.connections.per.ip.overrides property. New connections from the ip address are dropped if the limit is reached.

    Type:int
    Default:2147483647
    Valid Values:[0,...]
    Importance:medium
    Update Mode:cluster-wide

  • max.connections.per.ip.overrides

    A comma-separated list of per-ip or hostname overrides to the default maximum number of connections. An example value is "hostName:100,127.0.0.1:200"

    Type:string
    Default:""
    Valid Values:
    Importance:medium
    Update Mode:cluster-wide

  • max.incremental.fetch.session.cache.slots

    The maximum number of total incremental fetch sessions that we will maintain. FetchSessionCache is sharded into 8 shards and the limit is equally divided among all shards. Sessions are allocated to each shard in round-robin. Only entries within a shard are considered eligible for eviction.

    Type:int
    Default:1000
    Valid Values:[0,...]
    Importance:medium
    Update Mode:read-only

  • max.request.partition.size.limit

    The maximum number of partitions can be served in one request.

    Type:int
    Default:2000
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • num.partitions

    The default number of log partitions per topic

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • password.encoder.old.secret

    The old secret that was used for encoding dynamically configured passwords. This is required only when the secret is updated. If specified, all dynamically encoded passwords are decoded using this old secret and re-encoded using password.encoder.secret when broker starts up.

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • password.encoder.secret

    The secret used for encoding dynamically configured passwords for this broker.

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • principal.builder.class

    The fully qualified name of a class that implements the KafkaPrincipalBuilder interface, which is used to build the KafkaPrincipal object used during authorization. If no principal builder is defined, the default behavior depends on the security protocol in use. For SSL authentication, the principal will be derived using the rules defined by ssl.principal.mapping.rules applied on the distinguished name from the client certificate if one is provided; otherwise, if client authentication is not required, the principal name will be ANONYMOUS. For SASL authentication, the principal will be derived using the rules defined by sasl.kerberos.principal.to.local.rules if GSSAPI is in use, and the SASL authentication ID for other mechanisms. For PLAINTEXT, the principal will be ANONYMOUS.

    Type:class
    Default:org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • producer.purgatory.purge.interval.requests

    The purge interval (in number of requests) of the producer request purgatory

    Type:int
    Default:1000
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • queued.max.request.bytes

    The number of queued bytes allowed before no more requests are read

    Type:long
    Default:-1
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • remote.fetch.max.wait.ms

    The maximum amount of time the server will wait before answering the remote fetch request

    Type:int
    Default:500
    Valid Values:[1,...]
    Importance:medium
    Update Mode:cluster-wide

  • remote.log.manager.copier.thread.pool.size

    Size of the thread pool used in scheduling tasks to copy segments. The default value of -1 means that this will be set to the configured value of remote.log.manager.thread.pool.size, if available; otherwise, it defaults to 10.

    Type:int
    Default:-1
    Valid Values:The default value of -1 means that this will be set to the configured value of remote.log.manager.thread.pool.size, if available; otherwise, it defaults to 10.
    Importance:medium
    Update Mode:read-only

  • remote.log.manager.copy.max.bytes.per.second

    The maximum number of bytes that can be copied from local storage to remote storage per second. This is a global limit for all the partitions that are being copied from local storage to remote storage. The default value is Long.MAX_VALUE, which means there is no limit on the number of bytes that can be copied per second.

    Type:long
    Default:9223372036854775807
    Valid Values:[1,...]
    Importance:medium
    Update Mode:cluster-wide

  • remote.log.manager.copy.quota.window.num

    The number of samples to retain in memory for remote copy quota management. The default value is 11, which means there are 10 whole windows + 1 current window.

    Type:int
    Default:11
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • remote.log.manager.copy.quota.window.size.seconds

    The time span of each sample for remote copy quota management. The default value is 1 second.

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • remote.log.manager.expiration.thread.pool.size

    Size of the thread pool used in scheduling tasks to clean up remote log segments. The default value of -1 means that this will be set to the configured value of remote.log.manager.thread.pool.size, if available; otherwise, it defaults to 10.

    Type:int
    Default:-1
    Valid Values:The default value of -1 means that this will be set to the configured value of remote.log.manager.thread.pool.size, if available; otherwise, it defaults to 10.
    Importance:medium
    Update Mode:read-only

  • remote.log.manager.fetch.max.bytes.per.second

    The maximum number of bytes that can be fetched from remote storage to local storage per second. This is a global limit for all the partitions that are being fetched from remote storage to local storage. The default value is Long.MAX_VALUE, which means there is no limit on the number of bytes that can be fetched per second.

    Type:long
    Default:9223372036854775807
    Valid Values:[1,...]
    Importance:medium
    Update Mode:cluster-wide

  • remote.log.manager.fetch.quota.window.num

    The number of samples to retain in memory for remote fetch quota management. The default value is 11, which means there are 10 whole windows + 1 current window.

    Type:int
    Default:11
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • remote.log.manager.fetch.quota.window.size.seconds

    The time span of each sample for remote fetch quota management. The default value is 1 second.

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • remote.log.manager.thread.pool.size

    Deprecated. Size of the thread pool used in scheduling tasks to copy segments, fetch remote log indexes and clean up remote log segments.

    Type:int
    Default:10
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • remote.log.metadata.manager.class.name

    Fully qualified class name of `RemoteLogMetadataManager` implementation.

    Type:string
    Default:org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
    Valid Values:non-empty string
    Importance:medium
    Update Mode:read-only

  • remote.log.metadata.manager.class.path

    Class path of the `RemoteLogMetadataManager` implementation. If specified, the RemoteLogMetadataManager implementation and its dependent libraries will be loaded by a dedicated classloader which searches this class path before the Kafka broker class path. The syntax of this parameter is same as the standard Java class path string.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • remote.log.metadata.manager.impl.prefix

    Prefix used for properties to be passed to RemoteLogMetadataManager implementation. For example this value can be `rlmm.config.`.

    Type:string
    Default:rlmm.config.
    Valid Values:non-empty string
    Importance:medium
    Update Mode:read-only

  • remote.log.metadata.manager.listener.name

    Listener name of the local broker to which it should get connected if needed by RemoteLogMetadataManager implementation.

    Type:string
    Default:null
    Valid Values:non-empty string
    Importance:medium
    Update Mode:read-only

  • remote.log.reader.max.pending.tasks

    Maximum remote log reader thread pool task queue size. If the task queue is full, fetch requests are served with an error.

    Type:int
    Default:100
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • remote.log.reader.threads

    Size of the thread pool that is allocated for handling remote log reads.

    Type:int
    Default:10
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • remote.log.storage.manager.class.name

    Fully qualified class name of `RemoteStorageManager` implementation.

    Type:string
    Default:null
    Valid Values:non-empty string
    Importance:medium
    Update Mode:read-only

  • remote.log.storage.manager.class.path

    Class path of the `RemoteStorageManager` implementation. If specified, the RemoteStorageManager implementation and its dependent libraries will be loaded by a dedicated classloader which searches this class path before the Kafka broker class path. The syntax of this parameter is same as the standard Java class path string.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • remote.log.storage.manager.impl.prefix

    Prefix used for properties to be passed to RemoteStorageManager implementation. For example this value can be `rsm.config.`.

    Type:string
    Default:rsm.config.
    Valid Values:non-empty string
    Importance:medium
    Update Mode:read-only

  • remote.log.storage.system.enable

    Whether to enable tiered storage functionality in a broker or not. Valid values are `true` or `false` and the default value is false. When it is true broker starts all the services required for the tiered storage functionality.

    Type:boolean
    Default:false
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • replica.fetch.backoff.ms

    The amount of time to sleep when fetch partition error occurs.

    Type:int
    Default:1000 (1 second)
    Valid Values:[0,...]
    Importance:medium
    Update Mode:read-only

  • replica.fetch.max.bytes

    The number of bytes of messages to attempt to fetch for each partition. This is not an absolute maximum, if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config).

    Type:int
    Default:1048576 (1 mebibyte)
    Valid Values:[0,...]
    Importance:medium
    Update Mode:read-only

  • replica.fetch.response.max.bytes

    Maximum bytes expected for the entire fetch response. Records are fetched in batches, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config).

    Type:int
    Default:10485760 (10 mebibytes)
    Valid Values:[0,...]
    Importance:medium
    Update Mode:read-only

  • replica.selector.class

    The fully qualified class name that implements ReplicaSelector. This is used by the broker to find the preferred read replica. By default, we use an implementation that returns the leader.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • reserved.broker.max.id

    Max number that can be used for a broker.id

    Type:int
    Default:1000
    Valid Values:[0,...]
    Importance:medium
    Update Mode:read-only

  • sasl.client.callback.handler.class

    The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface.

    Type:class
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • sasl.enabled.mechanisms

    The list of SASL mechanisms enabled in the Kafka server. The list may contain any mechanism for which a security provider is available. Only GSSAPI is enabled by default.

    Type:list
    Default:GSSAPI
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.jaas.config

    JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is: loginModuleClass controlFlag (optionName=optionValue)*;. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.kerberos.kinit.cmd

    Kerberos kinit command path.

    Type:string
    Default:/usr/bin/kinit
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.kerberos.min.time.before.relogin

    Login thread sleep time between refresh attempts.

    Type:long
    Default:60000
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.kerberos.principal.to.local.rules

    A list of rules for mapping from principal names to short names (typically operating system usernames). The rules are evaluated in order and the first rule that matches a principal name is used to map it to a short name. Any later rules in the list are ignored. By default, principal names of the form {username}/{hostname}@{REALM} are mapped to {username}. For more details on the format please see security authorization and acls. Note that this configuration is ignored if an extension of KafkaPrincipalBuilder is provided by the principal.builder.class configuration.

    Type:list
    Default:DEFAULT
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.kerberos.service.name

    The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.kerberos.ticket.renew.jitter

    Percentage of random jitter added to the renewal time.

    Type:double
    Default:0.05
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.kerberos.ticket.renew.window.factor

    Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.

    Type:double
    Default:0.8
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.login.callback.handler.class

    The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler

    Type:class
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • sasl.login.class

    The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin

    Type:class
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • sasl.login.refresh.buffer.seconds

    The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.

    Type:short
    Default:300
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.login.refresh.min.period.seconds

    The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.

    Type:short
    Default:60
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.login.refresh.window.factor

    Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER.

    Type:double
    Default:0.8
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.login.refresh.window.jitter

    The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.

    Type:double
    Default:0.05
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.mechanism.inter.broker.protocol

    SASL mechanism used for inter-broker communication. Default is GSSAPI.

    Type:string
    Default:GSSAPI
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • sasl.oauthbearer.jwks.endpoint.url

    The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. If the URL is HTTP(S)-based, the JWKS data will be retrieved from the OAuth/OIDC provider via the configured URL on broker startup. All then-current keys will be cached on the broker for incoming requests. If an authentication request is received for a JWT that includes a "kid" header claim value that isn't yet in the cache, the JWKS endpoint will be queried again on demand. However, the broker polls the URL every sasl.oauthbearer.jwks.endpoint.refresh.ms milliseconds to refresh the cache with any forthcoming keys before any JWT requests that include them are received. If the URL is file-based, the broker will load the JWKS file from a configured location on startup. In the event that the JWT includes a "kid" header value that isn't in the JWKS file, the broker will reject the JWT and authentication will fail.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • sasl.oauthbearer.token.endpoint.url

    The URL for the OAuth/OIDC identity provider. If the URL is HTTP(S)-based, it is the issuer's token endpoint URL to which requests will be made to login based on the configuration in sasl.jaas.config. If the URL is file-based, it specifies a file containing an access token (in JWT serialized form) issued by the OAuth/OIDC identity provider to use for authorization.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • sasl.server.callback.handler.class

    The fully qualified name of a SASL server callback handler class that implements the AuthenticateCallbackHandler interface. Server callback handlers must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.plain.sasl.server.callback.handler.class=com.example.CustomPlainCallbackHandler.

    Type:class
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • sasl.server.max.receive.size

    The maximum receive size allowed before and during initial SASL authentication. Default receive size is 512KB. GSSAPI limits requests to 64K, but we allow upto 512KB by default for custom SASL mechanisms. In practice, PLAIN, SCRAM and OAUTH mechanisms can use much smaller limits.

    Type:int
    Default:524288
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • security.inter.broker.protocol

    Security protocol used to communicate between brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL. It is an error to set this and inter.broker.listener.name properties at the same time.

    Type:string
    Default:PLAINTEXT
    Valid Values:[PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL]
    Importance:medium
    Update Mode:read-only

  • socket.connection.setup.timeout.max.ms

    The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. To avoid connection storms, a randomization factor of 0.2 will be applied to the timeout resulting in a random range between 20% below and 20% above the computed value.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • socket.connection.setup.timeout.ms

    The amount of time the client will wait for the socket connection to be established. If the connection is not built before the timeout elapses, clients will close the socket channel. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the socket.connection.setup.timeout.max.ms value.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • socket.listen.backlog.size

    The maximum number of pending connections on the socket. In Linux, you may also need to configure somaxconn and tcp_max_syn_backlog kernel parameters accordingly to make the configuration takes effect.

    Type:int
    Default:50
    Valid Values:[1,...]
    Importance:medium
    Update Mode:read-only

  • ssl.cipher.suites

    A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.

    Type:list
    Default:""
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.client.auth

    Configures kafka broker to request client authentication. The following settings are common:

    • ssl.client.auth=required If set to required client authentication is required.
    • ssl.client.auth=requested This means client authentication is optional. unlike required, if this option is set client can choose not to provide authentication information about itself
    • ssl.client.auth=none This means client authentication is not needed.

    Type:string
    Default:none
    Valid Values:[required, requested, none]
    Importance:medium
    Update Mode:per-broker

  • ssl.enabled.protocols

    The list of protocols enabled for SSL connections. The default is 'TLSv1.2,TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). This default should be fine for most cases. Also see the config documentation for `ssl.protocol`.

    Type:list
    Default:TLSv1.2
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.key.password

    The password of the private key in the key store file or the PEM key specified in 'ssl.keystore.key'.

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.keymanager.algorithm

    The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.

    Type:string
    Default:SunX509
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.keystore.certificate.chain

    Certificate chain in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with a list of X.509 certificates

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.keystore.key

    Private key in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with PKCS#8 keys. If the key is encrypted, key password must be specified using 'ssl.key.password'

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.keystore.location

    The location of the key store file. This is optional for client and can be used for two-way authentication for client.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.keystore.password

    The store password for the key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured. Key store password is not supported for PEM format.

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.keystore.type

    The file format of the key store file. This is optional for client. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].

    Type:string
    Default:JKS
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.protocol

    The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'.

    Type:string
    Default:TLSv1.2
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.provider

    The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.trustmanager.algorithm

    The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.

    Type:string
    Default:PKIX
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.truststore.certificates

    Trusted certificates in the format specified by 'ssl.truststore.type'. Default SSL engine factory supports only PEM format with X.509 certificates.

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.truststore.location

    The location of the trust store file.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.truststore.password

    The password for the trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled. Trust store password is not supported for PEM format.

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • ssl.truststore.type

    The file format of the trust store file. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].

    Type:string
    Default:JKS
    Valid Values:
    Importance:medium
    Update Mode:per-broker

  • zookeeper.clientCnxnSocket

    Typically set to org.apache.zookeeper.ClientCnxnSocketNetty when using TLS connectivity to ZooKeeper. Overrides any explicit value set via the same-named zookeeper.clientCnxnSocket system property.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • zookeeper.ssl.client.enable

    Set client to use TLS when connecting to ZooKeeper. An explicit value overrides any value set via the zookeeper.client.secure system property (note the different name). Defaults to false if neither is set; when true, zookeeper.clientCnxnSocket must be set (typically to org.apache.zookeeper.ClientCnxnSocketNetty); other values to set may include zookeeper.ssl.cipher.suites, zookeeper.ssl.crl.enable, zookeeper.ssl.enabled.protocols, zookeeper.ssl.endpoint.identification.algorithm, zookeeper.ssl.keystore.location, zookeeper.ssl.keystore.password, zookeeper.ssl.keystore.type, zookeeper.ssl.ocsp.enable, zookeeper.ssl.protocol, zookeeper.ssl.truststore.location, zookeeper.ssl.truststore.password, zookeeper.ssl.truststore.type

    Type:boolean
    Default:false
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • zookeeper.ssl.keystore.location

    Keystore location when using a client-side certificate with TLS connectivity to ZooKeeper. Overrides any explicit value set via the zookeeper.ssl.keyStore.location system property (note the camelCase).

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • zookeeper.ssl.keystore.password

    Keystore password when using a client-side certificate with TLS connectivity to ZooKeeper. Overrides any explicit value set via the zookeeper.ssl.keyStore.password system property (note the camelCase). Note that ZooKeeper does not support a key password different from the keystore password, so be sure to set the key password in the keystore to be identical to the keystore password; otherwise the connection attempt to Zookeeper will fail.

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • zookeeper.ssl.keystore.type

    Keystore type when using a client-side certificate with TLS connectivity to ZooKeeper. Overrides any explicit value set via the zookeeper.ssl.keyStore.type system property (note the camelCase). The default value of null means the type will be auto-detected based on the filename extension of the keystore.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • zookeeper.ssl.truststore.location

    Truststore location when using TLS connectivity to ZooKeeper. Overrides any explicit value set via the zookeeper.ssl.trustStore.location system property (note the camelCase).

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • zookeeper.ssl.truststore.password

    Truststore password when using TLS connectivity to ZooKeeper. Overrides any explicit value set via the zookeeper.ssl.trustStore.password system property (note the camelCase).

    Type:password
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • zookeeper.ssl.truststore.type

    Truststore type when using TLS connectivity to ZooKeeper. Overrides any explicit value set via the zookeeper.ssl.trustStore.type system property (note the camelCase). The default value of null means the type will be auto-detected based on the filename extension of the truststore.

    Type:string
    Default:null
    Valid Values:
    Importance:medium
    Update Mode:read-only

  • alter.config.policy.class.name

    The alter configs policy class that should be used for validation. The class should implement the org.apache.kafka.server.policy.AlterConfigPolicy interface.

    Type:class
    Default:null
    Valid Values:
    Importance:low
    Update Mode:read-only

  • alter.log.dirs.replication.quota.window.num

    The number of samples to retain in memory for alter log dirs replication quotas

    Type:int
    Default:11
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • alter.log.dirs.replication.quota.window.size.seconds

    The time span of each sample for alter log dirs replication quotas

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • authorizer.class.name

    The fully qualified name of a class that implements org.apache.kafka.server.authorizer.Authorizer interface, which is used by the broker for authorization.

    Type:string
    Default:""
    Valid Values:non-null string
    Importance:low
    Update Mode:read-only

  • auto.include.jmx.reporter

    Deprecated. Whether to automatically include JmxReporter even if it's not listed in metric.reporters. This configuration will be removed in Kafka 4.0, users should instead include org.apache.kafka.common.metrics.JmxReporter in metric.reporters in order to enable the JmxReporter.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low
    Update Mode:read-only

  • client.quota.callback.class

    The fully qualified name of a class that implements the ClientQuotaCallback interface, which is used to determine quota limits applied to client requests. By default, the <user> and <client-id> quotas that are stored in ZooKeeper are applied. For any given request, the most specific quota that matches the user principal of the session and the client-id of the request is applied.

    Type:class
    Default:null
    Valid Values:
    Importance:low
    Update Mode:read-only

  • connection.failed.authentication.delay.ms

    Connection close delay on failed authentication: this is the time (in milliseconds) by which connection close will be delayed on authentication failure. This must be configured to be less than connections.max.idle.ms to prevent connection timeout.

    Type:int
    Default:100
    Valid Values:[0,...]
    Importance:low
    Update Mode:read-only

  • controller.quorum.retry.backoff.ms

    The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. This value is the initial backoff value and will increase exponentially for each failed request, up to the retry.backoff.max.ms value.

    Type:int
    Default:20
    Valid Values:
    Importance:low
    Update Mode:read-only

  • controller.quota.window.num

    The number of samples to retain in memory for controller mutation quotas

    Type:int
    Default:11
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • controller.quota.window.size.seconds

    The time span of each sample for controller mutations quotas

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • create.topic.policy.class.name

    The create topic policy class that should be used for validation. The class should implement the org.apache.kafka.server.policy.CreateTopicPolicy interface.

    Type:class
    Default:null
    Valid Values:
    Importance:low
    Update Mode:read-only

  • delegation.token.expiry.check.interval.ms

    Scan interval to remove expired delegation tokens.

    Type:long
    Default:3600000 (1 hour)
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • kafka.metrics.polling.interval.secs

    The metrics polling interval (in seconds) which can be used in kafka.metrics.reporters implementations.

    Type:int
    Default:10
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • kafka.metrics.reporters

    A list of classes to use as Yammer metrics custom reporters. The reporters should implement kafka.metrics.KafkaMetricsReporter trait. If a client wants to expose JMX operations on a custom reporter, the custom reporter needs to additionally implement an MBean trait that extends kafka.metrics.KafkaMetricsReporterMBean trait so that the registered MBean is compliant with the standard MBean convention.

    Type:list
    Default:""
    Valid Values:
    Importance:low
    Update Mode:read-only

  • listener.security.protocol.map

    Map between listener names and security protocols. This must be defined for the same security protocol to be usable in more than one port or IP. For example, internal and external traffic can be separated even if SSL is required for both. Concretely, the user could define listeners with names INTERNAL and EXTERNAL and this property as: INTERNAL:SSL,EXTERNAL:SSL. As shown, key and value are separated by a colon and map entries are separated by commas. Each listener name should only appear once in the map. Different security (SSL and SASL) settings can be configured for each listener by adding a normalised prefix (the listener name is lowercased) to the config name. For example, to set a different keystore for the INTERNAL listener, a config with name listener.name.internal.ssl.keystore.location would be set. If the config for the listener name is not set, the config will fallback to the generic config (i.e. ssl.keystore.location). Note that in KRaft a default mapping from the listener names defined by controller.listener.names to PLAINTEXT is assumed if no explicit mapping is provided and no other security protocol is in use.

    Type:string
    Default:SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT
    Valid Values:
    Importance:low
    Update Mode:per-broker

  • log.dir.failure.timeout.ms

    If the broker is unable to successfully communicate to the controller that some log directory has failed for longer than this time, the broker will fail and shut down.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • log.message.downconversion.enable

    This configuration controls whether down-conversion of message formats is enabled to satisfy consume requests. When set to false, broker will not perform down-conversion for consumers expecting an older message format. The broker responds with UNSUPPORTED_VERSION error for consume requests from such older clients. This configurationdoes not apply to any message format conversion that might be required for replication to followers.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low
    Update Mode:cluster-wide

  • metadata.max.idle.interval.ms

    This configuration controls how often the active controller should write no-op records to the metadata partition. If the value is 0, no-op records are not appended to the metadata partition. The default value is 500

    Type:int
    Default:500
    Valid Values:[0,...]
    Importance:low
    Update Mode:read-only

  • metric.reporters

    A list of classes to use as metrics reporters. Implementing the org.apache.kafka.common.metrics.MetricsReporter interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.

    Type:list
    Default:""
    Valid Values:
    Importance:low
    Update Mode:cluster-wide

  • metrics.num.samples

    The number of samples maintained to compute metrics.

    Type:int
    Default:2
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • metrics.recording.level

    The highest recording level for metrics.

    Type:string
    Default:INFO
    Valid Values:
    Importance:low
    Update Mode:read-only

  • metrics.sample.window.ms

    The window of time a metrics sample is computed over.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • password.encoder.cipher.algorithm

    The Cipher algorithm used for encoding dynamically configured passwords.

    Type:string
    Default:AES/CBC/PKCS5Padding
    Valid Values:
    Importance:low
    Update Mode:read-only

  • password.encoder.iterations

    The iteration count used for encoding dynamically configured passwords.

    Type:int
    Default:4096
    Valid Values:[1024,...]
    Importance:low
    Update Mode:read-only

  • password.encoder.key.length

    The key length used for encoding dynamically configured passwords.

    Type:int
    Default:128
    Valid Values:[8,...]
    Importance:low
    Update Mode:read-only

  • password.encoder.keyfactory.algorithm

    The SecretKeyFactory algorithm used for encoding dynamically configured passwords. Default is PBKDF2WithHmacSHA512 if available and PBKDF2WithHmacSHA1 otherwise.

    Type:string
    Default:null
    Valid Values:
    Importance:low
    Update Mode:read-only

  • producer.id.expiration.ms

    The time in ms that a topic partition leader will wait before expiring producer IDs. Producer IDs will not expire while a transaction associated to them is still ongoing. Note that producer IDs may expire sooner if the last write from the producer ID is deleted due to the topic's retention settings. Setting this value the same or higher than delivery.timeout.ms can help prevent expiration during retries and protect against message duplication, but the default should be reasonable for most use cases.

    Type:int
    Default:86400000 (1 day)
    Valid Values:[1,...]
    Importance:low
    Update Mode:cluster-wide

  • quota.window.num

    The number of samples to retain in memory for client quotas

    Type:int
    Default:11
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • quota.window.size.seconds

    The time span of each sample for client quotas

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • remote.log.index.file.cache.total.size.bytes

    The total size of the space allocated to store index files fetched from remote storage in the local storage.

    Type:long
    Default:1073741824 (1 gibibyte)
    Valid Values:[1,...]
    Importance:low
    Update Mode:cluster-wide

  • remote.log.manager.task.interval.ms

    Interval at which remote log manager runs the scheduled tasks like copy segments, and clean up remote log segments.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • remote.log.metadata.custom.metadata.max.bytes

    The maximum size of custom metadata in bytes that the broker should accept from a remote storage plugin. If custom metadata exceeds this limit, the updated segment metadata will not be stored, the copied data will be attempted to delete, and the remote copying task for this topic-partition will stop with an error.

    Type:int
    Default:128
    Valid Values:[0,...]
    Importance:low
    Update Mode:read-only

  • replication.quota.window.num

    The number of samples to retain in memory for replication quotas

    Type:int
    Default:11
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • replication.quota.window.size.seconds

    The time span of each sample for replication quotas

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • sasl.login.connect.timeout.ms

    The (optional) value in milliseconds for the external authentication provider connection timeout. Currently applies only to OAUTHBEARER.

    Type:int
    Default:null
    Valid Values:
    Importance:low
    Update Mode:read-only

  • sasl.login.read.timeout.ms

    The (optional) value in milliseconds for the external authentication provider read timeout. Currently applies only to OAUTHBEARER.

    Type:int
    Default:null
    Valid Values:
    Importance:low
    Update Mode:read-only

  • sasl.login.retry.backoff.max.ms

    The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:low
    Update Mode:read-only

  • sasl.login.retry.backoff.ms

    The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.

    Type:long
    Default:100
    Valid Values:
    Importance:low
    Update Mode:read-only

  • sasl.oauthbearer.clock.skew.seconds

    The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker.

    Type:int
    Default:30
    Valid Values:
    Importance:low
    Update Mode:read-only

  • sasl.oauthbearer.expected.audience

    The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWT's "aud" claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.

    Type:list
    Default:null
    Valid Values:
    Importance:low
    Update Mode:read-only

  • sasl.oauthbearer.expected.issuer

    The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth "iss" claim and if this value is set, the broker will match it exactly against what is in the JWT's "iss" claim. If there is no match, the broker will reject the JWT and authentication will fail.

    Type:string
    Default:null
    Valid Values:
    Importance:low
    Update Mode:read-only

  • sasl.oauthbearer.jwks.endpoint.refresh.ms

    The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.

    Type:long
    Default:3600000 (1 hour)
    Valid Values:
    Importance:low
    Update Mode:read-only

  • sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms

    The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:low
    Update Mode:read-only

  • sasl.oauthbearer.jwks.endpoint.retry.backoff.ms

    The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.

    Type:long
    Default:100
    Valid Values:
    Importance:low
    Update Mode:read-only

  • sasl.oauthbearer.scope.claim.name

    The OAuth claim for the scope is often named "scope", but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

    Type:string
    Default:scope
    Valid Values:
    Importance:low
    Update Mode:read-only

  • sasl.oauthbearer.sub.claim.name

    The OAuth claim for the subject is often named "sub", but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

    Type:string
    Default:sub
    Valid Values:
    Importance:low
    Update Mode:read-only

  • security.providers

    A list of configurable creator classes each returning a provider implementing security algorithms. These classes should implement the org.apache.kafka.common.security.auth.SecurityProviderCreator interface.

    Type:string
    Default:null
    Valid Values:
    Importance:low
    Update Mode:read-only

  • ssl.allow.dn.changes

    Indicates whether changes to the certificate distinguished name should be allowed during a dynamic reconfiguration of certificates or not.

    Type:boolean
    Default:false
    Valid Values:
    Importance:low
    Update Mode:read-only

  • ssl.allow.san.changes

    Indicates whether changes to the certificate subject alternative names should be allowed during a dynamic reconfiguration of certificates or not.

    Type:boolean
    Default:false
    Valid Values:
    Importance:low
    Update Mode:read-only

  • ssl.endpoint.identification.algorithm

    The endpoint identification algorithm to validate server hostname using server certificate.

    Type:string
    Default:https
    Valid Values:
    Importance:low
    Update Mode:per-broker

  • ssl.engine.factory.class

    The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is org.apache.kafka.common.security.ssl.DefaultSslEngineFactory. Alternatively, setting this to org.apache.kafka.common.security.ssl.CommonNameLoggingSslEngineFactory will log the common name of expired SSL certificates used by clients to authenticate at any of the brokers with log level INFO. Note that this will cause a tiny delay during establishment of new connections from mTLS clients to brokers due to the extra code for examining the certificate chain provided by the client. Note further that the implementation uses a custom truststore based on the standard Java truststore and thus might be considered a security risk due to not being as mature as the standard one.

    Type:class
    Default:null
    Valid Values:
    Importance:low
    Update Mode:per-broker

  • ssl.principal.mapping.rules

    A list of rules for mapping from distinguished name from the client certificate to short name. The rules are evaluated in order and the first rule that matches a principal name is used to map it to a short name. Any later rules in the list are ignored. By default, distinguished name of the X.500 certificate will be the principal. For more details on the format please see security authorization and acls. Note that this configuration is ignored if an extension of KafkaPrincipalBuilder is provided by the principal.builder.class configuration.

    Type:string
    Default:DEFAULT
    Valid Values:
    Importance:low
    Update Mode:read-only

  • ssl.secure.random.implementation

    The SecureRandom PRNG implementation to use for SSL cryptography operations.

    Type:string
    Default:null
    Valid Values:
    Importance:low
    Update Mode:per-broker

  • telemetry.max.bytes

    The maximum size (after compression if compression is used) of telemetry metrics pushed from a client to the broker. The default value is 1048576 (1 MB).

    Type:int
    Default:1048576 (1 mebibyte)
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • transaction.abort.timed.out.transaction.cleanup.interval.ms

    The interval at which to rollback transactions that have timed out

    Type:int
    Default:10000 (10 seconds)
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • transaction.partition.verification.enable

    Enable verification that checks that the partition has been added to the transaction before writing transactional records to the partition

    Type:boolean
    Default:true
    Valid Values:
    Importance:low
    Update Mode:cluster-wide

  • transaction.remove.expired.transaction.cleanup.interval.ms

    The interval at which to remove transactions that have expired due to transactional.id.expiration.ms passing

    Type:int
    Default:3600000 (1 hour)
    Valid Values:[1,...]
    Importance:low
    Update Mode:read-only

  • zookeeper.ssl.cipher.suites

    Specifies the enabled cipher suites to be used in ZooKeeper TLS negotiation (csv). Overrides any explicit value set via the zookeeper.ssl.ciphersuites system property (note the single word "ciphersuites"). The default value of null means the list of enabled cipher suites is determined by the Java runtime being used.

    Type:list
    Default:null
    Valid Values:
    Importance:low
    Update Mode:read-only

  • zookeeper.ssl.crl.enable

    Specifies whether to enable Certificate Revocation List in the ZooKeeper TLS protocols. Overrides any explicit value set via the zookeeper.ssl.crl system property (note the shorter name).

    Type:boolean
    Default:false
    Valid Values:
    Importance:low
    Update Mode:read-only

  • zookeeper.ssl.enabled.protocols

    Specifies the enabled protocol(s) in ZooKeeper TLS negotiation (csv). Overrides any explicit value set via the zookeeper.ssl.enabledProtocols system property (note the camelCase). The default value of null means the enabled protocol will be the value of the zookeeper.ssl.protocol configuration property.

    Type:list
    Default:null
    Valid Values:
    Importance:low
    Update Mode:read-only

  • zookeeper.ssl.endpoint.identification.algorithm

    Specifies whether to enable hostname verification in the ZooKeeper TLS negotiation process, with (case-insensitively) "https" meaning ZooKeeper hostname verification is enabled and an explicit blank value meaning it is disabled (disabling it is only recommended for testing purposes). An explicit value overrides any "true" or "false" value set via the zookeeper.ssl.hostnameVerification system property (note the different name and values; true implies https and false implies blank).

    Type:string
    Default:HTTPS
    Valid Values:
    Importance:low
    Update Mode:read-only

  • zookeeper.ssl.ocsp.enable

    Specifies whether to enable Online Certificate Status Protocol in the ZooKeeper TLS protocols. Overrides any explicit value set via the zookeeper.ssl.ocsp system property (note the shorter name).

    Type:boolean
    Default:false
    Valid Values:
    Importance:low
    Update Mode:read-only

  • zookeeper.ssl.protocol

    Specifies the protocol to be used in ZooKeeper TLS negotiation. An explicit value overrides any value set via the same-named zookeeper.ssl.protocol system property.

    Type:string
    Default:TLSv1.2
    Valid Values:
    Importance:low
    Update Mode:read-only

More details about broker configuration can be found in the scala class kafka.server.KafkaConfig.

3.1.1 Updating Broker Configs

From Kafka version 1.1 onwards, some of the broker configs can be updated without restarting the broker. See the Dynamic Update Mode column in Broker Configs for the update mode of each broker config.
  • read-only: Requires a broker restart for update
  • per-broker: May be updated dynamically for each broker
  • cluster-wide: May be updated dynamically as a cluster-wide default. May also be updated as a per-broker value for testing.
To alter the current broker configs for broker id 0 (for example, the number of log cleaner threads): 
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --add-config log.cleaner.threads=2
To describe the current dynamic broker configs for broker id 0: 
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --describe
To delete a config override and revert to the statically configured or default value for broker id 0 (for example, the number of log cleaner threads): 
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --delete-config log.cleaner.threads
Some configs may be configured as a cluster-wide default to maintain consistent values across the whole cluster. All brokers in the cluster will process the cluster default update. For example, to update log cleaner threads on all brokers: 
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config log.cleaner.threads=2
To describe the currently configured dynamic cluster-wide default configs: 
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --describe
All configs that are configurable at cluster level may also be configured at per-broker level (e.g. for testing). If a config value is defined at different levels, the following order of precedence is used:
  • Dynamic per-broker config stored in ZooKeeper
  • Dynamic cluster-wide default config stored in ZooKeeper
  • Static broker config from server.properties
  • Kafka default, see broker configs
Updating Password Configs Dynamically

Password config values that are dynamically updated are encrypted before storing in ZooKeeper. The broker config password.encoder.secret must be configured in server.properties to enable dynamic update of password configs. The secret may be different on different brokers.

The secret used for password encoding may be rotated with a rolling restart of brokers. The old secret used for encoding passwords currently in ZooKeeper must be provided in the static broker config password.encoder.old.secret and the new secret must be provided in password.encoder.secret. All dynamic password configs stored in ZooKeeper will be re-encoded with the new secret when the broker starts up.

In Kafka 1.1.x, all dynamically updated password configs must be provided in every alter request when updating configs using kafka-configs.sh even if the password config is not being altered. This constraint will be removed in a future release.

Updating Password Configs in ZooKeeper Before Starting Brokers
From Kafka 2.0.0 onwards, kafka-configs.sh enables dynamic broker configs to be updated using ZooKeeper before starting brokers for bootstrapping. This enables all password configs to be stored in encrypted form, avoiding the need for clear passwords in server.properties. The broker config password.encoder.secret must also be specified if any password configs are included in the alter command. Additional encryption parameters may also be specified. Password encoder configs will not be persisted in ZooKeeper. For example, to store SSL key password for listener INTERNAL on broker 0: 
$ bin/kafka-configs.sh --zookeeper localhost:2182 --zk-tls-config-file zk_tls_config.properties --entity-type brokers --entity-name 0 --alter --add-config
    'listener.name.internal.ssl.key.password=key-password,password.encoder.secret=secret,password.encoder.iterations=8192'
The configuration listener.name.internal.ssl.key.password will be persisted in ZooKeeper in encrypted form using the provided encoder configs. The encoder secret and iterations are not persisted in ZooKeeper.
Updating SSL Keystore of an Existing Listener
Brokers may be configured with SSL keystores with short validity periods to reduce the risk of compromised certificates. Keystores may be updated dynamically without restarting the broker. The config name must be prefixed with the listener prefix listener.name.{listenerName}. so that only the keystore config of a specific listener is updated. The following configs may be updated in a single alter request at per-broker level:
  • ssl.keystore.type
  • ssl.keystore.location
  • ssl.keystore.password
  • ssl.key.password
If the listener is the inter-broker listener, the update is allowed only if the new keystore is trusted by the truststore configured for that listener. For other listeners, no trust validation is performed on the keystore by the broker. Certificates must be signed by the same certificate authority that signed the old certificate to avoid any client authentication failures.
Updating SSL Truststore of an Existing Listener
Broker truststores may be updated dynamically without restarting the broker to add or remove certificates. Updated truststore will be used to authenticate new client connections. The config name must be prefixed with the listener prefix listener.name.{listenerName}. so that only the truststore config of a specific listener is updated. The following configs may be updated in a single alter request at per-broker level:
  • ssl.truststore.type
  • ssl.truststore.location
  • ssl.truststore.password
If the listener is the inter-broker listener, the update is allowed only if the existing keystore for that listener is trusted by the new truststore. For other listeners, no trust validation is performed by the broker before the update. Removal of CA certificates used to sign client certificates from the new truststore can lead to client authentication failures.
Updating Default Topic Configuration
Default topic configuration options used by brokers may be updated without broker restart. The configs are applied to topics without a topic config override for the equivalent per-topic config. One or more of these configs may be overridden at cluster-default level used by all brokers.
  • log.segment.bytes
  • log.roll.ms
  • log.roll.hours
  • log.roll.jitter.ms
  • log.roll.jitter.hours
  • log.index.size.max.bytes
  • log.flush.interval.messages
  • log.flush.interval.ms
  • log.retention.bytes
  • log.retention.ms
  • log.retention.minutes
  • log.retention.hours
  • log.index.interval.bytes
  • log.cleaner.delete.retention.ms
  • log.cleaner.min.compaction.lag.ms
  • log.cleaner.max.compaction.lag.ms
  • log.cleaner.min.cleanable.ratio
  • log.cleanup.policy
  • log.segment.delete.delay.ms
  • unclean.leader.election.enable
  • min.insync.replicas
  • max.message.bytes
  • compression.type
  • log.preallocate
  • log.message.timestamp.type
  • log.message.timestamp.difference.max.ms
From Kafka version 2.0.0 onwards, unclean leader election is automatically enabled by the controller when the config unclean.leader.election.enable is dynamically updated. In Kafka version 1.1.x, changes to unclean.leader.election.enable take effect only when a new controller is elected. Controller re-election may be forced by running: 
$ bin/zookeeper-shell.sh localhost
  rmr /controller
Updating Log Cleaner Configs
Log cleaner configs may be updated dynamically at cluster-default level used by all brokers. The changes take effect on the next iteration of log cleaning. One or more of these configs may be updated:
  • log.cleaner.threads
  • log.cleaner.io.max.bytes.per.second
  • log.cleaner.dedupe.buffer.size
  • log.cleaner.io.buffer.size
  • log.cleaner.io.buffer.load.factor
  • log.cleaner.backoff.ms
Updating Thread Configs
The size of various thread pools used by the broker may be updated dynamically at cluster-default level used by all brokers. Updates are restricted to the range currentSize / 2 to currentSize * 2 to ensure that config updates are handled gracefully.
  • num.network.threads
  • num.io.threads
  • num.replica.fetchers
  • num.recovery.threads.per.data.dir
  • log.cleaner.threads
  • background.threads
Updating ConnectionQuota Configs
The maximum number of connections allowed for a given IP/host by the broker may be updated dynamically at cluster-default level used by all brokers. The changes will apply for new connection creations and the existing connections count will be taken into account by the new limits.
  • max.connections.per.ip
  • max.connections.per.ip.overrides
Adding and Removing Listeners

Listeners may be added or removed dynamically. When a new listener is added, security configs of the listener must be provided as listener configs with the listener prefix listener.name.{listenerName}.. If the new listener uses SASL, the JAAS configuration of the listener must be provided using the JAAS configuration property sasl.jaas.config with the listener and mechanism prefix. See JAAS configuration for Kafka brokers for details.

In Kafka version 1.1.x, the listener used by the inter-broker listener may not be updated dynamically. To update the inter-broker listener to a new listener, the new listener may be added on all brokers without restarting the broker. A rolling restart is then required to update inter.broker.listener.name.

In addition to all the security configs of new listeners, the following configs may be updated dynamically at per-broker level:
  • listeners
  • advertised.listeners
  • listener.security.protocol.map
Inter-broker listener must be configured using the static broker configuration inter.broker.listener.name or security.inter.broker.protocol.

3.2 Topic-Level Configs

Configurations pertinent to topics have both a server default as well an optional per-topic override. If no per-topic configuration is given the server default is used. The override can be set at topic creation time by giving one or more --config options. This example creates a topic named my-topic with a custom max message size and flush rate: 
$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 1 \
  --replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1
Overrides can also be changed or set later using the alter configs command. This example updates the max message size for my-topic: 
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic
  --alter --add-config max.message.bytes=128000
To check overrides set on the topic you can do 
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic --describe
To remove an override you can do 
$ bin/kafka-configs.sh --bootstrap-server localhost:9092  --entity-type topics --entity-name my-topic
  --alter --delete-config max.message.bytes
The following are the topic-level configurations. The server's default configuration for this property is given under the Server Default Property heading. A given server default config value only applies to a topic if it does not have an explicit topic config override.

  • cleanup.policy

    This config designates the retention policy to use on log segments. The "delete" policy (which is the default) will discard old segments when their retention time or size limit has been reached. The "compact" policy will enable log compaction, which retains the latest value for each key. It is also possible to specify both policies in a comma-separated list (e.g. "delete,compact"). In this case, old segments will be discarded per the retention time and size configuration, while retained segments will be compacted.

    Type:list
    Default:delete
    Valid Values:[compact, delete]
    Server Default Property:log.cleanup.policy
    Importance:medium

  • compression.gzip.level

    The compression level to use if compression.type is set to gzip.

    Type:int
    Default:-1
    Valid Values:[1,...,9] or -1
    Server Default Property:compression.gzip.level
    Importance:medium

  • compression.lz4.level

    The compression level to use if compression.type is set to lz4.

    Type:int
    Default:9
    Valid Values:[1,...,17]
    Server Default Property:compression.lz4.level
    Importance:medium

  • compression.type

    Specify the final compression type for a given topic. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'uncompressed' which is equivalent to no compression; and 'producer' which means retain the original compression codec set by the producer.

    Type:string
    Default:producer
    Valid Values:[uncompressed, zstd, lz4, snappy, gzip, producer]
    Server Default Property:compression.type
    Importance:medium

  • compression.zstd.level

    The compression level to use if compression.type is set to zstd.

    Type:int
    Default:3
    Valid Values:[-131072,...,22]
    Server Default Property:compression.zstd.level
    Importance:medium

  • delete.retention.ms

    The amount of time to retain delete tombstone markers for log compacted topics. This setting also gives a bound on the time in which a consumer must complete a read if they begin from offset 0 to ensure that they get a valid snapshot of the final stage (otherwise delete tombstones may be collected before they complete their scan).

    Type:long
    Default:86400000 (1 day)
    Valid Values:[0,...]
    Server Default Property:log.cleaner.delete.retention.ms
    Importance:medium

  • file.delete.delay.ms

    The time to wait before deleting a file from the filesystem

    Type:long
    Default:60000 (1 minute)
    Valid Values:[0,...]
    Server Default Property:log.segment.delete.delay.ms
    Importance:medium

  • flush.messages

    This setting allows specifying an interval at which we will force an fsync of data written to the log. For example if this was set to 1 we would fsync after every message; if it were 5 we would fsync after every five messages. In general we recommend you not set this and use replication for durability and allow the operating system's background flush capabilities as it is more efficient. This setting can be overridden on a per-topic basis (see the per-topic configuration section).

    Type:long
    Default:9223372036854775807
    Valid Values:[1,...]
    Server Default Property:log.flush.interval.messages
    Importance:medium

  • flush.ms

    This setting allows specifying a time interval at which we will force an fsync of data written to the log. For example if this was set to 1000 we would fsync after 1000 ms had passed. In general we recommend you not set this and use replication for durability and allow the operating system's background flush capabilities as it is more efficient.

    Type:long
    Default:9223372036854775807
    Valid Values:[0,...]
    Server Default Property:log.flush.interval.ms
    Importance:medium

  • follower.replication.throttled.replicas

    A list of replicas for which log replication should be throttled on the follower side. The list should describe a set of replicas in the form [PartitionId]:[BrokerId],[PartitionId]:[BrokerId]:... or alternatively the wildcard '*' can be used to throttle all replicas for this topic.

    Type:list
    Default:""
    Valid Values:[partitionId]:[brokerId],[partitionId]:[brokerId],...
    Server Default Property:null
    Importance:medium

  • index.interval.bytes

    This setting controls how frequently Kafka adds an index entry to its offset index. The default setting ensures that we index a message roughly every 4096 bytes. More indexing allows reads to jump closer to the exact position in the log but makes the index larger. You probably don't need to change this.

    Type:int
    Default:4096 (4 kibibytes)
    Valid Values:[0,...]
    Server Default Property:log.index.interval.bytes
    Importance:medium

  • leader.replication.throttled.replicas

    A list of replicas for which log replication should be throttled on the leader side. The list should describe a set of replicas in the form [PartitionId]:[BrokerId],[PartitionId]:[BrokerId]:... or alternatively the wildcard '*' can be used to throttle all replicas for this topic.

    Type:list
    Default:""
    Valid Values:[partitionId]:[brokerId],[partitionId]:[brokerId],...
    Server Default Property:null
    Importance:medium

  • local.retention.bytes

    The maximum size of local log segments that can grow for a partition before it deletes the old segments. Default value is -2, it represents `retention.bytes` value to be used. The effective value should always be less than or equal to `retention.bytes` value.

    Type:long
    Default:-2
    Valid Values:[-2,...]
    Server Default Property:log.local.retention.bytes
    Importance:medium

  • local.retention.ms

    The number of milliseconds to keep the local log segment before it gets deleted. Default value is -2, it represents `retention.ms` value is to be used. The effective value should always be less than or equal to `retention.ms` value.

    Type:long
    Default:-2
    Valid Values:[-2,...]
    Server Default Property:log.local.retention.ms
    Importance:medium

  • max.compaction.lag.ms

    The maximum time a message will remain ineligible for compaction in the log. Only applicable for logs that are being compacted.

    Type:long
    Default:9223372036854775807
    Valid Values:[1,...]
    Server Default Property:log.cleaner.max.compaction.lag.ms
    Importance:medium

  • max.message.bytes

    The largest record batch size allowed by Kafka (after compression if compression is enabled). If this is increased and there are consumers older than 0.10.2, the consumers' fetch size must also be increased so that they can fetch record batches this large. In the latest message format version, records are always grouped into batches for efficiency. In previous message format versions, uncompressed records are not grouped into batches and this limit only applies to a single record in that case.

    Type:int
    Default:1048588
    Valid Values:[0,...]
    Server Default Property:message.max.bytes
    Importance:medium

  • message.format.version

    [DEPRECATED] Specify the message format version the broker will use to append messages to the logs. The value of this config is always assumed to be `3.0` if `inter.broker.protocol.version` is 3.0 or higher (the actual config value is ignored). Otherwise, the value should be a valid ApiVersion. Some examples are: 0.10.0, 1.1, 2.8, 3.0. By setting a particular message format version, the user is certifying that all the existing messages on disk are smaller or equal than the specified version. Setting this value incorrectly will cause consumers with older versions to break as they will receive messages with a format that they don't understand.

    Type:string
    Default:3.0-IV1
    Valid Values:[0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.10.0-IV0, 0.10.0-IV1, 0.10.1-IV0, 0.10.1-IV1, 0.10.1-IV2, 0.10.2-IV0, 0.11.0-IV0, 0.11.0-IV1, 0.11.0-IV2, 1.0-IV0, 1.1-IV0, 2.0-IV0, 2.0-IV1, 2.1-IV0, 2.1-IV1, 2.1-IV2, 2.2-IV0, 2.2-IV1, 2.3-IV0, 2.3-IV1, 2.4-IV0, 2.4-IV1, 2.5-IV0, 2.6-IV0, 2.7-IV0, 2.7-IV1, 2.7-IV2, 2.8-IV0, 2.8-IV1, 3.0-IV0, 3.0-IV1, 3.1-IV0, 3.2-IV0, 3.3-IV0, 3.3-IV1, 3.3-IV2, 3.3-IV3, 3.4-IV0, 3.5-IV0, 3.5-IV1, 3.5-IV2, 3.6-IV0, 3.6-IV1, 3.6-IV2, 3.7-IV0, 3.7-IV1, 3.7-IV2, 3.7-IV3, 3.7-IV4, 3.8-IV0, 3.9-IV0, 4.0-IV0, 4.0-IV1]
    Server Default Property:log.message.format.version
    Importance:medium

  • message.timestamp.after.max.ms

    This configuration sets the allowable timestamp difference between the message timestamp and the broker's timestamp. The message timestamp can be later than or equal to the broker's timestamp, with the maximum allowable difference determined by the value set in this configuration. If message.timestamp.type=CreateTime, the message will be rejected if the difference in timestamps exceeds this specified threshold. This configuration is ignored if message.timestamp.type=LogAppendTime.

    Type:long
    Default:9223372036854775807
    Valid Values:[0,...]
    Server Default Property:log.message.timestamp.after.max.ms
    Importance:medium

  • message.timestamp.before.max.ms

    This configuration sets the allowable timestamp difference between the broker's timestamp and the message timestamp. The message timestamp can be earlier than or equal to the broker's timestamp, with the maximum allowable difference determined by the value set in this configuration. If message.timestamp.type=CreateTime, the message will be rejected if the difference in timestamps exceeds this specified threshold. This configuration is ignored if message.timestamp.type=LogAppendTime.

    Type:long
    Default:9223372036854775807
    Valid Values:[0,...]
    Server Default Property:log.message.timestamp.before.max.ms
    Importance:medium

  • message.timestamp.difference.max.ms

    [DEPRECATED] The maximum difference allowed between the timestamp when a broker receives a message and the timestamp specified in the message. If message.timestamp.type=CreateTime, a message will be rejected if the difference in timestamp exceeds this threshold. This configuration is ignored if message.timestamp.type=LogAppendTime.

    Type:long
    Default:9223372036854775807
    Valid Values:[0,...]
    Server Default Property:log.message.timestamp.difference.max.ms
    Importance:medium

  • message.timestamp.type

    Define whether the timestamp in the message is message create time or log append time. The value should be either `CreateTime` or `LogAppendTime`

    Type:string
    Default:CreateTime
    Valid Values:[CreateTime, LogAppendTime]
    Server Default Property:log.message.timestamp.type
    Importance:medium

  • min.cleanable.dirty.ratio

    This configuration controls how frequently the log compactor will attempt to clean the log (assuming log compaction is enabled). By default we will avoid cleaning a log where more than 50% of the log has been compacted. This ratio bounds the maximum space wasted in the log by duplicates (at 50% at most 50% of the log could be duplicates). A higher ratio will mean fewer, more efficient cleanings but will mean more wasted space in the log. If the max.compaction.lag.ms or the min.compaction.lag.ms configurations are also specified, then the log compactor considers the log to be eligible for compaction as soon as either: (i) the dirty ratio threshold has been met and the log has had dirty (uncompacted) records for at least the min.compaction.lag.ms duration, or (ii) if the log has had dirty (uncompacted) records for at most the max.compaction.lag.ms period.

    Type:double
    Default:0.5
    Valid Values:[0,...,1]
    Server Default Property:log.cleaner.min.cleanable.ratio
    Importance:medium

  • min.compaction.lag.ms

    The minimum time a message will remain uncompacted in the log. Only applicable for logs that are being compacted.

    Type:long
    Default:0
    Valid Values:[0,...]
    Server Default Property:log.cleaner.min.compaction.lag.ms
    Importance:medium

  • min.insync.replicas

    When a producer sets acks to "all" (or "-1"), this configuration specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this minimum cannot be met, then the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend).
    When used together, min.insync.replicas and acks allow you to enforce greater durability guarantees. A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all". This will ensure that the producer raises an exception if a majority of replicas do not receive a write.

    Type:int
    Default:1
    Valid Values:[1,...]
    Server Default Property:min.insync.replicas
    Importance:medium

  • preallocate

    True if we should preallocate the file on disk when creating a new log segment.

    Type:boolean
    Default:false
    Valid Values:
    Server Default Property:log.preallocate
    Importance:medium

  • remote.log.copy.disable

    Determines whether tiered data for a topic should become read only, and no more data uploading on a topic. Once this config is set to true, the local retention configuration (i.e. local.retention.ms/bytes) becomes irrelevant, and all data expiration follows the topic-wide retention configuration(i.e. retention.ms/bytes).

    Type:boolean
    Default:false
    Valid Values:
    Server Default Property:null
    Importance:medium

  • remote.log.delete.on.disable

    Determines whether tiered data for a topic should be deleted after tiered storage is disabled on a topic. This configuration should be enabled when trying to set `remote.storage.enable` from true to false

    Type:boolean
    Default:false
    Valid Values:
    Server Default Property:null
    Importance:medium

  • remote.storage.enable

    To enable tiered storage for a topic, set this configuration as true. You can not disable this config once it is enabled. It will be provided in future versions.

    Type:boolean
    Default:false
    Valid Values:
    Server Default Property:null
    Importance:medium

  • retention.bytes

    This configuration controls the maximum size a partition (which consists of log segments) can grow to before we will discard old log segments to free up space if we are using the "delete" retention policy. By default there is no size limit only a time limit. Since this limit is enforced at the partition level, multiply it by the number of partitions to compute the topic retention in bytes. Additionally, retention.bytes configuration operates independently of "segment.ms" and "segment.bytes" configurations. Moreover, it triggers the rolling of new segment if the retention.bytes is configured to zero.

    Type:long
    Default:-1
    Valid Values:
    Server Default Property:log.retention.bytes
    Importance:medium

  • retention.ms

    This configuration controls the maximum time we will retain a log before we will discard old log segments to free up space if we are using the "delete" retention policy. This represents an SLA on how soon consumers must read their data. If set to -1, no time limit is applied. Additionally, retention.ms configuration operates independently of "segment.ms" and "segment.bytes" configurations. Moreover, it triggers the rolling of new segment if the retention.ms condition is satisfied.

    Type:long
    Default:604800000 (7 days)
    Valid Values:[-1,...]
    Server Default Property:log.retention.ms
    Importance:medium

  • segment.bytes

    This configuration controls the segment file size for the log. Retention and cleaning is always done a file at a time so a larger segment size means fewer files but less granular control over retention.

    Type:int
    Default:1073741824 (1 gibibyte)
    Valid Values:[14,...]
    Server Default Property:log.segment.bytes
    Importance:medium

  • segment.index.bytes

    This configuration controls the size of the index that maps offsets to file positions. We preallocate this index file and shrink it only after log rolls. You generally should not need to change this setting.

    Type:int
    Default:10485760 (10 mebibytes)
    Valid Values:[4,...]
    Server Default Property:log.index.size.max.bytes
    Importance:medium

  • segment.jitter.ms

    The maximum random jitter subtracted from the scheduled segment roll time to avoid thundering herds of segment rolling

    Type:long
    Default:0
    Valid Values:[0,...]
    Server Default Property:log.roll.jitter.ms
    Importance:medium

  • segment.ms

    This configuration controls the period of time after which Kafka will force the log to roll even if the segment file isn't full to ensure that retention can delete or compact old data.

    Type:long
    Default:604800000 (7 days)
    Valid Values:[1,...]
    Server Default Property:log.roll.ms
    Importance:medium

  • unclean.leader.election.enable

    Indicates whether to enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss.

    Note: In KRaft mode, when enabling this config dynamically, it needs to wait for the unclean leader electionthread to trigger election periodically (default is 5 minutes). Please run `kafka-leader-election.sh` with `unclean` option to trigger the unclean leader election immediately if needed.

    Type:boolean
    Default:false
    Valid Values:
    Server Default Property:unclean.leader.election.enable
    Importance:medium

  • message.downconversion.enable

    This configuration controls whether down-conversion of message formats is enabled to satisfy consume requests. When set to false, broker will not perform down-conversion for consumers expecting an older message format. The broker responds with UNSUPPORTED_VERSION error for consume requests from such older clients. This configurationdoes not apply to any message format conversion that might be required for replication to followers.

    Type:boolean
    Default:true
    Valid Values:
    Server Default Property:log.message.downconversion.enable
    Importance:low

3.3 Producer Configs

Below is the configuration of the producer:

  • key.serializer

    Serializer class for key that implements the org.apache.kafka.common.serialization.Serializer interface.

    Type:class
    Default:
    Valid Values:
    Importance:high

  • value.serializer

    Serializer class for value that implements the org.apache.kafka.common.serialization.Serializer interface.

    Type:class
    Default:
    Valid Values:
    Importance:high

  • bootstrap.servers

    A list of host/port pairs used to establish the initial connection to the Kafka cluster. Clients use this list to bootstrap and discover the full set of Kafka brokers. While the order of servers in the list does not matter, we recommend including more than one server to ensure resilience if any servers are down. This list does not need to contain the entire set of brokers, as Kafka clients automatically manage and update connections to the cluster efficiently. This list must be in the form host1:port1,host2:port2,....

    Type:list
    Default:""
    Valid Values:non-null string
    Importance:high

  • buffer.memory

    The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block for max.block.ms after which it will throw an exception.

    This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests.

    Type:long
    Default:33554432
    Valid Values:[0,...]
    Importance:high

  • compression.type

    The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression).

    Type:string
    Default:none
    Valid Values:[none, gzip, snappy, lz4, zstd]
    Importance:high

  • retries

    Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires first before successful acknowledgement. Users should generally prefer to leave this config unset and instead use delivery.timeout.ms to control retry behavior.

    Enabling idempotence requires this config value to be greater than 0. If conflicting configurations are set and idempotence is not explicitly enabled, idempotence is disabled.

    Allowing retries while setting enable.idempotence to false and max.in.flight.requests.per.connection to greater than 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first.

    Type:int
    Default:2147483647
    Valid Values:[0,...,2147483647]
    Importance:high

  • ssl.key.password

    The password of the private key in the key store file or the PEM key specified in 'ssl.keystore.key'.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.certificate.chain

    Certificate chain in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with a list of X.509 certificates

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.key

    Private key in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with PKCS#8 keys. If the key is encrypted, key password must be specified using 'ssl.key.password'

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.location

    The location of the key store file. This is optional for client and can be used for two-way authentication for client.

    Type:string
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.password

    The store password for the key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured. Key store password is not supported for PEM format.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.certificates

    Trusted certificates in the format specified by 'ssl.truststore.type'. Default SSL engine factory supports only PEM format with X.509 certificates.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.location

    The location of the trust store file.

    Type:string
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.password

    The password for the trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled. Trust store password is not supported for PEM format.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • batch.size

    The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes.

    No attempt will be made to batch records larger than this size.

    Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent.

    A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records.

    Note: This setting gives the upper bound of the batch size to be sent. If we have fewer than this many bytes accumulated for this partition, we will 'linger' for the linger.ms time waiting for more records to show up. This linger.ms setting defaults to 0, which means we'll immediately send out a record even the accumulated batch size is under this batch.size setting.

    Type:int
    Default:16384
    Valid Values:[0,...]
    Importance:medium

  • client.dns.lookup

    Controls how the client uses DNS lookups. If set to use_all_dns_ips, connect to each returned IP address in sequence until a successful connection is established. After a disconnection, the next IP is used. Once all IPs have been used once, the client resolves the IP(s) from the hostname again (both the JVM and the OS cache DNS name lookups, however). If set to resolve_canonical_bootstrap_servers_only, resolve each bootstrap address into a list of canonical names. After the bootstrap phase, this behaves the same as use_all_dns_ips.

    Type:string
    Default:use_all_dns_ips
    Valid Values:[use_all_dns_ips, resolve_canonical_bootstrap_servers_only]
    Importance:medium

  • client.id

    An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.

    Type:string
    Default:""
    Valid Values:
    Importance:medium

  • compression.gzip.level

    The compression level to use if compression.type is set to gzip.

    Type:int
    Default:-1
    Valid Values:[1,...,9] or -1
    Importance:medium

  • compression.lz4.level

    The compression level to use if compression.type is set to lz4.

    Type:int
    Default:9
    Valid Values:[1,...,17]
    Importance:medium

  • compression.zstd.level

    The compression level to use if compression.type is set to zstd.

    Type:int
    Default:3
    Valid Values:[-131072,...,22]
    Importance:medium

  • connections.max.idle.ms

    Close idle connections after the number of milliseconds specified by this config.

    Type:long
    Default:540000 (9 minutes)
    Valid Values:
    Importance:medium

  • delivery.timeout.ms

    An upper bound on the time to report success or failure after a call to send() returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgement from the broker (if expected), and the time allowed for retriable send failures. The producer may report failure to send a record earlier than this config if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch which reached an earlier delivery expiration deadline. The value of this config should be greater than or equal to the sum of request.timeout.ms and linger.ms.

    Type:int
    Default:120000 (2 minutes)
    Valid Values:[0,...]
    Importance:medium

  • linger.ms

    The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay—that is, rather than immediately sending out a record, the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle's algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it will be sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting linger.ms=5, for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

    Type:long
    Default:0
    Valid Values:[0,...]
    Importance:medium

  • max.block.ms

    The configuration controls how long the KafkaProducer's send(), partitionsFor(), initTransactions(), sendOffsetsToTransaction(), commitTransaction() and abortTransaction() methods will block. For send() this timeout bounds the total time waiting for both metadata fetch and buffer allocation (blocking in the user-supplied serializers or partitioner is not counted against this timeout). For partitionsFor() this timeout bounds the time spent waiting for metadata if it is unavailable. The transaction-related methods always block, but may timeout if the transaction coordinator could not be discovered or did not respond within the timeout.

    Type:long
    Default:60000 (1 minute)
    Valid Values:[0,...]
    Importance:medium

  • max.request.size

    The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum uncompressed record batch size. Note that the server has its own cap on the record batch size (after compression if compression is enabled) which may be different from this.

    Type:int
    Default:1048576
    Valid Values:[0,...]
    Importance:medium

  • partitioner.class

    Determines which partition to send a record to when records are produced. Available options are:

    • If not set, the default partitioning logic is used. This strategy send records to a partition until at least batch.size bytes is produced to the partition. It works with the strategy:
      1. If no partition is specified but a key is present, choose a partition based on a hash of the key.
      2. If no partition or key is present, choose the sticky partition that changes when at least batch.size bytes are produced to the partition.
    • org.apache.kafka.clients.producer.RoundRobinPartitioner: A partitioning strategy where each record in a series of consecutive records is sent to a different partition, regardless of whether the 'key' is provided or not, until partitions run out and the process starts over again. Note: There's a known issue that will cause uneven distribution when a new batch is created. See KAFKA-9965 for more detail.

    Implementing the org.apache.kafka.clients.producer.Partitioner interface allows you to plug in a custom partitioner.

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • partitioner.ignore.keys

    When set to 'true' the producer won't use record keys to choose a partition. If 'false', producer would choose a partition based on a hash of the key when a key is present. Note: this setting has no effect if a custom partitioner is used.

    Type:boolean
    Default:false
    Valid Values:
    Importance:medium

  • receive.buffer.bytes

    The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.

    Type:int
    Default:32768 (32 kibibytes)
    Valid Values:[-1,...]
    Importance:medium

  • request.timeout.ms

    The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This should be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

    Type:int
    Default:30000 (30 seconds)
    Valid Values:[0,...]
    Importance:medium

  • sasl.client.callback.handler.class

    The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface.

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.jaas.config

    JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is: loginModuleClass controlFlag (optionName=optionValue)*;. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;

    Type:password
    Default:null
    Valid Values:
    Importance:medium

  • sasl.kerberos.service.name

    The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • sasl.login.callback.handler.class

    The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.login.class

    The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.mechanism

    SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.

    Type:string
    Default:GSSAPI
    Valid Values:
    Importance:medium

  • sasl.oauthbearer.jwks.endpoint.url

    The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. If the URL is HTTP(S)-based, the JWKS data will be retrieved from the OAuth/OIDC provider via the configured URL on broker startup. All then-current keys will be cached on the broker for incoming requests. If an authentication request is received for a JWT that includes a "kid" header claim value that isn't yet in the cache, the JWKS endpoint will be queried again on demand. However, the broker polls the URL every sasl.oauthbearer.jwks.endpoint.refresh.ms milliseconds to refresh the cache with any forthcoming keys before any JWT requests that include them are received. If the URL is file-based, the broker will load the JWKS file from a configured location on startup. In the event that the JWT includes a "kid" header value that isn't in the JWKS file, the broker will reject the JWT and authentication will fail.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • sasl.oauthbearer.token.endpoint.url

    The URL for the OAuth/OIDC identity provider. If the URL is HTTP(S)-based, it is the issuer's token endpoint URL to which requests will be made to login based on the configuration in sasl.jaas.config. If the URL is file-based, it specifies a file containing an access token (in JWT serialized form) issued by the OAuth/OIDC identity provider to use for authorization.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • security.protocol

    Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.

    Type:string
    Default:PLAINTEXT
    Valid Values:(case insensitive) [SASL_SSL, PLAINTEXT, SSL, SASL_PLAINTEXT]
    Importance:medium

  • send.buffer.bytes

    The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.

    Type:int
    Default:131072 (128 kibibytes)
    Valid Values:[-1,...]
    Importance:medium

  • socket.connection.setup.timeout.max.ms

    The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. To avoid connection storms, a randomization factor of 0.2 will be applied to the timeout resulting in a random range between 20% below and 20% above the computed value.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:
    Importance:medium

  • socket.connection.setup.timeout.ms

    The amount of time the client will wait for the socket connection to be established. If the connection is not built before the timeout elapses, clients will close the socket channel. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the socket.connection.setup.timeout.max.ms value.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:medium

  • ssl.enabled.protocols

    The list of protocols enabled for SSL connections. The default is 'TLSv1.2,TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). This default should be fine for most cases. Also see the config documentation for `ssl.protocol`.

    Type:list
    Default:TLSv1.2
    Valid Values:
    Importance:medium

  • ssl.keystore.type

    The file format of the key store file. This is optional for client. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].

    Type:string
    Default:JKS
    Valid Values:
    Importance:medium

  • ssl.protocol

    The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'.

    Type:string
    Default:TLSv1.2
    Valid Values:
    Importance:medium

  • ssl.provider

    The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • ssl.truststore.type

    The file format of the trust store file. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].

    Type:string
    Default:JKS
    Valid Values:
    Importance:medium

  • acks

    The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. The following settings are allowed:

    • acks=0 If set to zero then the producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retries configuration will not take effect (as the client won't generally know of any failures). The offset given back for each record will always be set to -1.
    • acks=1 This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers. In this case should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost.
    • acks=all This means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee. This is equivalent to the acks=-1 setting.

    Note that enabling idempotence requires this config value to be 'all'. If conflicting configurations are set and idempotence is not explicitly enabled, idempotence is disabled.

    Type:string
    Default:all
    Valid Values:[all, -1, 0, 1]
    Importance:low

  • auto.include.jmx.reporter

    Deprecated. Whether to automatically include JmxReporter even if it's not listed in metric.reporters. This configuration will be removed in Kafka 4.0, users should instead include org.apache.kafka.common.metrics.JmxReporter in metric.reporters in order to enable the JmxReporter.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • enable.idempotence

    When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc., may write duplicates of the retried message in the stream. Note that enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5 (with message ordering preserved for any allowable value), retries to be greater than 0, and acks must be 'all'.

    Idempotence is enabled by default if no conflicting configurations are set. If conflicting configurations are set and idempotence is not explicitly enabled, idempotence is disabled. If idempotence is explicitly enabled and conflicting configurations are set, a ConfigException is thrown.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • enable.metrics.push

    Whether to enable pushing of client metrics to the cluster, if the cluster has a client metrics subscription which matches this client.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • interceptor.classes

    A list of classes to use as interceptors. Implementing the org.apache.kafka.clients.producer.ProducerInterceptor interface allows you to intercept (and possibly mutate) the records received by the producer before they are published to the Kafka cluster. By default, there are no interceptors.

    Type:list
    Default:""
    Valid Values:non-null string
    Importance:low

  • max.in.flight.requests.per.connection

    The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this configuration is set to be greater than 1 and enable.idempotence is set to false, there is a risk of message reordering after a failed send due to retries (i.e., if retries are enabled); if retries are disabled or if enable.idempotence is set to true, ordering will be preserved. Additionally, enabling idempotence requires the value of this configuration to be less than or equal to 5. If conflicting configurations are set and idempotence is not explicitly enabled, idempotence is disabled.

    Type:int
    Default:5
    Valid Values:[1,...]
    Importance:low

  • metadata.max.age.ms

    The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.

    Type:long
    Default:300000 (5 minutes)
    Valid Values:[0,...]
    Importance:low

  • metadata.max.idle.ms

    Controls how long the producer will cache metadata for a topic that's idle. If the elapsed time since a topic was last produced to exceeds the metadata idle duration, then the topic's metadata is forgotten and the next access to it will force a metadata fetch request.

    Type:long
    Default:300000 (5 minutes)
    Valid Values:[5000,...]
    Importance:low

  • metadata.recovery.strategy

    Controls how the client recovers when none of the brokers known to it is available. If set to none, the client fails. If set to rebootstrap, the client repeats the bootstrap process using bootstrap.servers. Rebootstrapping is useful when a client communicates with brokers so infrequently that the set of brokers may change entirely before the client refreshes metadata. Metadata recovery is triggered when all last-known brokers appear unavailable simultaneously. Brokers appear unavailable when disconnected and no current retry attempt is in-progress. Consider increasing reconnect.backoff.ms and reconnect.backoff.max.ms and decreasing socket.connection.setup.timeout.ms and socket.connection.setup.timeout.max.ms for the client.

    Type:string
    Default:none
    Valid Values:(case insensitive) [REBOOTSTRAP, NONE]
    Importance:low

  • metric.reporters

    A list of classes to use as metrics reporters. Implementing the org.apache.kafka.common.metrics.MetricsReporter interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.

    Type:list
    Default:""
    Valid Values:non-null string
    Importance:low

  • metrics.num.samples

    The number of samples maintained to compute metrics.

    Type:int
    Default:2
    Valid Values:[1,...]
    Importance:low

  • metrics.recording.level

    The highest recording level for metrics.

    Type:string
    Default:INFO
    Valid Values:[INFO, DEBUG, TRACE]
    Importance:low

  • metrics.sample.window.ms

    The window of time a metrics sample is computed over.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:[0,...]
    Importance:low

  • partitioner.adaptive.partitioning.enable

    When set to 'true', the producer will try to adapt to broker performance and produce more messages to partitions hosted on faster brokers. If 'false', producer will try to distribute messages uniformly. Note: this setting has no effect if a custom partitioner is used

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • partitioner.availability.timeout.ms

    If a broker cannot process produce requests from a partition for partitioner.availability.timeout.ms time, the partitioner treats that partition as not available. If the value is 0, this logic is disabled. Note: this setting has no effect if a custom partitioner is used or partitioner.adaptive.partitioning.enable is set to 'false'

    Type:long
    Default:0
    Valid Values:[0,...]
    Importance:low

  • reconnect.backoff.max.ms

    The maximum amount of time in milliseconds to wait when reconnecting to a broker that has repeatedly failed to connect. If provided, the backoff per host will increase exponentially for each consecutive connection failure, up to this maximum. After calculating the backoff increase, 20% random jitter is added to avoid connection storms.

    Type:long
    Default:1000 (1 second)
    Valid Values:[0,...]
    Importance:low

  • reconnect.backoff.ms

    The base amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all connection attempts by the client to a broker. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the reconnect.backoff.max.ms value.

    Type:long
    Default:50
    Valid Values:[0,...]
    Importance:low

  • retry.backoff.max.ms

    The maximum amount of time in milliseconds to wait when retrying a request to the broker that has repeatedly failed. If provided, the backoff per client will increase exponentially for each failed request, up to this maximum. To prevent all clients from being synchronized upon retry, a randomized jitter with a factor of 0.2 will be applied to the backoff, resulting in the backoff falling within a range between 20% below and 20% above the computed value. If retry.backoff.ms is set to be higher than retry.backoff.max.ms, then retry.backoff.max.ms will be used as a constant backoff from the beginning without any exponential increase

    Type:long
    Default:1000 (1 second)
    Valid Values:[0,...]
    Importance:low

  • retry.backoff.ms

    The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. This value is the initial backoff value and will increase exponentially for each failed request, up to the retry.backoff.max.ms value.

    Type:long
    Default:100
    Valid Values:[0,...]
    Importance:low

  • sasl.kerberos.kinit.cmd

    Kerberos kinit command path.

    Type:string
    Default:/usr/bin/kinit
    Valid Values:
    Importance:low

  • sasl.kerberos.min.time.before.relogin

    Login thread sleep time between refresh attempts.

    Type:long
    Default:60000
    Valid Values:
    Importance:low

  • sasl.kerberos.ticket.renew.jitter

    Percentage of random jitter added to the renewal time.

    Type:double
    Default:0.05
    Valid Values:
    Importance:low

  • sasl.kerberos.ticket.renew.window.factor

    Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.

    Type:double
    Default:0.8
    Valid Values:
    Importance:low

  • sasl.login.connect.timeout.ms

    The (optional) value in milliseconds for the external authentication provider connection timeout. Currently applies only to OAUTHBEARER.

    Type:int
    Default:null
    Valid Values:
    Importance:low

  • sasl.login.read.timeout.ms

    The (optional) value in milliseconds for the external authentication provider read timeout. Currently applies only to OAUTHBEARER.

    Type:int
    Default:null
    Valid Values:
    Importance:low

  • sasl.login.refresh.buffer.seconds

    The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.

    Type:short
    Default:300
    Valid Values:[0,...,3600]
    Importance:low

  • sasl.login.refresh.min.period.seconds

    The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.

    Type:short
    Default:60
    Valid Values:[0,...,900]
    Importance:low

  • sasl.login.refresh.window.factor

    Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER.

    Type:double
    Default:0.8
    Valid Values:[0.5,...,1.0]
    Importance:low

  • sasl.login.refresh.window.jitter

    The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.

    Type:double
    Default:0.05
    Valid Values:[0.0,...,0.25]
    Importance:low

  • sasl.login.retry.backoff.max.ms

    The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:low

  • sasl.login.retry.backoff.ms

    The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.

    Type:long
    Default:100
    Valid Values:
    Importance:low

  • sasl.oauthbearer.clock.skew.seconds

    The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker.

    Type:int
    Default:30
    Valid Values:
    Importance:low

  • sasl.oauthbearer.expected.audience

    The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWT's "aud" claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.

    Type:list
    Default:null
    Valid Values:
    Importance:low

  • sasl.oauthbearer.expected.issuer

    The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth "iss" claim and if this value is set, the broker will match it exactly against what is in the JWT's "iss" claim. If there is no match, the broker will reject the JWT and authentication will fail.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • sasl.oauthbearer.header.urlencode

    The (optional) setting to enable the OAuth client to URL-encode the client_id and client_secret in the authorization header in accordance with RFC6749, see here for more details. The default value is set to 'false' for backward compatibility

    Type:boolean
    Default:false
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.refresh.ms

    The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.

    Type:long
    Default:3600000 (1 hour)
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms

    The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.retry.backoff.ms

    The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.

    Type:long
    Default:100
    Valid Values:
    Importance:low

  • sasl.oauthbearer.scope.claim.name

    The OAuth claim for the scope is often named "scope", but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

    Type:string
    Default:scope
    Valid Values:
    Importance:low

  • sasl.oauthbearer.sub.claim.name

    The OAuth claim for the subject is often named "sub", but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

    Type:string
    Default:sub
    Valid Values:
    Importance:low

  • security.providers

    A list of configurable creator classes each returning a provider implementing security algorithms. These classes should implement the org.apache.kafka.common.security.auth.SecurityProviderCreator interface.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • ssl.cipher.suites

    A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.

    Type:list
    Default:null
    Valid Values:
    Importance:low

  • ssl.endpoint.identification.algorithm

    The endpoint identification algorithm to validate server hostname using server certificate.

    Type:string
    Default:https
    Valid Values:
    Importance:low

  • ssl.engine.factory.class

    The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is org.apache.kafka.common.security.ssl.DefaultSslEngineFactory. Alternatively, setting this to org.apache.kafka.common.security.ssl.CommonNameLoggingSslEngineFactory will log the common name of expired SSL certificates used by clients to authenticate at any of the brokers with log level INFO. Note that this will cause a tiny delay during establishment of new connections from mTLS clients to brokers due to the extra code for examining the certificate chain provided by the client. Note further that the implementation uses a custom truststore based on the standard Java truststore and thus might be considered a security risk due to not being as mature as the standard one.

    Type:class
    Default:null
    Valid Values:
    Importance:low

  • ssl.keymanager.algorithm

    The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.

    Type:string
    Default:SunX509
    Valid Values:
    Importance:low

  • ssl.secure.random.implementation

    The SecureRandom PRNG implementation to use for SSL cryptography operations.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • ssl.trustmanager.algorithm

    The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.

    Type:string
    Default:PKIX
    Valid Values:
    Importance:low

  • transaction.timeout.ms

    The maximum amount of time in milliseconds that a transaction will remain open before the coordinator proactively aborts it. The start of the transaction is set at the time that the first partition is added to it. If this value is larger than the transaction.max.timeout.ms setting in the broker, the request will fail with a InvalidTxnTimeoutException error.

    Type:int
    Default:60000 (1 minute)
    Valid Values:
    Importance:low

  • transactional.id

    The TransactionalId to use for transactional delivery. This enables reliability semantics which span multiple producer sessions since it allows the client to guarantee that transactions using the same TransactionalId have been completed prior to starting any new transactions. If no TransactionalId is provided, then the producer is limited to idempotent delivery. If a TransactionalId is configured, enable.idempotence is implied. By default the TransactionId is not configured, which means transactions cannot be used. Note that, by default, transactions require a cluster of at least three brokers which is the recommended setting for production; for development you can change this, by adjusting broker setting transaction.state.log.replication.factor.

    Type:string
    Default:null
    Valid Values:non-empty string
    Importance:low

3.4 Consumer Configs

Below is the configuration for the consumer:

  • key.deserializer

    Deserializer class for key that implements the org.apache.kafka.common.serialization.Deserializer interface.

    Type:class
    Default:
    Valid Values:
    Importance:high

  • value.deserializer

    Deserializer class for value that implements the org.apache.kafka.common.serialization.Deserializer interface.

    Type:class
    Default:
    Valid Values:
    Importance:high

  • bootstrap.servers

    A list of host/port pairs used to establish the initial connection to the Kafka cluster. Clients use this list to bootstrap and discover the full set of Kafka brokers. While the order of servers in the list does not matter, we recommend including more than one server to ensure resilience if any servers are down. This list does not need to contain the entire set of brokers, as Kafka clients automatically manage and update connections to the cluster efficiently. This list must be in the form host1:port1,host2:port2,....

    Type:list
    Default:""
    Valid Values:non-null string
    Importance:high

  • fetch.min.bytes

    The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as that many byte(s) of data is available or the fetch request times out waiting for data to arrive. Setting this to a larger value will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency.

    Type:int
    Default:1
    Valid Values:[0,...]
    Importance:high

  • group.id

    A unique string that identifies the consumer group this consumer belongs to. This property is required if the consumer uses either the group management functionality by using subscribe(topic) or the Kafka-based offset management strategy.

    Type:string
    Default:null
    Valid Values:
    Importance:high

  • group.protocol

    The group protocol consumer should use. We currently support "classic" or "consumer". If "consumer" is specified, then the consumer group protocol will be used. Otherwise, the classic group protocol will be used.

    Type:string
    Default:classic
    Valid Values:(case insensitive) [CONSUMER, CLASSIC]
    Importance:high

  • heartbeat.interval.ms

    The expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

    Type:int
    Default:3000 (3 seconds)
    Valid Values:
    Importance:high

  • max.partition.fetch.bytes

    The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). See fetch.max.bytes for limiting the consumer request size.

    Type:int
    Default:1048576 (1 mebibyte)
    Valid Values:[0,...]
    Importance:high

  • session.timeout.ms

    The timeout used to detect client failures when using Kafka's group management facility. The client sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this client from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

    Type:int
    Default:45000 (45 seconds)
    Valid Values:
    Importance:high

  • ssl.key.password

    The password of the private key in the key store file or the PEM key specified in 'ssl.keystore.key'.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.certificate.chain

    Certificate chain in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with a list of X.509 certificates

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.key

    Private key in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with PKCS#8 keys. If the key is encrypted, key password must be specified using 'ssl.key.password'

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.location

    The location of the key store file. This is optional for client and can be used for two-way authentication for client.

    Type:string
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.password

    The store password for the key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured. Key store password is not supported for PEM format.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.certificates

    Trusted certificates in the format specified by 'ssl.truststore.type'. Default SSL engine factory supports only PEM format with X.509 certificates.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.location

    The location of the trust store file.

    Type:string
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.password

    The password for the trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled. Trust store password is not supported for PEM format.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • allow.auto.create.topics

    Allow automatic topic creation on the broker when subscribing to or assigning a topic. A topic being subscribed to will be automatically created only if the broker allows for it using `auto.create.topics.enable` broker configuration. This configuration must be set to `false` when using brokers older than 0.11.0

    Type:boolean
    Default:true
    Valid Values:
    Importance:medium

  • auto.offset.reset

    What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):

    • earliest: automatically reset the offset to the earliest offset
    • latest: automatically reset the offset to the latest offset
    • none: throw exception to the consumer if no previous offset is found for the consumer's group
    • anything else: throw exception to the consumer.

    Note that altering partition numbers while setting this config to latest may cause message delivery loss since producers could start to send messages to newly added partitions (i.e. no initial offsets exist yet) before consumers reset their offsets.

    Type:string
    Default:latest
    Valid Values:[latest, earliest, none]
    Importance:medium

  • client.dns.lookup

    Controls how the client uses DNS lookups. If set to use_all_dns_ips, connect to each returned IP address in sequence until a successful connection is established. After a disconnection, the next IP is used. Once all IPs have been used once, the client resolves the IP(s) from the hostname again (both the JVM and the OS cache DNS name lookups, however). If set to resolve_canonical_bootstrap_servers_only, resolve each bootstrap address into a list of canonical names. After the bootstrap phase, this behaves the same as use_all_dns_ips.

    Type:string
    Default:use_all_dns_ips
    Valid Values:[use_all_dns_ips, resolve_canonical_bootstrap_servers_only]
    Importance:medium

  • connections.max.idle.ms

    Close idle connections after the number of milliseconds specified by this config.

    Type:long
    Default:540000 (9 minutes)
    Valid Values:
    Importance:medium

  • default.api.timeout.ms

    Specifies the timeout (in milliseconds) for client APIs. This configuration is used as the default timeout for all client operations that do not specify a timeout parameter.

    Type:int
    Default:60000 (1 minute)
    Valid Values:[0,...]
    Importance:medium

  • enable.auto.commit

    If true the consumer's offset will be periodically committed in the background.

    Type:boolean
    Default:true
    Valid Values:
    Importance:medium

  • exclude.internal.topics

    Whether internal topics matching a subscribed pattern should be excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

    Type:boolean
    Default:true
    Valid Values:
    Importance:medium

  • fetch.max.bytes

    The maximum amount of data the server should return for a fetch request. Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not a absolute maximum. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). Note that the consumer performs multiple fetches in parallel.

    Type:int
    Default:52428800 (50 mebibytes)
    Valid Values:[0,...]
    Importance:medium

  • group.instance.id

    A unique identifier of the consumer instance provided by the end user. Only non-empty strings are permitted. If set, the consumer is treated as a static member, which means that only one instance with this ID is allowed in the consumer group at any time. This can be used in combination with a larger session timeout to avoid group rebalances caused by transient unavailability (e.g. process restarts). If not set, the consumer will join the group as a dynamic member, which is the traditional behavior.

    Type:string
    Default:null
    Valid Values:non-empty string
    Importance:medium

  • group.remote.assignor

    The server-side assignor to use. If no assignor is specified, the group coordinator will pick one. This configuration is applied only if group.protocol is set to "consumer".

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • isolation.level

    Controls how to read messages written transactionally. If set to read_committed, consumer.poll() will only return transactional messages which have been committed. If set to read_uncommitted (the default), consumer.poll() will return all messages, even transactional messages which have been aborted. Non-transactional messages will be returned unconditionally in either mode.

    Messages will always be returned in offset order. Hence, in read_committed mode, consumer.poll() will only return messages up to the last stable offset (LSO), which is the one less than the offset of the first open transaction. In particular any messages appearing after messages belonging to ongoing transactions will be withheld until the relevant transaction has been completed. As a result, read_committed consumers will not be able to read up to the high watermark when there are in flight transactions.

    Further, when in read_committed the seekToEnd method will return the LSO

    Type:string
    Default:read_uncommitted
    Valid Values:[read_committed, read_uncommitted]
    Importance:medium

  • max.poll.interval.ms

    The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member. For consumers using a non-null group.instance.id which reach this timeout, partitions will not be immediately reassigned. Instead, the consumer will stop sending heartbeats and partitions will be reassigned after expiration of session.timeout.ms. This mirrors the behavior of a static consumer which has shutdown.

    Type:int
    Default:300000 (5 minutes)
    Valid Values:[1,...]
    Importance:medium

  • max.poll.records

    The maximum number of records returned in a single call to poll(). Note, that max.poll.records does not impact the underlying fetching behavior. The consumer will cache the records from each fetch request and returns them incrementally from each poll.

    Type:int
    Default:500
    Valid Values:[1,...]
    Importance:medium

  • partition.assignment.strategy

    A list of class names or class types, ordered by preference, of supported partition assignment strategies that the client will use to distribute partition ownership amongst consumer instances when group management is used. Available options are:

    • org.apache.kafka.clients.consumer.RangeAssignor: Assigns partitions on a per-topic basis.
    • org.apache.kafka.clients.consumer.RoundRobinAssignor: Assigns partitions to consumers in a round-robin fashion.
    • org.apache.kafka.clients.consumer.StickyAssignor: Guarantees an assignment that is maximally balanced while preserving as many existing partition assignments as possible.
    • org.apache.kafka.clients.consumer.CooperativeStickyAssignor: Follows the same StickyAssignor logic, but allows for cooperative rebalancing.

    The default assignor is [RangeAssignor, CooperativeStickyAssignor], which will use the RangeAssignor by default, but allows upgrading to the CooperativeStickyAssignor with just a single rolling bounce that removes the RangeAssignor from the list.

    Implementing the org.apache.kafka.clients.consumer.ConsumerPartitionAssignor interface allows you to plug in a custom assignment strategy.

    Type:list
    Default:class org.apache.kafka.clients.consumer.RangeAssignor,class org.apache.kafka.clients.consumer.CooperativeStickyAssignor
    Valid Values:non-null string
    Importance:medium

  • receive.buffer.bytes

    The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.

    Type:int
    Default:65536 (64 kibibytes)
    Valid Values:[-1,...]
    Importance:medium

  • request.timeout.ms

    The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

    Type:int
    Default:30000 (30 seconds)
    Valid Values:[0,...]
    Importance:medium

  • sasl.client.callback.handler.class

    The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface.

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.jaas.config

    JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is: loginModuleClass controlFlag (optionName=optionValue)*;. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;

    Type:password
    Default:null
    Valid Values:
    Importance:medium

  • sasl.kerberos.service.name

    The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • sasl.login.callback.handler.class

    The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.login.class

    The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.mechanism

    SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.

    Type:string
    Default:GSSAPI
    Valid Values:
    Importance:medium

  • sasl.oauthbearer.jwks.endpoint.url

    The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. If the URL is HTTP(S)-based, the JWKS data will be retrieved from the OAuth/OIDC provider via the configured URL on broker startup. All then-current keys will be cached on the broker for incoming requests. If an authentication request is received for a JWT that includes a "kid" header claim value that isn't yet in the cache, the JWKS endpoint will be queried again on demand. However, the broker polls the URL every sasl.oauthbearer.jwks.endpoint.refresh.ms milliseconds to refresh the cache with any forthcoming keys before any JWT requests that include them are received. If the URL is file-based, the broker will load the JWKS file from a configured location on startup. In the event that the JWT includes a "kid" header value that isn't in the JWKS file, the broker will reject the JWT and authentication will fail.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • sasl.oauthbearer.token.endpoint.url

    The URL for the OAuth/OIDC identity provider. If the URL is HTTP(S)-based, it is the issuer's token endpoint URL to which requests will be made to login based on the configuration in sasl.jaas.config. If the URL is file-based, it specifies a file containing an access token (in JWT serialized form) issued by the OAuth/OIDC identity provider to use for authorization.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • security.protocol

    Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.

    Type:string
    Default:PLAINTEXT
    Valid Values:(case insensitive) [SASL_SSL, PLAINTEXT, SSL, SASL_PLAINTEXT]
    Importance:medium

  • send.buffer.bytes

    The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.

    Type:int
    Default:131072 (128 kibibytes)
    Valid Values:[-1,...]
    Importance:medium

  • socket.connection.setup.timeout.max.ms

    The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. To avoid connection storms, a randomization factor of 0.2 will be applied to the timeout resulting in a random range between 20% below and 20% above the computed value.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:
    Importance:medium

  • socket.connection.setup.timeout.ms

    The amount of time the client will wait for the socket connection to be established. If the connection is not built before the timeout elapses, clients will close the socket channel. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the socket.connection.setup.timeout.max.ms value.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:medium

  • ssl.enabled.protocols

    The list of protocols enabled for SSL connections. The default is 'TLSv1.2,TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). This default should be fine for most cases. Also see the config documentation for `ssl.protocol`.

    Type:list
    Default:TLSv1.2
    Valid Values:
    Importance:medium

  • ssl.keystore.type

    The file format of the key store file. This is optional for client. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].

    Type:string
    Default:JKS
    Valid Values:
    Importance:medium

  • ssl.protocol

    The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'.

    Type:string
    Default:TLSv1.2
    Valid Values:
    Importance:medium

  • ssl.provider

    The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • ssl.truststore.type

    The file format of the trust store file. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].

    Type:string
    Default:JKS
    Valid Values:
    Importance:medium

  • auto.commit.interval.ms

    The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if enable.auto.commit is set to true.

    Type:int
    Default:5000 (5 seconds)
    Valid Values:[0,...]
    Importance:low

  • auto.include.jmx.reporter

    Deprecated. Whether to automatically include JmxReporter even if it's not listed in metric.reporters. This configuration will be removed in Kafka 4.0, users should instead include org.apache.kafka.common.metrics.JmxReporter in metric.reporters in order to enable the JmxReporter.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • check.crcs

    Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • client.id

    An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.

    Type:string
    Default:""
    Valid Values:
    Importance:low

  • client.rack

    A rack identifier for this client. This can be any string value which indicates where this client is physically located. It corresponds with the broker config 'broker.rack'

    Type:string
    Default:""
    Valid Values:
    Importance:low

  • enable.metrics.push

    Whether to enable pushing of client metrics to the cluster, if the cluster has a client metrics subscription which matches this client.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • fetch.max.wait.ms

    The maximum amount of time the server will block before answering the fetch request there isn't sufficient data to immediately satisfy the requirement given by fetch.min.bytes. This config is used only for local log fetch. To tune the remote fetch maximum wait time, please refer to 'remote.fetch.max.wait.ms' broker config

    Type:int
    Default:500
    Valid Values:[0,...]
    Importance:low

  • interceptor.classes

    A list of classes to use as interceptors. Implementing the org.apache.kafka.clients.consumer.ConsumerInterceptor interface allows you to intercept (and possibly mutate) records received by the consumer. By default, there are no interceptors.

    Type:list
    Default:""
    Valid Values:non-null string
    Importance:low

  • metadata.max.age.ms

    The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.

    Type:long
    Default:300000 (5 minutes)
    Valid Values:[0,...]
    Importance:low

  • metadata.recovery.strategy

    Controls how the client recovers when none of the brokers known to it is available. If set to none, the client fails. If set to rebootstrap, the client repeats the bootstrap process using bootstrap.servers. Rebootstrapping is useful when a client communicates with brokers so infrequently that the set of brokers may change entirely before the client refreshes metadata. Metadata recovery is triggered when all last-known brokers appear unavailable simultaneously. Brokers appear unavailable when disconnected and no current retry attempt is in-progress. Consider increasing reconnect.backoff.ms and reconnect.backoff.max.ms and decreasing socket.connection.setup.timeout.ms and socket.connection.setup.timeout.max.ms for the client.

    Type:string
    Default:none
    Valid Values:(case insensitive) [REBOOTSTRAP, NONE]
    Importance:low

  • metric.reporters

    A list of classes to use as metrics reporters. Implementing the org.apache.kafka.common.metrics.MetricsReporter interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.

    Type:list
    Default:""
    Valid Values:non-null string
    Importance:low

  • metrics.num.samples

    The number of samples maintained to compute metrics.

    Type:int
    Default:2
    Valid Values:[1,...]
    Importance:low

  • metrics.recording.level

    The highest recording level for metrics.

    Type:string
    Default:INFO
    Valid Values:[INFO, DEBUG, TRACE]
    Importance:low

  • metrics.sample.window.ms

    The window of time a metrics sample is computed over.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:[0,...]
    Importance:low

  • reconnect.backoff.max.ms

    The maximum amount of time in milliseconds to wait when reconnecting to a broker that has repeatedly failed to connect. If provided, the backoff per host will increase exponentially for each consecutive connection failure, up to this maximum. After calculating the backoff increase, 20% random jitter is added to avoid connection storms.

    Type:long
    Default:1000 (1 second)
    Valid Values:[0,...]
    Importance:low

  • reconnect.backoff.ms

    The base amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all connection attempts by the client to a broker. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the reconnect.backoff.max.ms value.

    Type:long
    Default:50
    Valid Values:[0,...]
    Importance:low

  • retry.backoff.max.ms

    The maximum amount of time in milliseconds to wait when retrying a request to the broker that has repeatedly failed. If provided, the backoff per client will increase exponentially for each failed request, up to this maximum. To prevent all clients from being synchronized upon retry, a randomized jitter with a factor of 0.2 will be applied to the backoff, resulting in the backoff falling within a range between 20% below and 20% above the computed value. If retry.backoff.ms is set to be higher than retry.backoff.max.ms, then retry.backoff.max.ms will be used as a constant backoff from the beginning without any exponential increase

    Type:long
    Default:1000 (1 second)
    Valid Values:[0,...]
    Importance:low

  • retry.backoff.ms

    The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. This value is the initial backoff value and will increase exponentially for each failed request, up to the retry.backoff.max.ms value.

    Type:long
    Default:100
    Valid Values:[0,...]
    Importance:low

  • sasl.kerberos.kinit.cmd

    Kerberos kinit command path.

    Type:string
    Default:/usr/bin/kinit
    Valid Values:
    Importance:low

  • sasl.kerberos.min.time.before.relogin

    Login thread sleep time between refresh attempts.

    Type:long
    Default:60000
    Valid Values:
    Importance:low

  • sasl.kerberos.ticket.renew.jitter

    Percentage of random jitter added to the renewal time.

    Type:double
    Default:0.05
    Valid Values:
    Importance:low

  • sasl.kerberos.ticket.renew.window.factor

    Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.

    Type:double
    Default:0.8
    Valid Values:
    Importance:low

  • sasl.login.connect.timeout.ms

    The (optional) value in milliseconds for the external authentication provider connection timeout. Currently applies only to OAUTHBEARER.

    Type:int
    Default:null
    Valid Values:
    Importance:low

  • sasl.login.read.timeout.ms

    The (optional) value in milliseconds for the external authentication provider read timeout. Currently applies only to OAUTHBEARER.

    Type:int
    Default:null
    Valid Values:
    Importance:low

  • sasl.login.refresh.buffer.seconds

    The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.

    Type:short
    Default:300
    Valid Values:[0,...,3600]
    Importance:low

  • sasl.login.refresh.min.period.seconds

    The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.

    Type:short
    Default:60
    Valid Values:[0,...,900]
    Importance:low

  • sasl.login.refresh.window.factor

    Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER.

    Type:double
    Default:0.8
    Valid Values:[0.5,...,1.0]
    Importance:low

  • sasl.login.refresh.window.jitter

    The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.

    Type:double
    Default:0.05
    Valid Values:[0.0,...,0.25]
    Importance:low

  • sasl.login.retry.backoff.max.ms

    The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:low

  • sasl.login.retry.backoff.ms

    The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.

    Type:long
    Default:100
    Valid Values:
    Importance:low

  • sasl.oauthbearer.clock.skew.seconds

    The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker.

    Type:int
    Default:30
    Valid Values:
    Importance:low

  • sasl.oauthbearer.expected.audience

    The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWT's "aud" claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.

    Type:list
    Default:null
    Valid Values:
    Importance:low

  • sasl.oauthbearer.expected.issuer

    The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth "iss" claim and if this value is set, the broker will match it exactly against what is in the JWT's "iss" claim. If there is no match, the broker will reject the JWT and authentication will fail.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • sasl.oauthbearer.header.urlencode

    The (optional) setting to enable the OAuth client to URL-encode the client_id and client_secret in the authorization header in accordance with RFC6749, see here for more details. The default value is set to 'false' for backward compatibility

    Type:boolean
    Default:false
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.refresh.ms

    The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.

    Type:long
    Default:3600000 (1 hour)
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms

    The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.retry.backoff.ms

    The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.

    Type:long
    Default:100
    Valid Values:
    Importance:low

  • sasl.oauthbearer.scope.claim.name

    The OAuth claim for the scope is often named "scope", but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

    Type:string
    Default:scope
    Valid Values:
    Importance:low

  • sasl.oauthbearer.sub.claim.name

    The OAuth claim for the subject is often named "sub", but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

    Type:string
    Default:sub
    Valid Values:
    Importance:low

  • security.providers

    A list of configurable creator classes each returning a provider implementing security algorithms. These classes should implement the org.apache.kafka.common.security.auth.SecurityProviderCreator interface.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • ssl.cipher.suites

    A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.

    Type:list
    Default:null
    Valid Values:
    Importance:low

  • ssl.endpoint.identification.algorithm

    The endpoint identification algorithm to validate server hostname using server certificate.

    Type:string
    Default:https
    Valid Values:
    Importance:low

  • ssl.engine.factory.class

    The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is org.apache.kafka.common.security.ssl.DefaultSslEngineFactory. Alternatively, setting this to org.apache.kafka.common.security.ssl.CommonNameLoggingSslEngineFactory will log the common name of expired SSL certificates used by clients to authenticate at any of the brokers with log level INFO. Note that this will cause a tiny delay during establishment of new connections from mTLS clients to brokers due to the extra code for examining the certificate chain provided by the client. Note further that the implementation uses a custom truststore based on the standard Java truststore and thus might be considered a security risk due to not being as mature as the standard one.

    Type:class
    Default:null
    Valid Values:
    Importance:low

  • ssl.keymanager.algorithm

    The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.

    Type:string
    Default:SunX509
    Valid Values:
    Importance:low

  • ssl.secure.random.implementation

    The SecureRandom PRNG implementation to use for SSL cryptography operations.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • ssl.trustmanager.algorithm

    The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.

    Type:string
    Default:PKIX
    Valid Values:
    Importance:low

3.5 Kafka Connect Configs

Below is the configuration of the Kafka Connect framework.

  • config.storage.topic

    The name of the Kafka topic where connector configurations are stored

    Type:string
    Default:
    Valid Values:
    Importance:high

  • group.id

    A unique string that identifies the Connect cluster group this worker belongs to.

    Type:string
    Default:
    Valid Values:
    Importance:high

  • key.converter

    Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.

    Type:class
    Default:
    Valid Values:
    Importance:high

  • offset.storage.topic

    The name of the Kafka topic where source connector offsets are stored

    Type:string
    Default:
    Valid Values:
    Importance:high

  • status.storage.topic

    The name of the Kafka topic where connector and task status are stored

    Type:string
    Default:
    Valid Values:
    Importance:high

  • value.converter

    Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.

    Type:class
    Default:
    Valid Values:
    Importance:high

  • bootstrap.servers

    A list of host/port pairs used to establish the initial connection to the Kafka cluster. Clients use this list to bootstrap and discover the full set of Kafka brokers. While the order of servers in the list does not matter, we recommend including more than one server to ensure resilience if any servers are down. This list does not need to contain the entire set of brokers, as Kafka clients automatically manage and update connections to the cluster efficiently. This list must be in the form host1:port1,host2:port2,....

    Type:list
    Default:localhost:9092
    Valid Values:
    Importance:high

  • exactly.once.source.support

    Whether to enable exactly-once support for source connectors in the cluster by using transactions to write source records and their source offsets, and by proactively fencing out old task generations before bringing up new ones.
    To enable exactly-once source support on a new cluster, set this property to 'enabled'. To enable support on an existing cluster, first set to 'preparing' on every worker in the cluster, then set to 'enabled'. A rolling upgrade may be used for both changes. For more information on this feature, see the exactly-once source support documentation.

    Type:string
    Default:disabled
    Valid Values:(case insensitive) [DISABLED, ENABLED, PREPARING]
    Importance:high

  • heartbeat.interval.ms

    The expected time between heartbeats to the group coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the worker's session stays active and to facilitate rebalancing when new members join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

    Type:int
    Default:3000 (3 seconds)
    Valid Values:
    Importance:high

  • rebalance.timeout.ms

    The maximum allowed time for each worker to join the group once a rebalance has begun. This is basically a limit on the amount of time needed for all tasks to flush any pending data and commit offsets. If the timeout is exceeded, then the worker will be removed from the group, which will cause offset commit failures.

    Type:int
    Default:60000 (1 minute)
    Valid Values:
    Importance:high

  • session.timeout.ms

    The timeout used to detect worker failures. The worker sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove the worker from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

    Type:int
    Default:10000 (10 seconds)
    Valid Values:
    Importance:high

  • ssl.key.password

    The password of the private key in the key store file or the PEM key specified in 'ssl.keystore.key'.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.certificate.chain

    Certificate chain in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with a list of X.509 certificates

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.key

    Private key in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with PKCS#8 keys. If the key is encrypted, key password must be specified using 'ssl.key.password'

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.location

    The location of the key store file. This is optional for client and can be used for two-way authentication for client.

    Type:string
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.password

    The store password for the key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured. Key store password is not supported for PEM format.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.certificates

    Trusted certificates in the format specified by 'ssl.truststore.type'. Default SSL engine factory supports only PEM format with X.509 certificates.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.location

    The location of the trust store file.

    Type:string
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.password

    The password for the trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled. Trust store password is not supported for PEM format.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • client.dns.lookup

    Controls how the client uses DNS lookups. If set to use_all_dns_ips, connect to each returned IP address in sequence until a successful connection is established. After a disconnection, the next IP is used. Once all IPs have been used once, the client resolves the IP(s) from the hostname again (both the JVM and the OS cache DNS name lookups, however). If set to resolve_canonical_bootstrap_servers_only, resolve each bootstrap address into a list of canonical names. After the bootstrap phase, this behaves the same as use_all_dns_ips.

    Type:string
    Default:use_all_dns_ips
    Valid Values:[use_all_dns_ips, resolve_canonical_bootstrap_servers_only]
    Importance:medium

  • connections.max.idle.ms

    Close idle connections after the number of milliseconds specified by this config.

    Type:long
    Default:540000 (9 minutes)
    Valid Values:
    Importance:medium

  • connector.client.config.override.policy

    Class name or alias of implementation of ConnectorClientConfigOverridePolicy. Defines what client configurations can be overridden by the connector. The default implementation is `All`, meaning connector configurations can override all client properties. The other possible policies in the framework include `None` to disallow connectors from overriding client properties, and `Principal` to allow connectors to override only client principals.

    Type:string
    Default:All
    Valid Values:
    Importance:medium

  • receive.buffer.bytes

    The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.

    Type:int
    Default:32768 (32 kibibytes)
    Valid Values:[-1,...]
    Importance:medium

  • request.timeout.ms

    The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

    Type:int
    Default:40000 (40 seconds)
    Valid Values:[0,...]
    Importance:medium

  • sasl.client.callback.handler.class

    The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface.

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.jaas.config

    JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is: loginModuleClass controlFlag (optionName=optionValue)*;. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;

    Type:password
    Default:null
    Valid Values:
    Importance:medium

  • sasl.kerberos.service.name

    The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • sasl.login.callback.handler.class

    The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.login.class

    The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.mechanism

    SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.

    Type:string
    Default:GSSAPI
    Valid Values:
    Importance:medium

  • sasl.oauthbearer.jwks.endpoint.url

    The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. If the URL is HTTP(S)-based, the JWKS data will be retrieved from the OAuth/OIDC provider via the configured URL on broker startup. All then-current keys will be cached on the broker for incoming requests. If an authentication request is received for a JWT that includes a "kid" header claim value that isn't yet in the cache, the JWKS endpoint will be queried again on demand. However, the broker polls the URL every sasl.oauthbearer.jwks.endpoint.refresh.ms milliseconds to refresh the cache with any forthcoming keys before any JWT requests that include them are received. If the URL is file-based, the broker will load the JWKS file from a configured location on startup. In the event that the JWT includes a "kid" header value that isn't in the JWKS file, the broker will reject the JWT and authentication will fail.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • sasl.oauthbearer.token.endpoint.url

    The URL for the OAuth/OIDC identity provider. If the URL is HTTP(S)-based, it is the issuer's token endpoint URL to which requests will be made to login based on the configuration in sasl.jaas.config. If the URL is file-based, it specifies a file containing an access token (in JWT serialized form) issued by the OAuth/OIDC identity provider to use for authorization.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • security.protocol

    Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.

    Type:string
    Default:PLAINTEXT
    Valid Values:(case insensitive) [SASL_SSL, PLAINTEXT, SSL, SASL_PLAINTEXT]
    Importance:medium

  • send.buffer.bytes

    The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.

    Type:int
    Default:131072 (128 kibibytes)
    Valid Values:[-1,...]
    Importance:medium

  • ssl.enabled.protocols

    The list of protocols enabled for SSL connections. The default is 'TLSv1.2,TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). This default should be fine for most cases. Also see the config documentation for `ssl.protocol`.

    Type:list
    Default:TLSv1.2
    Valid Values:
    Importance:medium

  • ssl.keystore.type

    The file format of the key store file. This is optional for client. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].

    Type:string
    Default:JKS
    Valid Values:
    Importance:medium

  • ssl.protocol

    The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'.

    Type:string
    Default:TLSv1.2
    Valid Values:
    Importance:medium

  • ssl.provider

    The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • ssl.truststore.type

    The file format of the trust store file. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].

    Type:string
    Default:JKS
    Valid Values:
    Importance:medium

  • worker.sync.timeout.ms

    When the worker is out of sync with other workers and needs to resynchronize configurations, wait up to this amount of time before giving up, leaving the group, and waiting a backoff period before rejoining.

    Type:int
    Default:3000 (3 seconds)
    Valid Values:
    Importance:medium

  • worker.unsync.backoff.ms

    When the worker is out of sync with other workers and fails to catch up within worker.sync.timeout.ms, leave the Connect cluster for this long before rejoining.

    Type:int
    Default:300000 (5 minutes)
    Valid Values:
    Importance:medium

  • access.control.allow.methods

    Sets the methods supported for cross origin requests by setting the Access-Control-Allow-Methods header. The default value of the Access-Control-Allow-Methods header allows cross origin requests for GET, POST and HEAD.

    Type:string
    Default:""
    Valid Values:
    Importance:low

  • access.control.allow.origin

    Value to set the Access-Control-Allow-Origin header to for REST API requests.To enable cross origin access, set this to the domain of the application that should be permitted to access the API, or '*' to allow access from any domain. The default value only allows access from the domain of the REST API.

    Type:string
    Default:""
    Valid Values:
    Importance:low

  • admin.listeners

    List of comma-separated URIs the Admin REST API will listen on. The supported protocols are HTTP and HTTPS. An empty or blank string will disable this feature. The default behavior is to use the regular listener (specified by the 'listeners' property).

    Type:list
    Default:null
    Valid Values:List of comma-separated URLs, ex: http://localhost:8080,https://localhost:8443.
    Importance:low

  • auto.include.jmx.reporter

    Deprecated. Whether to automatically include JmxReporter even if it's not listed in metric.reporters. This configuration will be removed in Kafka 4.0, users should instead include org.apache.kafka.common.metrics.JmxReporter in metric.reporters in order to enable the JmxReporter.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • client.id

    An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.

    Type:string
    Default:""
    Valid Values:
    Importance:low

  • config.providers

    Comma-separated names of ConfigProvider classes, loaded and used in the order specified. Implementing the interface ConfigProvider allows you to replace variable references in connector configurations, such as for externalized secrets.

    Type:list
    Default:""
    Valid Values:
    Importance:low

  • config.storage.replication.factor

    Replication factor used when creating the configuration storage topic

    Type:short
    Default:3
    Valid Values:Positive number not larger than the number of brokers in the Kafka cluster, or -1 to use the broker's default
    Importance:low

  • connect.protocol

    Compatibility mode for Kafka Connect Protocol

    Type:string
    Default:sessioned
    Valid Values:[eager, compatible, sessioned]
    Importance:low

  • header.converter

    HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas.

    Type:class
    Default:org.apache.kafka.connect.storage.SimpleHeaderConverter
    Valid Values:
    Importance:low

  • inter.worker.key.generation.algorithm

    The algorithm to use for generating internal request keys. The algorithm 'HmacSHA256' will be used as a default on JVMs that support it; on other JVMs, no default is used and a value for this property must be manually specified in the worker config.

    Type:string
    Default:HmacSHA256
    Valid Values:Any KeyGenerator algorithm supported by the worker JVM
    Importance:low

  • inter.worker.key.size

    The size of the key to use for signing internal requests, in bits. If null, the default key size for the key generation algorithm will be used.

    Type:int
    Default:null
    Valid Values:
    Importance:low

  • inter.worker.key.ttl.ms

    The TTL of generated session keys used for internal request validation (in milliseconds)

    Type:int
    Default:3600000 (1 hour)
    Valid Values:[0,...,2147483647]
    Importance:low

  • inter.worker.signature.algorithm

    The algorithm used to sign internal requestsThe algorithm 'inter.worker.signature.algorithm' will be used as a default on JVMs that support it; on other JVMs, no default is used and a value for this property must be manually specified in the worker config.

    Type:string
    Default:HmacSHA256
    Valid Values:Any MAC algorithm supported by the worker JVM
    Importance:low

  • inter.worker.verification.algorithms

    A list of permitted algorithms for verifying internal requests, which must include the algorithm used for the inter.worker.signature.algorithm property. The algorithm(s) '[HmacSHA256]' will be used as a default on JVMs that provide them; on other JVMs, no default is used and a value for this property must be manually specified in the worker config.

    Type:list
    Default:HmacSHA256
    Valid Values:A list of one or more MAC algorithms, each supported by the worker JVM
    Importance:low

  • listeners

    List of comma-separated URIs the REST API will listen on. The supported protocols are HTTP and HTTPS.
    Specify hostname as 0.0.0.0 to bind to all interfaces.
    Leave hostname empty to bind to default interface.
    Examples of legal listener lists: HTTP://myhost:8083,HTTPS://myhost:8084

    Type:list
    Default:http://:8083
    Valid Values:List of comma-separated URLs, ex: http://localhost:8080,https://localhost:8443.
    Importance:low

  • metadata.max.age.ms

    The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.

    Type:long
    Default:300000 (5 minutes)
    Valid Values:[0,...]
    Importance:low

  • metadata.recovery.strategy

    Controls how the client recovers when none of the brokers known to it is available. If set to none, the client fails. If set to rebootstrap, the client repeats the bootstrap process using bootstrap.servers. Rebootstrapping is useful when a client communicates with brokers so infrequently that the set of brokers may change entirely before the client refreshes metadata. Metadata recovery is triggered when all last-known brokers appear unavailable simultaneously. Brokers appear unavailable when disconnected and no current retry attempt is in-progress. Consider increasing reconnect.backoff.ms and reconnect.backoff.max.ms and decreasing socket.connection.setup.timeout.ms and socket.connection.setup.timeout.max.ms for the client.

    Type:string
    Default:none
    Valid Values:(case insensitive) [REBOOTSTRAP, NONE]
    Importance:low

  • metric.reporters

    A list of classes to use as metrics reporters. Implementing the org.apache.kafka.common.metrics.MetricsReporter interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.

    Type:list
    Default:""
    Valid Values:
    Importance:low

  • metrics.num.samples

    The number of samples maintained to compute metrics.

    Type:int
    Default:2
    Valid Values:[1,...]
    Importance:low

  • metrics.recording.level

    The highest recording level for metrics.

    Type:string
    Default:INFO
    Valid Values:[INFO, DEBUG]
    Importance:low

  • metrics.sample.window.ms

    The window of time a metrics sample is computed over.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:[0,...]
    Importance:low

  • offset.flush.interval.ms

    Interval at which to try committing offsets for tasks.

    Type:long
    Default:60000 (1 minute)
    Valid Values:
    Importance:low

  • offset.flush.timeout.ms

    Maximum number of milliseconds to wait for records to flush and partition offset data to be committed to offset storage before cancelling the process and restoring the offset data to be committed in a future attempt. This property has no effect for source connectors running with exactly-once support.

    Type:long
    Default:5000 (5 seconds)
    Valid Values:
    Importance:low

  • offset.storage.partitions

    The number of partitions used when creating the offset storage topic

    Type:int
    Default:25
    Valid Values:Positive number, or -1 to use the broker's default
    Importance:low

  • offset.storage.replication.factor

    Replication factor used when creating the offset storage topic

    Type:short
    Default:3
    Valid Values:Positive number not larger than the number of brokers in the Kafka cluster, or -1 to use the broker's default
    Importance:low

  • plugin.discovery

    Method to use to discover plugins present in the classpath and plugin.path configuration. This can be one of multiple values with the following meanings:
    * only_scan: Discover plugins only by reflection. Plugins which are not discoverable by ServiceLoader will not impact worker startup.
    * hybrid_warn: Discover plugins reflectively and by ServiceLoader. Plugins which are not discoverable by ServiceLoader will print warnings during worker startup.
    * hybrid_fail: Discover plugins reflectively and by ServiceLoader. Plugins which are not discoverable by ServiceLoader will cause worker startup to fail.
    * service_load: Discover plugins only by ServiceLoader. Faster startup than other modes. Plugins which are not discoverable by ServiceLoader may not be usable.

    Type:string
    Default:hybrid_warn
    Valid Values:(case insensitive) [ONLY_SCAN, SERVICE_LOAD, HYBRID_WARN, HYBRID_FAIL]
    Importance:low

  • plugin.path

    List of paths separated by commas (,) that contain plugins (connectors, converters, transformations). The list should consist of top level directories that include any combination of:
    a) directories immediately containing jars with plugins and their dependencies
    b) uber-jars with plugins and their dependencies
    c) directories immediately containing the package directory structure of classes of plugins and their dependencies
    Note: symlinks will be followed to discover dependencies or plugins.
    Examples: plugin.path=/usr/local/share/java,/usr/local/share/kafka/plugins,/opt/connectors
    Do not use config provider variables in this property, since the raw path is used by the worker's scanner before config providers are initialized and used to replace variables.

    Type:list
    Default:null
    Valid Values:
    Importance:low

  • reconnect.backoff.max.ms

    The maximum amount of time in milliseconds to wait when reconnecting to a broker that has repeatedly failed to connect. If provided, the backoff per host will increase exponentially for each consecutive connection failure, up to this maximum. After calculating the backoff increase, 20% random jitter is added to avoid connection storms.

    Type:long
    Default:1000 (1 second)
    Valid Values:[0,...]
    Importance:low

  • reconnect.backoff.ms

    The base amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all connection attempts by the client to a broker. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the reconnect.backoff.max.ms value.

    Type:long
    Default:50
    Valid Values:[0,...]
    Importance:low

  • response.http.headers.config

    Rules for REST API HTTP response headers

    Type:string
    Default:""
    Valid Values:Comma-separated header rules, where each header rule is of the form '[action] [header name]:[header value]' and optionally surrounded by double quotes if any part of a header rule contains a comma
    Importance:low

  • rest.advertised.host.name

    If this is set, this is the hostname that will be given out to other workers to connect to.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • rest.advertised.listener

    Sets the advertised listener (HTTP or HTTPS) which will be given to other workers to use.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • rest.advertised.port

    If this is set, this is the port that will be given out to other workers to connect to.

    Type:int
    Default:null
    Valid Values:
    Importance:low

  • rest.extension.classes

    Comma-separated names of ConnectRestExtension classes, loaded and called in the order specified. Implementing the interface ConnectRestExtension allows you to inject into Connect's REST API user defined resources like filters. Typically used to add custom capability like logging, security, etc.

    Type:list
    Default:""
    Valid Values:
    Importance:low

  • retry.backoff.max.ms

    The maximum amount of time in milliseconds to wait when retrying a request to the broker that has repeatedly failed. If provided, the backoff per client will increase exponentially for each failed request, up to this maximum. To prevent all clients from being synchronized upon retry, a randomized jitter with a factor of 0.2 will be applied to the backoff, resulting in the backoff falling within a range between 20% below and 20% above the computed value. If retry.backoff.ms is set to be higher than retry.backoff.max.ms, then retry.backoff.max.ms will be used as a constant backoff from the beginning without any exponential increase

    Type:long
    Default:1000 (1 second)
    Valid Values:[0,...]
    Importance:low

  • retry.backoff.ms

    The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. This value is the initial backoff value and will increase exponentially for each failed request, up to the retry.backoff.max.ms value.

    Type:long
    Default:100
    Valid Values:[0,...]
    Importance:low

  • sasl.kerberos.kinit.cmd

    Kerberos kinit command path.

    Type:string
    Default:/usr/bin/kinit
    Valid Values:
    Importance:low

  • sasl.kerberos.min.time.before.relogin

    Login thread sleep time between refresh attempts.

    Type:long
    Default:60000
    Valid Values:
    Importance:low

  • sasl.kerberos.ticket.renew.jitter

    Percentage of random jitter added to the renewal time.

    Type:double
    Default:0.05
    Valid Values:
    Importance:low

  • sasl.kerberos.ticket.renew.window.factor

    Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.

    Type:double
    Default:0.8
    Valid Values:
    Importance:low

  • sasl.login.connect.timeout.ms

    The (optional) value in milliseconds for the external authentication provider connection timeout. Currently applies only to OAUTHBEARER.

    Type:int
    Default:null
    Valid Values:
    Importance:low

  • sasl.login.read.timeout.ms

    The (optional) value in milliseconds for the external authentication provider read timeout. Currently applies only to OAUTHBEARER.

    Type:int
    Default:null
    Valid Values:
    Importance:low

  • sasl.login.refresh.buffer.seconds

    The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.

    Type:short
    Default:300
    Valid Values:[0,...,3600]
    Importance:low

  • sasl.login.refresh.min.period.seconds

    The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.

    Type:short
    Default:60
    Valid Values:[0,...,900]
    Importance:low

  • sasl.login.refresh.window.factor

    Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER.

    Type:double
    Default:0.8
    Valid Values:[0.5,...,1.0]
    Importance:low

  • sasl.login.refresh.window.jitter

    The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.

    Type:double
    Default:0.05
    Valid Values:[0.0,...,0.25]
    Importance:low

  • sasl.login.retry.backoff.max.ms

    The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:low

  • sasl.login.retry.backoff.ms

    The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.

    Type:long
    Default:100
    Valid Values:
    Importance:low

  • sasl.oauthbearer.clock.skew.seconds

    The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker.

    Type:int
    Default:30
    Valid Values:
    Importance:low

  • sasl.oauthbearer.expected.audience

    The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWT's "aud" claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.

    Type:list
    Default:null
    Valid Values:
    Importance:low

  • sasl.oauthbearer.expected.issuer

    The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth "iss" claim and if this value is set, the broker will match it exactly against what is in the JWT's "iss" claim. If there is no match, the broker will reject the JWT and authentication will fail.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • sasl.oauthbearer.header.urlencode

    The (optional) setting to enable the OAuth client to URL-encode the client_id and client_secret in the authorization header in accordance with RFC6749, see here for more details. The default value is set to 'false' for backward compatibility

    Type:boolean
    Default:false
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.refresh.ms

    The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.

    Type:long
    Default:3600000 (1 hour)
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms

    The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.retry.backoff.ms

    The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.

    Type:long
    Default:100
    Valid Values:
    Importance:low

  • sasl.oauthbearer.scope.claim.name

    The OAuth claim for the scope is often named "scope", but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

    Type:string
    Default:scope
    Valid Values:
    Importance:low

  • sasl.oauthbearer.sub.claim.name

    The OAuth claim for the subject is often named "sub", but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

    Type:string
    Default:sub
    Valid Values:
    Importance:low

  • scheduled.rebalance.max.delay.ms

    The maximum delay that is scheduled in order to wait for the return of one or more departed workers before rebalancing and reassigning their connectors and tasks to the group. During this period the connectors and tasks of the departed workers remain unassigned

    Type:int
    Default:300000 (5 minutes)
    Valid Values:[0,...,2147483647]
    Importance:low

  • socket.connection.setup.timeout.max.ms

    The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. To avoid connection storms, a randomization factor of 0.2 will be applied to the timeout resulting in a random range between 20% below and 20% above the computed value.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:[0,...]
    Importance:low

  • socket.connection.setup.timeout.ms

    The amount of time the client will wait for the socket connection to be established. If the connection is not built before the timeout elapses, clients will close the socket channel. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the socket.connection.setup.timeout.max.ms value.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:[0,...]
    Importance:low

  • ssl.cipher.suites

    A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.

    Type:list
    Default:null
    Valid Values:
    Importance:low

  • ssl.client.auth

    Configures kafka broker to request client authentication. The following settings are common:

    • ssl.client.auth=required If set to required client authentication is required.
    • ssl.client.auth=requested This means client authentication is optional. unlike required, if this option is set client can choose not to provide authentication information about itself
    • ssl.client.auth=none This means client authentication is not needed.

    Type:string
    Default:none
    Valid Values:[required, requested, none]
    Importance:low

  • ssl.endpoint.identification.algorithm

    The endpoint identification algorithm to validate server hostname using server certificate.

    Type:string
    Default:https
    Valid Values:
    Importance:low

  • ssl.engine.factory.class

    The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is org.apache.kafka.common.security.ssl.DefaultSslEngineFactory. Alternatively, setting this to org.apache.kafka.common.security.ssl.CommonNameLoggingSslEngineFactory will log the common name of expired SSL certificates used by clients to authenticate at any of the brokers with log level INFO. Note that this will cause a tiny delay during establishment of new connections from mTLS clients to brokers due to the extra code for examining the certificate chain provided by the client. Note further that the implementation uses a custom truststore based on the standard Java truststore and thus might be considered a security risk due to not being as mature as the standard one.

    Type:class
    Default:null
    Valid Values:
    Importance:low

  • ssl.keymanager.algorithm

    The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.

    Type:string
    Default:SunX509
    Valid Values:
    Importance:low

  • ssl.secure.random.implementation

    The SecureRandom PRNG implementation to use for SSL cryptography operations.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • ssl.trustmanager.algorithm

    The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.

    Type:string
    Default:PKIX
    Valid Values:
    Importance:low

  • status.storage.partitions

    The number of partitions used when creating the status storage topic

    Type:int
    Default:5
    Valid Values:Positive number, or -1 to use the broker's default
    Importance:low

  • status.storage.replication.factor

    Replication factor used when creating the status storage topic

    Type:short
    Default:3
    Valid Values:Positive number not larger than the number of brokers in the Kafka cluster, or -1 to use the broker's default
    Importance:low

  • task.shutdown.graceful.timeout.ms

    Amount of time to wait for tasks to shutdown gracefully. This is the total amount of time, not per task. All task have shutdown triggered, then they are waited on sequentially.

    Type:long
    Default:5000 (5 seconds)
    Valid Values:
    Importance:low

  • topic.creation.enable

    Whether to allow automatic creation of topics used by source connectors, when source connectors are configured with `topic.creation.` properties. Each task will use an admin client to create its topics and will not depend on the Kafka brokers to create topics automatically.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • topic.tracking.allow.reset

    If set to true, it allows user requests to reset the set of active topics per connector.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • topic.tracking.enable

    Enable tracking the set of active topics per connector during runtime.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

3.5.1 Source Connector Configs

Below is the configuration of a source connector.

  • name

    Globally unique name to use for this connector.

    Type:string
    Default:
    Valid Values:non-empty string without ISO control characters
    Importance:high

  • connector.class

    Name or alias of the class for this connector. Must be a subclass of org.apache.kafka.connect.connector.Connector. If the connector is org.apache.kafka.connect.file.FileStreamSinkConnector, you can either specify this full name, or use "FileStreamSink" or "FileStreamSinkConnector" to make the configuration a bit shorter

    Type:string
    Default:
    Valid Values:
    Importance:high

  • tasks.max

    Maximum number of tasks to use for this connector.

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:high

  • tasks.max.enforce

    (Deprecated) Whether to enforce that the tasks.max property is respected by the connector. By default, connectors that generate too many tasks will fail, and existing sets of tasks that exceed the tasks.max property will also be failed. If this property is set to false, then connectors will be allowed to generate more than the maximum number of tasks, and existing sets of tasks that exceed the tasks.max property will be allowed to run. This property is deprecated and will be removed in an upcoming major release.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • key.converter

    Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.

    Type:class
    Default:null
    Valid Values:A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor
    Importance:low

  • value.converter

    Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.

    Type:class
    Default:null
    Valid Values:A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor
    Importance:low

  • header.converter

    HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas.

    Type:class
    Default:null
    Valid Values:A concrete subclass of org.apache.kafka.connect.storage.HeaderConverter, A class with a public, no-argument constructor
    Importance:low

  • config.action.reload

    The action that Connect should take on the connector when changes in external configuration providers result in a change in the connector's configuration properties. A value of 'none' indicates that Connect will do nothing. A value of 'restart' indicates that Connect should restart/reload the connector with the updated configuration properties.The restart may actually be scheduled in the future if the external configuration provider indicates that a configuration value will expire in the future.

    Type:string
    Default:restart
    Valid Values:[none, restart]
    Importance:low

  • transforms

    Aliases for the transformations to be applied to records.

    Type:list
    Default:""
    Valid Values:non-null string, unique transformation aliases
    Importance:low

  • predicates

    Aliases for the predicates used by transformations.

    Type:list
    Default:""
    Valid Values:non-null string, unique predicate aliases
    Importance:low

  • errors.retry.timeout

    The maximum duration in milliseconds that a failed operation will be reattempted. The default is 0, which means no retries will be attempted. Use -1 for infinite retries.

    Type:long
    Default:0
    Valid Values:
    Importance:medium

  • errors.retry.delay.max.ms

    The maximum duration in milliseconds between consecutive retry attempts. Jitter will be added to the delay once this limit is reached to prevent thundering herd issues.

    Type:long
    Default:60000 (1 minute)
    Valid Values:
    Importance:medium

  • errors.tolerance

    Behavior for tolerating errors during connector operation. 'none' is the default value and signals that any error will result in an immediate connector task failure; 'all' changes the behavior to skip over problematic records.

    Type:string
    Default:none
    Valid Values:[none, all]
    Importance:medium

  • errors.log.enable

    If true, write each error and the details of the failed operation and problematic record to the Connect application log. This is 'false' by default, so that only errors that are not tolerated are reported.

    Type:boolean
    Default:false
    Valid Values:
    Importance:medium

  • errors.log.include.messages

    Whether to include in the log the Connect record that resulted in a failure. For sink records, the topic, partition, offset, and timestamp will be logged. For source records, the key and value (and their schemas), all headers, and the timestamp, Kafka topic, Kafka partition, source partition, and source offset will be logged. This is 'false' by default, which will prevent record keys, values, and headers from being written to log files.

    Type:boolean
    Default:false
    Valid Values:
    Importance:medium

  • topic.creation.groups

    Groups of configurations for topics created by source connectors

    Type:list
    Default:""
    Valid Values:non-null string, unique topic creation groups
    Importance:low

  • exactly.once.support

    Permitted values are requested, required. If set to "required", forces a preflight check for the connector to ensure that it can provide exactly-once semantics with the given configuration. Some connectors may be capable of providing exactly-once semantics but not signal to Connect that they support this; in that case, documentation for the connector should be consulted carefully before creating it, and the value for this property should be set to "requested". Additionally, if the value is set to "required" but the worker that performs preflight validation does not have exactly-once support enabled for source connectors, requests to create or validate the connector will fail.

    Type:string
    Default:requested
    Valid Values:(case insensitive) [REQUIRED, REQUESTED]
    Importance:medium

  • transaction.boundary

    Permitted values are: poll, interval, connector. If set to 'poll', a new producer transaction will be started and committed for every batch of records that each task from this connector provides to Connect. If set to 'connector', relies on connector-defined transaction boundaries; note that not all connectors are capable of defining their own transaction boundaries, and in that case, attempts to instantiate a connector with this value will fail. Finally, if set to 'interval', commits transactions only after a user-defined time interval has passed.

    Type:string
    Default:poll
    Valid Values:(case insensitive) [INTERVAL, POLL, CONNECTOR]
    Importance:medium

  • transaction.boundary.interval.ms

    If 'transaction.boundary' is set to 'interval', determines the interval for producer transaction commits by connector tasks. If unset, defaults to the value of the worker-level 'offset.flush.interval.ms' property. It has no effect if a different transaction.boundary is specified.

    Type:long
    Default:null
    Valid Values:[0,...]
    Importance:low

  • offsets.storage.topic

    The name of a separate offsets topic to use for this connector. If empty or not specified, the worker’s global offsets topic name will be used. If specified, the offsets topic will be created if it does not already exist on the Kafka cluster targeted by this connector (which may be different from the one used for the worker's global offsets topic if the bootstrap.servers property of the connector's producer has been overridden from the worker's). Only applicable in distributed mode; in standalone mode, setting this property will have no effect.

    Type:string
    Default:null
    Valid Values:non-empty string
    Importance:low

3.5.2 Sink Connector Configs

Below is the configuration of a sink connector.

  • name

    Globally unique name to use for this connector.

    Type:string
    Default:
    Valid Values:non-empty string without ISO control characters
    Importance:high

  • connector.class

    Name or alias of the class for this connector. Must be a subclass of org.apache.kafka.connect.connector.Connector. If the connector is org.apache.kafka.connect.file.FileStreamSinkConnector, you can either specify this full name, or use "FileStreamSink" or "FileStreamSinkConnector" to make the configuration a bit shorter

    Type:string
    Default:
    Valid Values:
    Importance:high

  • tasks.max

    Maximum number of tasks to use for this connector.

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:high

  • topics

    List of topics to consume, separated by commas

    Type:list
    Default:""
    Valid Values:
    Importance:high

  • topics.regex

    Regular expression giving topics to consume. Under the hood, the regex is compiled to a java.util.regex.Pattern. Only one of topics or topics.regex should be specified.

    Type:string
    Default:""
    Valid Values:valid regex
    Importance:high

  • tasks.max.enforce

    (Deprecated) Whether to enforce that the tasks.max property is respected by the connector. By default, connectors that generate too many tasks will fail, and existing sets of tasks that exceed the tasks.max property will also be failed. If this property is set to false, then connectors will be allowed to generate more than the maximum number of tasks, and existing sets of tasks that exceed the tasks.max property will be allowed to run. This property is deprecated and will be removed in an upcoming major release.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • key.converter

    Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.

    Type:class
    Default:null
    Valid Values:A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor
    Importance:low

  • value.converter

    Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.

    Type:class
    Default:null
    Valid Values:A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor
    Importance:low

  • header.converter

    HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas.

    Type:class
    Default:null
    Valid Values:A concrete subclass of org.apache.kafka.connect.storage.HeaderConverter, A class with a public, no-argument constructor
    Importance:low

  • config.action.reload

    The action that Connect should take on the connector when changes in external configuration providers result in a change in the connector's configuration properties. A value of 'none' indicates that Connect will do nothing. A value of 'restart' indicates that Connect should restart/reload the connector with the updated configuration properties.The restart may actually be scheduled in the future if the external configuration provider indicates that a configuration value will expire in the future.

    Type:string
    Default:restart
    Valid Values:[none, restart]
    Importance:low

  • transforms

    Aliases for the transformations to be applied to records.

    Type:list
    Default:""
    Valid Values:non-null string, unique transformation aliases
    Importance:low

  • predicates

    Aliases for the predicates used by transformations.

    Type:list
    Default:""
    Valid Values:non-null string, unique predicate aliases
    Importance:low

  • errors.retry.timeout

    The maximum duration in milliseconds that a failed operation will be reattempted. The default is 0, which means no retries will be attempted. Use -1 for infinite retries.

    Type:long
    Default:0
    Valid Values:
    Importance:medium

  • errors.retry.delay.max.ms

    The maximum duration in milliseconds between consecutive retry attempts. Jitter will be added to the delay once this limit is reached to prevent thundering herd issues.

    Type:long
    Default:60000 (1 minute)
    Valid Values:
    Importance:medium

  • errors.tolerance

    Behavior for tolerating errors during connector operation. 'none' is the default value and signals that any error will result in an immediate connector task failure; 'all' changes the behavior to skip over problematic records.

    Type:string
    Default:none
    Valid Values:[none, all]
    Importance:medium

  • errors.log.enable

    If true, write each error and the details of the failed operation and problematic record to the Connect application log. This is 'false' by default, so that only errors that are not tolerated are reported.

    Type:boolean
    Default:false
    Valid Values:
    Importance:medium

  • errors.log.include.messages

    Whether to include in the log the Connect record that resulted in a failure. For sink records, the topic, partition, offset, and timestamp will be logged. For source records, the key and value (and their schemas), all headers, and the timestamp, Kafka topic, Kafka partition, source partition, and source offset will be logged. This is 'false' by default, which will prevent record keys, values, and headers from being written to log files.

    Type:boolean
    Default:false
    Valid Values:
    Importance:medium

  • errors.deadletterqueue.topic.name

    The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, or its transformations or converters. The topic name is blank by default, which means that no messages are to be recorded in the DLQ.

    Type:string
    Default:""
    Valid Values:
    Importance:medium

  • errors.deadletterqueue.topic.replication.factor

    Replication factor used to create the dead letter queue topic when it doesn't already exist.

    Type:short
    Default:3
    Valid Values:
    Importance:medium

  • errors.deadletterqueue.context.headers.enable

    If true, add headers containing error context to the messages written to the dead letter queue. To avoid clashing with headers from the original record, all error context header keys, all error context header keys will start with __connect.errors.

    Type:boolean
    Default:false
    Valid Values:
    Importance:medium

3.6 Kafka Streams Configs

Below is the configuration of the Kafka Streams client library.

  • application.id

    An identifier for the stream processing application. Must be unique within the Kafka cluster. It is used as 1) the default client-id prefix, 2) the group-id for membership management, 3) the changelog topic prefix.

    Type:string
    Default:
    Valid Values:
    Importance:high

  • bootstrap.servers

    A list of host/port pairs used to establish the initial connection to the Kafka cluster. Clients use this list to bootstrap and discover the full set of Kafka brokers. While the order of servers in the list does not matter, we recommend including more than one server to ensure resilience if any servers are down. This list does not need to contain the entire set of brokers, as Kafka clients automatically manage and update connections to the cluster efficiently. This list must be in the form host1:port1,host2:port2,....

    Type:list
    Default:
    Valid Values:
    Importance:high

  • num.standby.replicas

    The number of standby replicas for each task.

    Type:int
    Default:0
    Valid Values:
    Importance:high

  • state.dir

    Directory location for state store. This path must be unique for each streams instance sharing the same underlying filesystem. Note that if not configured, then the default location will be different in each environment as it is computed using System.getProperty("java.io.tmpdir")

    Type:string
    Default:${java.io.tmpdir}
    Valid Values:
    Importance:high

  • acceptable.recovery.lag

    The maximum acceptable lag (number of offsets to catch up) for a client to be considered caught-up enough to receive an active task assignment. Upon assignment, it will still restore the rest of the changelog before processing. To avoid a pause in processing during rebalances, this config should correspond to a recovery time of well under a minute for a given workload. Must be at least 0.

    Type:long
    Default:10000
    Valid Values:[0,...]
    Importance:medium

  • cache.max.bytes.buffering

    Maximum number of memory bytes to be used for buffering across all threads

    Type:long
    Default:10485760
    Valid Values:[0,...]
    Importance:medium

  • client.id

    An ID prefix string used for the client IDs of internal (main, restore, and global) consumers , producers, and admin clients with pattern <client.id>-[Global]StreamThread[-<threadSequenceNumber>]-<consumer|producer|restore-consumer|global-consumer>.

    Type:string
    Default:<application.id>-<random-UUID>
    Valid Values:
    Importance:medium

  • default.deserialization.exception.handler

    Exception handling class that implements the org.apache.kafka.streams.errors.DeserializationExceptionHandler interface.

    Type:class
    Default:org.apache.kafka.streams.errors.LogAndFailExceptionHandler
    Valid Values:
    Importance:medium

  • default.key.serde

    Default serializer / deserializer class for key that implements the org.apache.kafka.common.serialization.Serde interface. Note when windowed serde class is used, one needs to set the inner serde class that implements the org.apache.kafka.common.serialization.Serde interface via 'default.windowed.key.serde.inner' or 'default.windowed.value.serde.inner' as well

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • default.list.key.serde.inner

    Default inner class of list serde for key that implements the org.apache.kafka.common.serialization.Serde interface. This configuration will be read if and only if default.key.serde configuration is set to org.apache.kafka.common.serialization.Serdes.ListSerde

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • default.list.key.serde.type

    Default class for key that implements the java.util.List interface. This configuration will be read if and only if default.key.serde configuration is set to org.apache.kafka.common.serialization.Serdes.ListSerde Note when list serde class is used, one needs to set the inner serde class that implements the org.apache.kafka.common.serialization.Serde interface via 'default.list.key.serde.inner'

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • default.list.value.serde.inner

    Default inner class of list serde for value that implements the org.apache.kafka.common.serialization.Serde interface. This configuration will be read if and only if default.value.serde configuration is set to org.apache.kafka.common.serialization.Serdes.ListSerde

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • default.list.value.serde.type

    Default class for value that implements the java.util.List interface. This configuration will be read if and only if default.value.serde configuration is set to org.apache.kafka.common.serialization.Serdes.ListSerde Note when list serde class is used, one needs to set the inner serde class that implements the org.apache.kafka.common.serialization.Serde interface via 'default.list.value.serde.inner'

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • default.production.exception.handler

    Exception handling class that implements the org.apache.kafka.streams.errors.ProductionExceptionHandler interface.

    Type:class
    Default:org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
    Valid Values:
    Importance:medium

  • default.timestamp.extractor

    Default timestamp extractor class that implements the org.apache.kafka.streams.processor.TimestampExtractor interface.

    Type:class
    Default:org.apache.kafka.streams.processor.FailOnInvalidTimestamp
    Valid Values:
    Importance:medium

  • default.value.serde

    Default serializer / deserializer class for value that implements the org.apache.kafka.common.serialization.Serde interface. Note when windowed serde class is used, one needs to set the inner serde class that implements the org.apache.kafka.common.serialization.Serde interface via 'default.windowed.key.serde.inner' or 'default.windowed.value.serde.inner' as well

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • max.task.idle.ms

    This config controls whether joins and merges may produce out-of-order results. The config value is the maximum amount of time in milliseconds a stream task will stay idle when it is fully caught up on some (but not all) input partitions to wait for producers to send additional records and avoid potential out-of-order record processing across multiple input streams. The default (zero) does not wait for producers to send more records, but it does wait to fetch data that is already present on the brokers. This default means that for records that are already present on the brokers, Streams will process them in timestamp order. Set to -1 to disable idling entirely and process any locally available data, even though doing so may produce out-of-order processing.

    Type:long
    Default:0
    Valid Values:
    Importance:medium

  • max.warmup.replicas

    The maximum number of warmup replicas (extra standbys beyond the configured num.standbys) that can be assigned at once for the purpose of keeping the task available on one instance while it is warming up on another instance it has been reassigned to. Used to throttle how much extra broker traffic and cluster state can be used for high availability. Must be at least 1.Note that one warmup replica corresponds to one Stream Task. Furthermore, note that each warmup replica can only be promoted to an active task during a rebalance (normally during a so-called probing rebalance, which occur at a frequency specified by the `probing.rebalance.interval.ms` config). This means that the maximum rate at which active tasks can be migrated from one Kafka Streams Instance to another instance can be determined by (`max.warmup.replicas` / `probing.rebalance.interval.ms`).

    Type:int
    Default:2
    Valid Values:[1,...]
    Importance:medium

  • num.stream.threads

    The number of threads to execute stream processing.

    Type:int
    Default:1
    Valid Values:
    Importance:medium

  • processing.exception.handler

    Exception handling class that implements the org.apache.kafka.streams.errors.ProcessingExceptionHandler interface.

    Type:class
    Default:org.apache.kafka.streams.errors.LogAndFailProcessingExceptionHandler
    Valid Values:
    Importance:medium

  • processing.guarantee

    The processing guarantee that should be used. Possible values are at_least_once (default) and exactly_once_v2 (requires brokers version 2.5 or higher). Deprecated options are exactly_once (requires brokers version 0.11.0 or higher) and exactly_once_beta (requires brokers version 2.5 or higher). Note that exactly-once processing requires a cluster of at least three brokers by default what is the recommended setting for production; for development you can change this, by adjusting broker setting transaction.state.log.replication.factor and transaction.state.log.min.isr.

    Type:string
    Default:at_least_once
    Valid Values:[at_least_once, exactly_once, exactly_once_beta, exactly_once_v2]
    Importance:medium

  • rack.aware.assignment.non_overlap_cost

    Cost associated with moving tasks from existing assignment. This config and rack.aware.assignment.traffic_cost controls whether the optimization algorithm favors minimizing cross rack traffic or minimize the movement of tasks in existing assignment. If set a larger value org.apache.kafka.streams.processor.internals.assignment.RackAwareTaskAssignor will optimize to maintain the existing assignment. The default value is null which means it will use default non_overlap cost values in different assignors.

    Type:int
    Default:null
    Valid Values:
    Importance:medium

  • rack.aware.assignment.strategy

    The strategy we use for rack aware assignment. Rack aware assignment will take client.rack and racks of TopicPartition into account when assigning tasks to minimize cross rack traffic. Valid settings are : none (default), which will disable rack aware assignment; min_traffic, which will compute minimum cross rack traffic assignment; balance_subtopology, which will compute minimum cross rack traffic and try to balance the tasks of same subtopolgies across different clients

    Type:string
    Default:none
    Valid Values:[none, min_traffic, balance_subtopology]
    Importance:medium

  • rack.aware.assignment.tags

    List of client tag keys used to distribute standby replicas across Kafka Streams instances. When configured, Kafka Streams will make a best-effort to distribute the standby tasks over each client tag dimension.

    Type:list
    Default:""
    Valid Values:List containing maximum of 5 elements
    Importance:medium

  • rack.aware.assignment.traffic_cost

    Cost associated with cross rack traffic. This config and rack.aware.assignment.non_overlap_cost controls whether the optimization algorithm favors minimizing cross rack traffic or minimize the movement of tasks in existing assignment. If set a larger value org.apache.kafka.streams.processor.internals.assignment.RackAwareTaskAssignor will optimize for minimizing cross rack traffic. The default value is null which means it will use default traffic cost values in different assignors.

    Type:int
    Default:null
    Valid Values:
    Importance:medium

  • replication.factor

    The replication factor for change log topics and repartition topics created by the stream processing application. The default of -1 (meaning: use broker default replication factor) requires broker version 2.4 or newer

    Type:int
    Default:-1
    Valid Values:
    Importance:medium

  • security.protocol

    Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.

    Type:string
    Default:PLAINTEXT
    Valid Values:(case insensitive) [SASL_SSL, PLAINTEXT, SSL, SASL_PLAINTEXT]
    Importance:medium

  • statestore.cache.max.bytes

    Maximum number of memory bytes to be used for statestore cache across all threads

    Type:long
    Default:10485760 (10 mebibytes)
    Valid Values:[0,...]
    Importance:medium

  • task.assignor.class

    A task assignor class or class name implementing the org.apache.kafka.streams.processor.assignment.TaskAssignor interface. Defaults to the HighAvailabilityTaskAssignor class.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • task.timeout.ms

    The maximum amount of time in milliseconds a task might stall due to internal errors and retries until an error is raised. For a timeout of 0ms, a task would raise an error for the first internal error. For any timeout larger than 0ms, a task will retry at least once before an error is raised.

    Type:long
    Default:300000 (5 minutes)
    Valid Values:[0,...]
    Importance:medium

  • topology.optimization

    A configuration telling Kafka Streams if it should optimize the topology and what optimizations to apply. Acceptable values are: "+NO_OPTIMIZATION+", "+OPTIMIZE+", or a comma separated list of specific optimizations: ("+REUSE_KTABLE_SOURCE_TOPICS+", "+MERGE_REPARTITION_TOPICS+" + "SINGLE_STORE_SELF_JOIN+")."NO_OPTIMIZATION" by default.

    Type:string
    Default:none
    Valid Values:[all, none, reuse.ktable.source.topics, merge.repartition.topics, single.store.self.join]
    Importance:medium

  • application.server

    A host:port pair pointing to a user-defined endpoint that can be used for state store discovery and interactive queries on this KafkaStreams instance.

    Type:string
    Default:""
    Valid Values:
    Importance:low

  • auto.include.jmx.reporter

    Deprecated. Whether to automatically include JmxReporter even if it's not listed in metric.reporters. This configuration will be removed in Kafka 4.0, users should instead include org.apache.kafka.common.metrics.JmxReporter in metric.reporters in order to enable the JmxReporter.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • buffered.records.per.partition

    Maximum number of records to buffer per partition.

    Type:int
    Default:1000
    Valid Values:
    Importance:low

  • built.in.metrics.version

    Version of the built-in metrics to use.

    Type:string
    Default:latest
    Valid Values:[latest]
    Importance:low

  • commit.interval.ms

    The frequency in milliseconds with which to commit processing progress. For at-least-once processing, committing means to save the position (ie, offsets) of the processor. For exactly-once processing, it means to commit the transaction which includes to save the position and to make the committed data in the output topic visible to consumers with isolation level read_committed. (Note, if processing.guarantee is set to exactly_once_v2, exactly_once,the default value is 100, otherwise the default value is 30000.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:[0,...]
    Importance:low

  • connections.max.idle.ms

    Close idle connections after the number of milliseconds specified by this config.

    Type:long
    Default:540000 (9 minutes)
    Valid Values:
    Importance:low

  • default.client.supplier

    Client supplier class that implements the org.apache.kafka.streams.KafkaClientSupplier interface.

    Type:class
    Default:org.apache.kafka.streams.processor.internals.DefaultKafkaClientSupplier
    Valid Values:
    Importance:low

  • default.dsl.store

    The default state store type used by DSL operators.

    Type:string
    Default:rocksDB
    Valid Values:[rocksDB, in_memory]
    Importance:low

  • dsl.store.suppliers.class

    Defines which store implementations to plug in to DSL operators. Must implement the org.apache.kafka.streams.state.DslStoreSuppliers interface.

    Type:class
    Default:org.apache.kafka.streams.state.BuiltInDslStoreSuppliers$RocksDBDslStoreSuppliers
    Valid Values:
    Importance:low

  • enable.metrics.push

    Whether to enable pushing of internal client metrics for (main, restore, and global) consumers, producers, and admin clients. The cluster must have a client metrics subscription which corresponds to a client.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • log.summary.interval.ms

    This configuration controls the output interval for summary information.
    If greater or equal to 0, the summary log will be output according to the set time interval;
    If less than 0, summary output is disabled.

    Type:long
    Default:120000 (2 minutes)
    Valid Values:
    Importance:low

  • metadata.max.age.ms

    The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.

    Type:long
    Default:300000 (5 minutes)
    Valid Values:[0,...]
    Importance:low

  • metric.reporters

    A list of classes to use as metrics reporters. Implementing the org.apache.kafka.common.metrics.MetricsReporter interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.

    Type:list
    Default:""
    Valid Values:
    Importance:low

  • metrics.num.samples

    The number of samples maintained to compute metrics.

    Type:int
    Default:2
    Valid Values:[1,...]
    Importance:low

  • metrics.recording.level

    The highest recording level for metrics.

    Type:string
    Default:INFO
    Valid Values:[INFO, DEBUG, TRACE]
    Importance:low

  • metrics.sample.window.ms

    The window of time a metrics sample is computed over.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:[0,...]
    Importance:low

  • poll.ms

    The amount of time in milliseconds to block waiting for input.

    Type:long
    Default:100
    Valid Values:
    Importance:low

  • probing.rebalance.interval.ms

    The maximum time in milliseconds to wait before triggering a rebalance to probe for warmup replicas that have finished warming up and are ready to become active. Probing rebalances will continue to be triggered until the assignment is balanced. Must be at least 1 minute.

    Type:long
    Default:600000 (10 minutes)
    Valid Values:[60000,...]
    Importance:low

  • receive.buffer.bytes

    The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.

    Type:int
    Default:32768 (32 kibibytes)
    Valid Values:[-1,...]
    Importance:low

  • reconnect.backoff.max.ms

    The maximum amount of time in milliseconds to wait when reconnecting to a broker that has repeatedly failed to connect. If provided, the backoff per host will increase exponentially for each consecutive connection failure, up to this maximum. After calculating the backoff increase, 20% random jitter is added to avoid connection storms.

    Type:long
    Default:1000 (1 second)
    Valid Values:[0,...]
    Importance:low

  • reconnect.backoff.ms

    The base amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all connection attempts by the client to a broker. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the reconnect.backoff.max.ms value.

    Type:long
    Default:50
    Valid Values:[0,...]
    Importance:low

  • repartition.purge.interval.ms

    The frequency in milliseconds with which to delete fully consumed records from repartition topics. Purging will occur after at least this value since the last purge, but may be delayed until later. (Note, unlike commit.interval.ms, the default for this value remains unchanged when processing.guarantee is set to exactly_once_v2).

    Type:long
    Default:30000 (30 seconds)
    Valid Values:[0,...]
    Importance:low

  • request.timeout.ms

    The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

    Type:int
    Default:40000 (40 seconds)
    Valid Values:[0,...]
    Importance:low

  • retries

    Setting a value greater than zero will cause the client to resend any request that fails with a potentially transient error. It is recommended to set the value to either zero or `MAX_VALUE` and use corresponding timeout parameters to control how long a client should retry a request.

    Type:int
    Default:0
    Valid Values:[0,...,2147483647]
    Importance:low

  • retry.backoff.ms

    The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. This value is the initial backoff value and will increase exponentially for each failed request, up to the retry.backoff.max.ms value.

    Type:long
    Default:100
    Valid Values:[0,...]
    Importance:low

  • rocksdb.config.setter

    A Rocks DB config setter class or class name that implements the org.apache.kafka.streams.state.RocksDBConfigSetter interface

    Type:class
    Default:null
    Valid Values:
    Importance:low

  • send.buffer.bytes

    The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.

    Type:int
    Default:131072 (128 kibibytes)
    Valid Values:[-1,...]
    Importance:low

  • state.cleanup.delay.ms

    The amount of time in milliseconds to wait before deleting state when a partition has migrated. Only state directories that have not been modified for at least state.cleanup.delay.ms will be removed

    Type:long
    Default:600000 (10 minutes)
    Valid Values:
    Importance:low

  • upgrade.from

    Allows upgrading in a backward compatible way. This is needed when upgrading from [0.10.0, 1.1] to 2.0+, or when upgrading from [2.0, 2.3] to 2.4+. When upgrading from 3.3 to a newer version it is not required to specify this config. Default is `null`. Accepted values are "0.10.0", "0.10.1", "0.10.2", "0.11.0", "1.0", "1.1", "2.0", "2.1", "2.2", "2.3", "2.4", "2.5", "2.6", "2.7", "2.8", "3.0", "3.1", "3.2", "3.3", "3.4", "3.5", "3.6", "3.7", "3.8(for upgrading from the corresponding old version).

    Type:string
    Default:null
    Valid Values:[null, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0, 1.1, 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8]
    Importance:low

  • window.size.ms

    Sets window size for the deserializer in order to calculate window end times.

    Type:long
    Default:null
    Valid Values:
    Importance:low

  • windowed.inner.class.serde

    Default serializer / deserializer for the inner class of a windowed record. Must implement the org.apache.kafka.common.serialization.Serde interface. Note that setting this config in KafkaStreams application would result in an error as it is meant to be used only from Plain consumer client.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • windowstore.changelog.additional.retention.ms

    Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day

    Type:long
    Default:86400000 (1 day)
    Valid Values:
    Importance:low

3.7 Admin Configs

Below is the configuration of the Kafka Admin client library.

  • bootstrap.controllers

    A list of host/port pairs to use for establishing the initial connection to the KRaft controller quorum. This list should be in the form host1:port1,host2:port2,....

    Type:list
    Default:""
    Valid Values:
    Importance:high

  • bootstrap.servers

    A list of host/port pairs used to establish the initial connection to the Kafka cluster. Clients use this list to bootstrap and discover the full set of Kafka brokers. While the order of servers in the list does not matter, we recommend including more than one server to ensure resilience if any servers are down. This list does not need to contain the entire set of brokers, as Kafka clients automatically manage and update connections to the cluster efficiently. This list must be in the form host1:port1,host2:port2,....

    Type:list
    Default:""
    Valid Values:
    Importance:high

  • ssl.key.password

    The password of the private key in the key store file or the PEM key specified in 'ssl.keystore.key'.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.certificate.chain

    Certificate chain in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with a list of X.509 certificates

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.key

    Private key in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with PKCS#8 keys. If the key is encrypted, key password must be specified using 'ssl.key.password'

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.location

    The location of the key store file. This is optional for client and can be used for two-way authentication for client.

    Type:string
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.password

    The store password for the key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured. Key store password is not supported for PEM format.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.certificates

    Trusted certificates in the format specified by 'ssl.truststore.type'. Default SSL engine factory supports only PEM format with X.509 certificates.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.location

    The location of the trust store file.

    Type:string
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.password

    The password for the trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled. Trust store password is not supported for PEM format.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • client.dns.lookup

    Controls how the client uses DNS lookups. If set to use_all_dns_ips, connect to each returned IP address in sequence until a successful connection is established. After a disconnection, the next IP is used. Once all IPs have been used once, the client resolves the IP(s) from the hostname again (both the JVM and the OS cache DNS name lookups, however). If set to resolve_canonical_bootstrap_servers_only, resolve each bootstrap address into a list of canonical names. After the bootstrap phase, this behaves the same as use_all_dns_ips.

    Type:string
    Default:use_all_dns_ips
    Valid Values:[use_all_dns_ips, resolve_canonical_bootstrap_servers_only]
    Importance:medium

  • client.id

    An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.

    Type:string
    Default:""
    Valid Values:
    Importance:medium

  • connections.max.idle.ms

    Close idle connections after the number of milliseconds specified by this config.

    Type:long
    Default:300000 (5 minutes)
    Valid Values:
    Importance:medium

  • default.api.timeout.ms

    Specifies the timeout (in milliseconds) for client APIs. This configuration is used as the default timeout for all client operations that do not specify a timeout parameter.

    Type:int
    Default:60000 (1 minute)
    Valid Values:[0,...]
    Importance:medium

  • receive.buffer.bytes

    The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.

    Type:int
    Default:65536 (64 kibibytes)
    Valid Values:[-1,...]
    Importance:medium

  • request.timeout.ms

    The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

    Type:int
    Default:30000 (30 seconds)
    Valid Values:[0,...]
    Importance:medium

  • sasl.client.callback.handler.class

    The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface.

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.jaas.config

    JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is: loginModuleClass controlFlag (optionName=optionValue)*;. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;

    Type:password
    Default:null
    Valid Values:
    Importance:medium

  • sasl.kerberos.service.name

    The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • sasl.login.callback.handler.class

    The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.login.class

    The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.mechanism

    SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.

    Type:string
    Default:GSSAPI
    Valid Values:
    Importance:medium

  • sasl.oauthbearer.jwks.endpoint.url

    The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. If the URL is HTTP(S)-based, the JWKS data will be retrieved from the OAuth/OIDC provider via the configured URL on broker startup. All then-current keys will be cached on the broker for incoming requests. If an authentication request is received for a JWT that includes a "kid" header claim value that isn't yet in the cache, the JWKS endpoint will be queried again on demand. However, the broker polls the URL every sasl.oauthbearer.jwks.endpoint.refresh.ms milliseconds to refresh the cache with any forthcoming keys before any JWT requests that include them are received. If the URL is file-based, the broker will load the JWKS file from a configured location on startup. In the event that the JWT includes a "kid" header value that isn't in the JWKS file, the broker will reject the JWT and authentication will fail.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • sasl.oauthbearer.token.endpoint.url

    The URL for the OAuth/OIDC identity provider. If the URL is HTTP(S)-based, it is the issuer's token endpoint URL to which requests will be made to login based on the configuration in sasl.jaas.config. If the URL is file-based, it specifies a file containing an access token (in JWT serialized form) issued by the OAuth/OIDC identity provider to use for authorization.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • security.protocol

    Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.

    Type:string
    Default:PLAINTEXT
    Valid Values:(case insensitive) [SASL_SSL, PLAINTEXT, SSL, SASL_PLAINTEXT]
    Importance:medium

  • send.buffer.bytes

    The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.

    Type:int
    Default:131072 (128 kibibytes)
    Valid Values:[-1,...]
    Importance:medium

  • socket.connection.setup.timeout.max.ms

    The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. To avoid connection storms, a randomization factor of 0.2 will be applied to the timeout resulting in a random range between 20% below and 20% above the computed value.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:
    Importance:medium

  • socket.connection.setup.timeout.ms

    The amount of time the client will wait for the socket connection to be established. If the connection is not built before the timeout elapses, clients will close the socket channel. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the socket.connection.setup.timeout.max.ms value.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:medium

  • ssl.enabled.protocols

    The list of protocols enabled for SSL connections. The default is 'TLSv1.2,TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). This default should be fine for most cases. Also see the config documentation for `ssl.protocol`.

    Type:list
    Default:TLSv1.2
    Valid Values:
    Importance:medium

  • ssl.keystore.type

    The file format of the key store file. This is optional for client. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].

    Type:string
    Default:JKS
    Valid Values:
    Importance:medium

  • ssl.protocol

    The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'.

    Type:string
    Default:TLSv1.2
    Valid Values:
    Importance:medium

  • ssl.provider

    The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • ssl.truststore.type

    The file format of the trust store file. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].

    Type:string
    Default:JKS
    Valid Values:
    Importance:medium

  • auto.include.jmx.reporter

    Deprecated. Whether to automatically include JmxReporter even if it's not listed in metric.reporters. This configuration will be removed in Kafka 4.0, users should instead include org.apache.kafka.common.metrics.JmxReporter in metric.reporters in order to enable the JmxReporter.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • enable.metrics.push

    Whether to enable pushing of client metrics to the cluster, if the cluster has a client metrics subscription which matches this client.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • metadata.max.age.ms

    The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.

    Type:long
    Default:300000 (5 minutes)
    Valid Values:[0,...]
    Importance:low

  • metadata.recovery.strategy

    Controls how the client recovers when none of the brokers known to it is available. If set to none, the client fails. If set to rebootstrap, the client repeats the bootstrap process using bootstrap.servers. Rebootstrapping is useful when a client communicates with brokers so infrequently that the set of brokers may change entirely before the client refreshes metadata. Metadata recovery is triggered when all last-known brokers appear unavailable simultaneously. Brokers appear unavailable when disconnected and no current retry attempt is in-progress. Consider increasing reconnect.backoff.ms and reconnect.backoff.max.ms and decreasing socket.connection.setup.timeout.ms and socket.connection.setup.timeout.max.ms for the client.

    Type:string
    Default:none
    Valid Values:(case insensitive) [REBOOTSTRAP, NONE]
    Importance:low

  • metric.reporters

    A list of classes to use as metrics reporters. Implementing the org.apache.kafka.common.metrics.MetricsReporter interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.

    Type:list
    Default:""
    Valid Values:
    Importance:low

  • metrics.num.samples

    The number of samples maintained to compute metrics.

    Type:int
    Default:2
    Valid Values:[1,...]
    Importance:low

  • metrics.recording.level

    The highest recording level for metrics.

    Type:string
    Default:INFO
    Valid Values:[INFO, DEBUG, TRACE]
    Importance:low

  • metrics.sample.window.ms

    The window of time a metrics sample is computed over.

    Type:long
    Default:30000 (30 seconds)
    Valid Values:[0,...]
    Importance:low

  • reconnect.backoff.max.ms

    The maximum amount of time in milliseconds to wait when reconnecting to a broker that has repeatedly failed to connect. If provided, the backoff per host will increase exponentially for each consecutive connection failure, up to this maximum. After calculating the backoff increase, 20% random jitter is added to avoid connection storms.

    Type:long
    Default:1000 (1 second)
    Valid Values:[0,...]
    Importance:low

  • reconnect.backoff.ms

    The base amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all connection attempts by the client to a broker. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the reconnect.backoff.max.ms value.

    Type:long
    Default:50
    Valid Values:[0,...]
    Importance:low

  • retries

    Setting a value greater than zero will cause the client to resend any request that fails with a potentially transient error. It is recommended to set the value to either zero or `MAX_VALUE` and use corresponding timeout parameters to control how long a client should retry a request.

    Type:int
    Default:2147483647
    Valid Values:[0,...,2147483647]
    Importance:low

  • retry.backoff.max.ms

    The maximum amount of time in milliseconds to wait when retrying a request to the broker that has repeatedly failed. If provided, the backoff per client will increase exponentially for each failed request, up to this maximum. To prevent all clients from being synchronized upon retry, a randomized jitter with a factor of 0.2 will be applied to the backoff, resulting in the backoff falling within a range between 20% below and 20% above the computed value. If retry.backoff.ms is set to be higher than retry.backoff.max.ms, then retry.backoff.max.ms will be used as a constant backoff from the beginning without any exponential increase

    Type:long
    Default:1000 (1 second)
    Valid Values:[0,...]
    Importance:low

  • retry.backoff.ms

    The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. This value is the initial backoff value and will increase exponentially for each failed request, up to the retry.backoff.max.ms value.

    Type:long
    Default:100
    Valid Values:[0,...]
    Importance:low

  • sasl.kerberos.kinit.cmd

    Kerberos kinit command path.

    Type:string
    Default:/usr/bin/kinit
    Valid Values:
    Importance:low

  • sasl.kerberos.min.time.before.relogin

    Login thread sleep time between refresh attempts.

    Type:long
    Default:60000
    Valid Values:
    Importance:low

  • sasl.kerberos.ticket.renew.jitter

    Percentage of random jitter added to the renewal time.

    Type:double
    Default:0.05
    Valid Values:
    Importance:low

  • sasl.kerberos.ticket.renew.window.factor

    Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.

    Type:double
    Default:0.8
    Valid Values:
    Importance:low

  • sasl.login.connect.timeout.ms

    The (optional) value in milliseconds for the external authentication provider connection timeout. Currently applies only to OAUTHBEARER.

    Type:int
    Default:null
    Valid Values:
    Importance:low

  • sasl.login.read.timeout.ms

    The (optional) value in milliseconds for the external authentication provider read timeout. Currently applies only to OAUTHBEARER.

    Type:int
    Default:null
    Valid Values:
    Importance:low

  • sasl.login.refresh.buffer.seconds

    The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.

    Type:short
    Default:300
    Valid Values:[0,...,3600]
    Importance:low

  • sasl.login.refresh.min.period.seconds

    The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.

    Type:short
    Default:60
    Valid Values:[0,...,900]
    Importance:low

  • sasl.login.refresh.window.factor

    Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER.

    Type:double
    Default:0.8
    Valid Values:[0.5,...,1.0]
    Importance:low

  • sasl.login.refresh.window.jitter

    The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.

    Type:double
    Default:0.05
    Valid Values:[0.0,...,0.25]
    Importance:low

  • sasl.login.retry.backoff.max.ms

    The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:low

  • sasl.login.retry.backoff.ms

    The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.

    Type:long
    Default:100
    Valid Values:
    Importance:low

  • sasl.oauthbearer.clock.skew.seconds

    The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker.

    Type:int
    Default:30
    Valid Values:
    Importance:low

  • sasl.oauthbearer.expected.audience

    The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWT's "aud" claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.

    Type:list
    Default:null
    Valid Values:
    Importance:low

  • sasl.oauthbearer.expected.issuer

    The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth "iss" claim and if this value is set, the broker will match it exactly against what is in the JWT's "iss" claim. If there is no match, the broker will reject the JWT and authentication will fail.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • sasl.oauthbearer.header.urlencode

    The (optional) setting to enable the OAuth client to URL-encode the client_id and client_secret in the authorization header in accordance with RFC6749, see here for more details. The default value is set to 'false' for backward compatibility

    Type:boolean
    Default:false
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.refresh.ms

    The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.

    Type:long
    Default:3600000 (1 hour)
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms

    The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.retry.backoff.ms

    The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.

    Type:long
    Default:100
    Valid Values:
    Importance:low

  • sasl.oauthbearer.scope.claim.name

    The OAuth claim for the scope is often named "scope", but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

    Type:string
    Default:scope
    Valid Values:
    Importance:low

  • sasl.oauthbearer.sub.claim.name

    The OAuth claim for the subject is often named "sub", but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

    Type:string
    Default:sub
    Valid Values:
    Importance:low

  • security.providers

    A list of configurable creator classes each returning a provider implementing security algorithms. These classes should implement the org.apache.kafka.common.security.auth.SecurityProviderCreator interface.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • ssl.cipher.suites

    A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.

    Type:list
    Default:null
    Valid Values:
    Importance:low

  • ssl.endpoint.identification.algorithm

    The endpoint identification algorithm to validate server hostname using server certificate.

    Type:string
    Default:https
    Valid Values:
    Importance:low

  • ssl.engine.factory.class

    The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is org.apache.kafka.common.security.ssl.DefaultSslEngineFactory. Alternatively, setting this to org.apache.kafka.common.security.ssl.CommonNameLoggingSslEngineFactory will log the common name of expired SSL certificates used by clients to authenticate at any of the brokers with log level INFO. Note that this will cause a tiny delay during establishment of new connections from mTLS clients to brokers due to the extra code for examining the certificate chain provided by the client. Note further that the implementation uses a custom truststore based on the standard Java truststore and thus might be considered a security risk due to not being as mature as the standard one.

    Type:class
    Default:null
    Valid Values:
    Importance:low

  • ssl.keymanager.algorithm

    The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.

    Type:string
    Default:SunX509
    Valid Values:
    Importance:low

  • ssl.secure.random.implementation

    The SecureRandom PRNG implementation to use for SSL cryptography operations.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • ssl.trustmanager.algorithm

    The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.

    Type:string
    Default:PKIX
    Valid Values:
    Importance:low

3.8 MirrorMaker Configs

Below is the configuration of the connectors that make up MirrorMaker 2.

3.8.1 MirrorMaker Common Configs

Below are the common configuration properties that apply to all three connectors.

  • source.cluster.alias

    Alias of source cluster

    Type:string
    Default:
    Valid Values:
    Importance:high

  • ssl.key.password

    The password of the private key in the key store file or the PEM key specified in 'ssl.keystore.key'.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.certificate.chain

    Certificate chain in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with a list of X.509 certificates

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.key

    Private key in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with PKCS#8 keys. If the key is encrypted, key password must be specified using 'ssl.key.password'

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.location

    The location of the key store file. This is optional for client and can be used for two-way authentication for client.

    Type:string
    Default:null
    Valid Values:
    Importance:high

  • ssl.keystore.password

    The store password for the key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured. Key store password is not supported for PEM format.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.certificates

    Trusted certificates in the format specified by 'ssl.truststore.type'. Default SSL engine factory supports only PEM format with X.509 certificates.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.location

    The location of the trust store file.

    Type:string
    Default:null
    Valid Values:
    Importance:high

  • ssl.truststore.password

    The password for the trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled. Trust store password is not supported for PEM format.

    Type:password
    Default:null
    Valid Values:
    Importance:high

  • target.cluster.alias

    Alias of target cluster. Used in metrics reporting.

    Type:string
    Default:target
    Valid Values:
    Importance:high

  • sasl.client.callback.handler.class

    The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface.

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.jaas.config

    JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is: loginModuleClass controlFlag (optionName=optionValue)*;. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;

    Type:password
    Default:null
    Valid Values:
    Importance:medium

  • sasl.kerberos.service.name

    The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • sasl.login.callback.handler.class

    The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.login.class

    The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin

    Type:class
    Default:null
    Valid Values:
    Importance:medium

  • sasl.mechanism

    SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.

    Type:string
    Default:GSSAPI
    Valid Values:
    Importance:medium

  • sasl.oauthbearer.jwks.endpoint.url

    The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. If the URL is HTTP(S)-based, the JWKS data will be retrieved from the OAuth/OIDC provider via the configured URL on broker startup. All then-current keys will be cached on the broker for incoming requests. If an authentication request is received for a JWT that includes a "kid" header claim value that isn't yet in the cache, the JWKS endpoint will be queried again on demand. However, the broker polls the URL every sasl.oauthbearer.jwks.endpoint.refresh.ms milliseconds to refresh the cache with any forthcoming keys before any JWT requests that include them are received. If the URL is file-based, the broker will load the JWKS file from a configured location on startup. In the event that the JWT includes a "kid" header value that isn't in the JWKS file, the broker will reject the JWT and authentication will fail.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • sasl.oauthbearer.token.endpoint.url

    The URL for the OAuth/OIDC identity provider. If the URL is HTTP(S)-based, it is the issuer's token endpoint URL to which requests will be made to login based on the configuration in sasl.jaas.config. If the URL is file-based, it specifies a file containing an access token (in JWT serialized form) issued by the OAuth/OIDC identity provider to use for authorization.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • security.protocol

    Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.

    Type:string
    Default:PLAINTEXT
    Valid Values:(case insensitive) [SASL_SSL, PLAINTEXT, SSL, SASL_PLAINTEXT]
    Importance:medium

  • ssl.enabled.protocols

    The list of protocols enabled for SSL connections. The default is 'TLSv1.2,TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). This default should be fine for most cases. Also see the config documentation for `ssl.protocol`.

    Type:list
    Default:TLSv1.2
    Valid Values:
    Importance:medium

  • ssl.keystore.type

    The file format of the key store file. This is optional for client. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].

    Type:string
    Default:JKS
    Valid Values:
    Importance:medium

  • ssl.protocol

    The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'.

    Type:string
    Default:TLSv1.2
    Valid Values:
    Importance:medium

  • ssl.provider

    The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.

    Type:string
    Default:null
    Valid Values:
    Importance:medium

  • ssl.truststore.type

    The file format of the trust store file. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].

    Type:string
    Default:JKS
    Valid Values:
    Importance:medium

  • admin.timeout.ms

    Timeout for administrative tasks, e.g. detecting new topics.

    Type:long
    Default:60000 (1 minute)
    Valid Values:
    Importance:low

  • auto.include.jmx.reporter

    Deprecated. Whether to automatically include JmxReporter even if it's not listed in metric.reporters. This configuration will be removed in Kafka 4.0, users should instead include org.apache.kafka.common.metrics.JmxReporter in metric.reporters in order to enable the JmxReporter.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • enabled

    Whether to replicate source->target.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • forwarding.admin.class

    Class which extends ForwardingAdmin to define custom cluster resource management (topics, configs, etc). The class must have a constructor with signature (Map config) that is used to configure a KafkaAdminClient and may also be used to configure clients for external systems if necessary.

    Type:class
    Default:org.apache.kafka.clients.admin.ForwardingAdmin
    Valid Values:
    Importance:low

  • metric.reporters

    A list of classes to use as metrics reporters. Implementing the org.apache.kafka.common.metrics.MetricsReporter interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.

    Type:list
    Default:null
    Valid Values:
    Importance:low

  • replication.policy.class

    Class which defines the remote topic naming convention.

    Type:class
    Default:org.apache.kafka.connect.mirror.DefaultReplicationPolicy
    Valid Values:
    Importance:low

  • replication.policy.internal.topic.separator.enabled

    Whether to use replication.policy.separator to control the names of topics used for checkpoints and offset syncs. By default, custom separators are used in these topic names; however, if upgrading MirrorMaker 2 from older versions that did not allow for these topic names to be customized, it may be necessary to set this property to 'false' in order to continue using the same names for those topics.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • replication.policy.separator

    Separator used in remote topic naming convention.

    Type:string
    Default:.
    Valid Values:
    Importance:low

  • sasl.kerberos.kinit.cmd

    Kerberos kinit command path.

    Type:string
    Default:/usr/bin/kinit
    Valid Values:
    Importance:low

  • sasl.kerberos.min.time.before.relogin

    Login thread sleep time between refresh attempts.

    Type:long
    Default:60000
    Valid Values:
    Importance:low

  • sasl.kerberos.ticket.renew.jitter

    Percentage of random jitter added to the renewal time.

    Type:double
    Default:0.05
    Valid Values:
    Importance:low

  • sasl.kerberos.ticket.renew.window.factor

    Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.

    Type:double
    Default:0.8
    Valid Values:
    Importance:low

  • sasl.login.connect.timeout.ms

    The (optional) value in milliseconds for the external authentication provider connection timeout. Currently applies only to OAUTHBEARER.

    Type:int
    Default:null
    Valid Values:
    Importance:low

  • sasl.login.read.timeout.ms

    The (optional) value in milliseconds for the external authentication provider read timeout. Currently applies only to OAUTHBEARER.

    Type:int
    Default:null
    Valid Values:
    Importance:low

  • sasl.login.refresh.buffer.seconds

    The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.

    Type:short
    Default:300
    Valid Values:[0,...,3600]
    Importance:low

  • sasl.login.refresh.min.period.seconds

    The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.

    Type:short
    Default:60
    Valid Values:[0,...,900]
    Importance:low

  • sasl.login.refresh.window.factor

    Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER.

    Type:double
    Default:0.8
    Valid Values:[0.5,...,1.0]
    Importance:low

  • sasl.login.refresh.window.jitter

    The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.

    Type:double
    Default:0.05
    Valid Values:[0.0,...,0.25]
    Importance:low

  • sasl.login.retry.backoff.max.ms

    The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:low

  • sasl.login.retry.backoff.ms

    The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.

    Type:long
    Default:100
    Valid Values:
    Importance:low

  • sasl.oauthbearer.clock.skew.seconds

    The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker.

    Type:int
    Default:30
    Valid Values:
    Importance:low

  • sasl.oauthbearer.expected.audience

    The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWT's "aud" claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.

    Type:list
    Default:null
    Valid Values:
    Importance:low

  • sasl.oauthbearer.expected.issuer

    The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth "iss" claim and if this value is set, the broker will match it exactly against what is in the JWT's "iss" claim. If there is no match, the broker will reject the JWT and authentication will fail.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • sasl.oauthbearer.header.urlencode

    The (optional) setting to enable the OAuth client to URL-encode the client_id and client_secret in the authorization header in accordance with RFC6749, see here for more details. The default value is set to 'false' for backward compatibility

    Type:boolean
    Default:false
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.refresh.ms

    The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.

    Type:long
    Default:3600000 (1 hour)
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms

    The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.

    Type:long
    Default:10000 (10 seconds)
    Valid Values:
    Importance:low

  • sasl.oauthbearer.jwks.endpoint.retry.backoff.ms

    The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.

    Type:long
    Default:100
    Valid Values:
    Importance:low

  • sasl.oauthbearer.scope.claim.name

    The OAuth claim for the scope is often named "scope", but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

    Type:string
    Default:scope
    Valid Values:
    Importance:low

  • sasl.oauthbearer.sub.claim.name

    The OAuth claim for the subject is often named "sub", but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

    Type:string
    Default:sub
    Valid Values:
    Importance:low

  • ssl.cipher.suites

    A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.

    Type:list
    Default:null
    Valid Values:
    Importance:low

  • ssl.endpoint.identification.algorithm

    The endpoint identification algorithm to validate server hostname using server certificate.

    Type:string
    Default:https
    Valid Values:
    Importance:low

  • ssl.engine.factory.class

    The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is org.apache.kafka.common.security.ssl.DefaultSslEngineFactory. Alternatively, setting this to org.apache.kafka.common.security.ssl.CommonNameLoggingSslEngineFactory will log the common name of expired SSL certificates used by clients to authenticate at any of the brokers with log level INFO. Note that this will cause a tiny delay during establishment of new connections from mTLS clients to brokers due to the extra code for examining the certificate chain provided by the client. Note further that the implementation uses a custom truststore based on the standard Java truststore and thus might be considered a security risk due to not being as mature as the standard one.

    Type:class
    Default:null
    Valid Values:
    Importance:low

  • ssl.keymanager.algorithm

    The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.

    Type:string
    Default:SunX509
    Valid Values:
    Importance:low

  • ssl.secure.random.implementation

    The SecureRandom PRNG implementation to use for SSL cryptography operations.

    Type:string
    Default:null
    Valid Values:
    Importance:low

  • ssl.trustmanager.algorithm

    The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.

    Type:string
    Default:PKIX
    Valid Values:
    Importance:low

  • name

    Globally unique name to use for this connector.

    Type:string
    Default:
    Valid Values:non-empty string without ISO control characters
    Importance:high

  • connector.class

    Name or alias of the class for this connector. Must be a subclass of org.apache.kafka.connect.connector.Connector. If the connector is org.apache.kafka.connect.file.FileStreamSinkConnector, you can either specify this full name, or use "FileStreamSink" or "FileStreamSinkConnector" to make the configuration a bit shorter

    Type:string
    Default:
    Valid Values:
    Importance:high

  • tasks.max

    Maximum number of tasks to use for this connector.

    Type:int
    Default:1
    Valid Values:[1,...]
    Importance:high

  • tasks.max.enforce

    (Deprecated) Whether to enforce that the tasks.max property is respected by the connector. By default, connectors that generate too many tasks will fail, and existing sets of tasks that exceed the tasks.max property will also be failed. If this property is set to false, then connectors will be allowed to generate more than the maximum number of tasks, and existing sets of tasks that exceed the tasks.max property will be allowed to run. This property is deprecated and will be removed in an upcoming major release.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • key.converter

    Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.

    Type:class
    Default:null
    Valid Values:A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor
    Importance:low

  • value.converter

    Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.

    Type:class
    Default:null
    Valid Values:A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor
    Importance:low

  • header.converter

    HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas.

    Type:class
    Default:null
    Valid Values:A concrete subclass of org.apache.kafka.connect.storage.HeaderConverter, A class with a public, no-argument constructor
    Importance:low

  • config.action.reload

    The action that Connect should take on the connector when changes in external configuration providers result in a change in the connector's configuration properties. A value of 'none' indicates that Connect will do nothing. A value of 'restart' indicates that Connect should restart/reload the connector with the updated configuration properties.The restart may actually be scheduled in the future if the external configuration provider indicates that a configuration value will expire in the future.

    Type:string
    Default:restart
    Valid Values:[none, restart]
    Importance:low

  • transforms

    Aliases for the transformations to be applied to records.

    Type:list
    Default:""
    Valid Values:non-null string, unique transformation aliases
    Importance:low

  • predicates

    Aliases for the predicates used by transformations.

    Type:list
    Default:""
    Valid Values:non-null string, unique predicate aliases
    Importance:low

  • errors.retry.timeout

    The maximum duration in milliseconds that a failed operation will be reattempted. The default is 0, which means no retries will be attempted. Use -1 for infinite retries.

    Type:long
    Default:0
    Valid Values:
    Importance:medium

  • errors.retry.delay.max.ms

    The maximum duration in milliseconds between consecutive retry attempts. Jitter will be added to the delay once this limit is reached to prevent thundering herd issues.

    Type:long
    Default:60000 (1 minute)
    Valid Values:
    Importance:medium

  • errors.tolerance

    Behavior for tolerating errors during connector operation. 'none' is the default value and signals that any error will result in an immediate connector task failure; 'all' changes the behavior to skip over problematic records.

    Type:string
    Default:none
    Valid Values:[none, all]
    Importance:medium

  • errors.log.enable

    If true, write each error and the details of the failed operation and problematic record to the Connect application log. This is 'false' by default, so that only errors that are not tolerated are reported.

    Type:boolean
    Default:false
    Valid Values:
    Importance:medium

  • errors.log.include.messages

    Whether to include in the log the Connect record that resulted in a failure. For sink records, the topic, partition, offset, and timestamp will be logged. For source records, the key and value (and their schemas), all headers, and the timestamp, Kafka topic, Kafka partition, source partition, and source offset will be logged. This is 'false' by default, which will prevent record keys, values, and headers from being written to log files.

    Type:boolean
    Default:false
    Valid Values:
    Importance:medium

3.8.2 MirrorMaker Source Configs

Below is the configuration of MirrorMaker 2 source connector for replicating topics.

  • config.properties.blacklist

    Deprecated. Use config.properties.exclude instead.

    Type:list
    Default:null
    Valid Values:
    Importance:high

  • config.properties.exclude

    Topic config properties that should not be replicated. Supports comma-separated property names and regexes.

    Type:list
    Default:follower\.replication\.throttled\.replicas,leader\.replication\.throttled\.replicas,message\.timestamp\.difference\.max\.ms,message\.timestamp\.type,unclean\.leader\.election\.enable,min\.insync\.replicas
    Valid Values:
    Importance:high

  • topics

    Topics to replicate. Supports comma-separated topic names and regexes.

    Type:list
    Default:.*
    Valid Values:
    Importance:high

  • topics.blacklist

    Deprecated. Use topics.exclude instead.

    Type:list
    Default:null
    Valid Values:
    Importance:high

  • topics.exclude

    Excluded topics. Supports comma-separated topic names and regexes. Excludes take precedence over includes.

    Type:list
    Default:.*[\-\.]internal,.*\.replica,__.*
    Valid Values:
    Importance:high

  • add.source.alias.to.metrics

    Deprecated. Whether to tag metrics with the source cluster alias. Metrics have the target, topic and partition tags. When this setting is enabled, it adds the source tag. This configuration will be removed in Kafka 4.0 and the default behavior will be to always have the source tag.

    Type:boolean
    Default:false
    Valid Values:
    Importance:low

  • config.property.filter.class

    ConfigPropertyFilter to use. Selects topic config properties to replicate.

    Type:class
    Default:org.apache.kafka.connect.mirror.DefaultConfigPropertyFilter
    Valid Values:
    Importance:low

  • consumer.poll.timeout.ms

    Timeout when polling source cluster.

    Type:long
    Default:1000 (1 second)
    Valid Values:
    Importance:low

  • emit.offset-syncs.enabled

    Whether to store the new offset of the replicated records in offset-syncs topic or not. MirrorCheckpointConnector will not be able to sync group offsets or emit checkpoints if emit.checkpoints.enabled and/or sync.group.offsets.enabled are enabled while emit.offset-syncs.enabled is disabled.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • offset-syncs.topic.location

    The location (source/target) of the offset-syncs topic.

    Type:string
    Default:source
    Valid Values:[source, target]
    Importance:low

  • offset-syncs.topic.replication.factor

    Replication factor for offset-syncs topic.

    Type:short
    Default:3
    Valid Values:
    Importance:low

  • offset.lag.max

    How out-of-sync a remote partition can be before it is resynced.

    Type:long
    Default:100
    Valid Values:
    Importance:low

  • refresh.topics.enabled

    Whether to periodically check for new topics and partitions.

    Type:boolean
    Default:true
    Valid Values:
    Importance:low

  • refresh.topics.interval.seconds

    Frequency of topic refresh.

    Type:long
    Default:600
    Valid Values:
    Importance: