- 1. Getting Started
- 2. APIs
- 3. Configuration
- 4. Design
- 5. Implementation
- 6. Operations
- 6.1 Basic Kafka Operations
- Adding and removing topics
- Modifying topics
- Graceful shutdown
- Balancing leadership
- Balancing Replicas Across Racks
- Mirroring data between clusters
- Checking consumer position
- Managing Consumer Groups
- Expanding your cluster
- Decommissioning brokers
- Increasing replication factor
- Limiting Bandwidth Usage during Data Migration
- Setting quotas
- 6.2 Datacenters
- 6.3 Geo-Replication (Cross-Cluster Data Mirroring)
- 6.4 Multi-Tenancy
- 6.5 Important Configs
- 6.6 Java Version
- 6.7 Hardware and OS
- 6.8 Monitoring
- 6.9 ZooKeeper
- 6.10 KRaft
- 6.11 Tiered Storage
- 6.1 Basic Kafka Operations
- 7. Security
- 8. Kafka Connect
- 9. Kafka Streams
Documentation
Kafka 3.9 Documentation
Prior releases: 0.7.x, 0.8.0, 0.8.1.X, 0.8.2.X, 0.9.0.X, 0.10.0.X, 0.10.1.X, 0.10.2.X, 0.11.0.X, 1.0.X, 1.1.X, 2.0.X, 2.1.X, 2.2.X, 2.3.X, 2.4.X, 2.5.X, 2.6.X, 2.7.X, 2.8.X, 3.0.X, 3.1.X, 3.2.X, 3.3.X, 3.4.X, 3.5.X, 3.6.X, 3.7.X, 3.8.X.1. Getting Started
1.1 Introduction
What is event streaming?
Event streaming is the digital equivalent of the human body's central nervous system. It is the technological foundation for the 'always-on' world where businesses are increasingly software-defined and automated, and where the user of software is more software.
Technically speaking, event streaming is the practice of capturing data in real-time from event sources like databases, sensors, mobile devices, cloud services, and software applications in the form of streams of events; storing these event streams durably for later retrieval; manipulating, processing, and reacting to the event streams in real-time as well as retrospectively; and routing the event streams to different destination technologies as needed. Event streaming thus ensures a continuous flow and interpretation of data so that the right information is at the right place, at the right time.
What can I use event streaming for?
Event streaming is applied to a wide variety of use cases across a plethora of industries and organizations. Its many examples include:
- To process payments and financial transactions in real-time, such as in stock exchanges, banks, and insurances.
- To track and monitor cars, trucks, fleets, and shipments in real-time, such as in logistics and the automotive industry.
- To continuously capture and analyze sensor data from IoT devices or other equipment, such as in factories and wind parks.
- To collect and immediately react to customer interactions and orders, such as in retail, the hotel and travel industry, and mobile applications.
- To monitor patients in hospital care and predict changes in condition to ensure timely treatment in emergencies.
- To connect, store, and make available data produced by different divisions of a company.
- To serve as the foundation for data platforms, event-driven architectures, and microservices.
Apache Kafka® is an event streaming platform. What does that mean?
Kafka combines three key capabilities so you can implement your use cases for event streaming end-to-end with a single battle-tested solution:
- To publish (write) and subscribe to (read) streams of events, including continuous import/export of your data from other systems.
- To store streams of events durably and reliably for as long as you want.
- To process streams of events as they occur or retrospectively.
And all this functionality is provided in a distributed, highly scalable, elastic, fault-tolerant, and secure manner. Kafka can be deployed on bare-metal hardware, virtual machines, and containers, and on-premises as well as in the cloud. You can choose between self-managing your Kafka environments and using fully managed services offered by a variety of vendors.
How does Kafka work in a nutshell?
Kafka is a distributed system consisting of servers and clients that communicate via a high-performance TCP network protocol. It can be deployed on bare-metal hardware, virtual machines, and containers in on-premise as well as cloud environments.
Servers: Kafka is run as a cluster of one or more servers that can span multiple datacenters or cloud regions. Some of these servers form the storage layer, called the brokers. Other servers run Kafka Connect to continuously import and export data as event streams to integrate Kafka with your existing systems such as relational databases as well as other Kafka clusters. To let you implement mission-critical use cases, a Kafka cluster is highly scalable and fault-tolerant: if any of its servers fails, the other servers will take over their work to ensure continuous operations without any data loss.
Clients: They allow you to write distributed applications and microservices that read, write, and process streams of events in parallel, at scale, and in a fault-tolerant manner even in the case of network problems or machine failures. Kafka ships with some such clients included, which are augmented by dozens of clients provided by the Kafka community: clients are available for Java and Scala including the higher-level Kafka Streams library, for Go, Python, C/C++, and many other programming languages as well as REST APIs.
Main Concepts and Terminology
An event records the fact that "something happened" in the world or in your business. It is also called record or message in the documentation. When you read or write data to Kafka, you do this in the form of events. Conceptually, an event has a key, value, timestamp, and optional metadata headers. Here's an example event:
- Event key: "Alice"
- Event value: "Made a payment of $200 to Bob"
- Event timestamp: "Jun. 25, 2020 at 2:06 p.m."
Producers are those client applications that publish (write) events to Kafka, and consumers are those that subscribe to (read and process) these events. In Kafka, producers and consumers are fully decoupled and agnostic of each other, which is a key design element to achieve the high scalability that Kafka is known for. For example, producers never need to wait for consumers. Kafka provides various guarantees such as the ability to process events exactly-once.
Events are organized and durably stored in topics. Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder. An example topic name could be "payments". Topics in Kafka are always multi-producer and multi-subscriber: a topic can have zero, one, or many producers that write events to it, as well as zero, one, or many consumers that subscribe to these events. Events in a topic can be read as often as needed—unlike traditional messaging systems, events are not deleted after consumption. Instead, you define for how long Kafka should retain your events through a per-topic configuration setting, after which old events will be discarded. Kafka's performance is effectively constant with respect to data size, so storing data for a long time is perfectly fine.
Topics are partitioned, meaning a topic is spread over a number of "buckets" located on different Kafka brokers. This distributed placement of your data is very important for scalability because it allows client applications to both read and write the data from/to many brokers at the same time. When a new event is published to a topic, it is actually appended to one of the topic's partitions. Events with the same event key (e.g., a customer or vehicle ID) are written to the same partition, and Kafka guarantees that any consumer of a given topic-partition will always read that partition's events in exactly the same order as they were written.
To make your data fault-tolerant and highly-available, every topic can be replicated, even across geo-regions or datacenters, so that there are always multiple brokers that have a copy of the data just in case things go wrong, you want to do maintenance on the brokers, and so on. A common production setting is a replication factor of 3, i.e., there will always be three copies of your data. This replication is performed at the level of topic-partitions.
This primer should be sufficient for an introduction. The Design section of the documentation explains Kafka's various concepts in full detail, if you are interested.
Kafka APIs
In addition to command line tooling for management and administration tasks, Kafka has five core APIs for Java and Scala:
- The Admin API to manage and inspect topics, brokers, and other Kafka objects.
- The Producer API to publish (write) a stream of events to one or more Kafka topics.
- The Consumer API to subscribe to (read) one or more topics and to process the stream of events produced to them.
- The Kafka Streams API to implement stream processing applications and microservices. It provides higher-level functions to process event streams, including transformations, stateful operations like aggregations and joins, windowing, processing based on event-time, and more. Input is read from one or more topics in order to generate output to one or more topics, effectively transforming the input streams to output streams.
- The Kafka Connect API to build and run reusable data import/export connectors that consume (read) or produce (write) streams of events from and to external systems and applications so they can integrate with Kafka. For example, a connector to a relational database like PostgreSQL might capture every change to a set of tables. However, in practice, you typically don't need to implement your own connectors because the Kafka community already provides hundreds of ready-to-use connectors.
Where to go from here
- To get hands-on experience with Kafka, follow the Quickstart.
- To understand Kafka in more detail, read the Documentation. You also have your choice of Kafka books and academic papers.
- Browse through the Use Cases to learn how other users in our world-wide community are getting value out of Kafka.
- Join a local Kafka meetup group and watch talks from Kafka Summit, the main conference of the Kafka community.
1.2 Use Cases
Here is a description of a few of the popular use cases for Apache Kafka®. For an overview of a number of these areas in action, see this blog post.
Messaging
Kafka works well as a replacement for a more traditional message broker. Message brokers are used for a variety of reasons (to decouple processing from data producers, to buffer unprocessed messages, etc). In comparison to most messaging systems Kafka has better throughput, built-in partitioning, replication, and fault-tolerance which makes it a good solution for large scale message processing applications.In our experience messaging uses are often comparatively low-throughput, but may require low end-to-end latency and often depend on the strong durability guarantees Kafka provides.
In this domain Kafka is comparable to traditional messaging systems such as ActiveMQ or RabbitMQ.
Website Activity Tracking
The original use case for Kafka was to be able to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds. This means site activity (page views, searches, or other actions users may take) is published to central topics with one topic per activity type. These feeds are available for subscription for a range of use cases including real-time processing, real-time monitoring, and loading into Hadoop or offline data warehousing systems for offline processing and reporting.Activity tracking is often very high volume as many activity messages are generated for each user page view.
Metrics
Kafka is often used for operational monitoring data. This involves aggregating statistics from distributed applications to produce centralized feeds of operational data.Log Aggregation
Many people use Kafka as a replacement for a log aggregation solution. Log aggregation typically collects physical log files off servers and puts them in a central place (a file server or HDFS perhaps) for processing. Kafka abstracts away the details of files and gives a cleaner abstraction of log or event data as a stream of messages. This allows for lower-latency processing and easier support for multiple data sources and distributed data consumption. In comparison to log-centric systems like Scribe or Flume, Kafka offers equally good performance, stronger durability guarantees due to replication, and much lower end-to-end latency.Stream Processing
Many users of Kafka process data in processing pipelines consisting of multiple stages, where raw input data is consumed from Kafka topics and then aggregated, enriched, or otherwise transformed into new topics for further consumption or follow-up processing. For example, a processing pipeline for recommending news articles might crawl article content from RSS feeds and publish it to an "articles" topic; further processing might normalize or deduplicate this content and publish the cleansed article content to a new topic; a final processing stage might attempt to recommend this content to users. Such processing pipelines create graphs of real-time data flows based on the individual topics. Starting in 0.10.0.0, a light-weight but powerful stream processing library called Kafka Streams is available in Apache Kafka to perform such data processing as described above. Apart from Kafka Streams, alternative open source stream processing tools include Apache Storm and Apache Samza.Event Sourcing
Event sourcing is a style of application design where state changes are logged as a time-ordered sequence of records. Kafka's support for very large stored log data makes it an excellent backend for an application built in this style.Commit Log
Kafka can serve as a kind of external commit-log for a distributed system. The log helps replicate data between nodes and acts as a re-syncing mechanism for failed nodes to restore their data. The log compaction feature in Kafka helps support this usage. In this usage Kafka is similar to Apache BookKeeper project.1.3 Quick Start
Step 1: Get Kafka
Download the latest Kafka release and extract it:
$ tar -xzf kafka_2.13-3.9.0.tgz
$ cd kafka_2.13-3.9.0
Step 2: Start the Kafka environment
NOTE: Your local environment must have Java 8+ installed.
Apache Kafka can be started using KRaft or ZooKeeper. To get started with either configuration follow one of the sections below but not both.
Kafka with KRaft
Kafka can be run using KRaft mode using local scripts and downloaded files or the docker image. Follow one of the sections below but not both to start the kafka server.
Using downloaded files
Generate a Cluster UUID
$ KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
Format Log Directories
$ bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/kraft/reconfig-server.properties
Start the Kafka Server
$ bin/kafka-server-start.sh config/kraft/reconfig-server.properties
Once the Kafka server has successfully launched, you will have a basic Kafka environment running and ready to use.
Using JVM Based Apache Kafka Docker Image
Get the Docker image:
$ docker pull apache/kafka:3.9.0
Start the Kafka Docker container:
$ docker run -p 9092:9092 apache/kafka:3.9.0
Using GraalVM Based Native Apache Kafka Docker Image
Get the Docker image:
$ docker pull apache/kafka-native:3.9.0
Start the Kafka Docker container:
$ docker run -p 9092:9092 apache/kafka-native:3.9.0
Kafka with ZooKeeper
Run the following commands in order to start all services in the correct order:
# Start the ZooKeeper service
$ bin/zookeeper-server-start.sh config/zookeeper.properties
Open another terminal session and run:
# Start the Kafka broker service
$ bin/kafka-server-start.sh config/server.properties
Once all services have successfully launched, you will have a basic Kafka environment running and ready to use.
Step 3: Create a topic to store your events
Kafka is a distributed event streaming platform that lets you read, write, store, and process events (also called records or messages in the documentation) across many machines.
Example events are payment transactions, geolocation updates from mobile phones, shipping orders, sensor measurements from IoT devices or medical equipment, and much more. These events are organized and stored in topics. Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder.
So before you can write your first events, you must create a topic. Open another terminal session and run:
$ bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
All of Kafka's command line tools have additional options: run the kafka-topics.sh
command without any
arguments to display usage information. For example, it can also show you
details such as the partition count
of the new topic:
$ bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
Topic: quickstart-events TopicId: NPmZHyhbR9y00wMglMH2sg PartitionCount: 1 ReplicationFactor: 1 Configs:
Topic: quickstart-events Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Step 4: Write some events into the topic
A Kafka client communicates with the Kafka brokers via the network for writing (or reading) events. Once received, the brokers will store the events in a durable and fault-tolerant manner for as long as you need—even forever.
Run the console producer client to write a few events into your topic. By default, each line you enter will result in a separate event being written to the topic.
$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
>This is my first event
>This is my second event
You can stop the producer client with Ctrl-C
at any time.
Step 5: Read the events
Open another terminal session and run the console consumer client to read the events you just created:
$ bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
This is my first event
This is my second event
You can stop the consumer client with Ctrl-C
at any time.
Feel free to experiment: for example, switch back to your producer terminal (previous step) to write additional events, and see how the events immediately show up in your consumer terminal.
Because events are durably stored in Kafka, they can be read as many times and by as many consumers as you want. You can easily verify this by opening yet another terminal session and re-running the previous command again.
Step 6: Import/export your data as streams of events with Kafka Connect
You probably have lots of data in existing systems like relational databases or traditional messaging systems, along with many applications that already use these systems. Kafka Connect allows you to continuously ingest data from external systems into Kafka, and vice versa. It is an extensible tool that runs connectors, which implement the custom logic for interacting with an external system. It is thus very easy to integrate existing systems with Kafka. To make this process even easier, there are hundreds of such connectors readily available.
In this quickstart we'll see how to run Kafka Connect with simple connectors that import data from a file to a Kafka topic and export data from a Kafka topic to a file.
First, make sure to add connect-file-3.9.0.jar
to the plugin.path
property in the Connect worker's configuration.
For the purpose of this quickstart we'll use a relative path and consider the connectors' package as an uber jar, which works when the quickstart commands are run from the installation directory.
However, it's worth noting that for production deployments using absolute paths is always preferable. See plugin.path for a detailed description of how to set this config.
Edit the config/connect-standalone.properties
file, add or change the plugin.path
configuration property match the following, and save the file:
$ echo "plugin.path=libs/connect-file-3.9.0.jar" >> config/connect-standalone.properties
Then, start by creating some seed data to test with:
$ echo -e "foo\nbar" > test.txt
Or on Windows:
$ echo foo > test.txt
$ echo bar >> test.txt
Next, we'll start two connectors running in standalone mode, which means they run in a single, local, dedicated process. We provide three configuration files as parameters. The first is always the configuration for the Kafka Connect process, containing common configuration such as the Kafka brokers to connect to and the serialization format for data. The remaining configuration files each specify a connector to create. These files include a unique connector name, the connector class to instantiate, and any other configuration required by the connector.
$ bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties
These sample configuration files, included with Kafka, use the default local cluster configuration you started earlier and create two connectors: the first is a source connector that reads lines from an input file and produces each to a Kafka topic and the second is a sink connector that reads messages from a Kafka topic and produces each as a line in an output file.
During startup you'll see a number of log messages, including some indicating that the connectors are being instantiated.
Once the Kafka Connect process has started, the source connector should start reading lines from test.txt
and
producing them to the topic connect-test
, and the sink connector should start reading messages from the topic connect-test
and write them to the file test.sink.txt
. We can verify the data has been delivered through the entire pipeline
by examining the contents of the output file:
$ more test.sink.txt
foo
bar
Note that the data is being stored in the Kafka topic connect-test
, so we can also run a console consumer to see the
data in the topic (or use custom consumer code to process it):
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic connect-test --from-beginning
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
…
The connectors continue to process data, so we can add data to the file and see it move through the pipeline:
$ echo "Another line" >> test.txt
You should see the line appear in the console consumer output and in the sink file.
Step 7: Process your events with Kafka Streams
Once your data is stored in Kafka as events, you can process the data with the Kafka Streams client library for Java/Scala. It allows you to implement mission-critical real-time applications and microservices, where the input and/or output data is stored in Kafka topics. Kafka Streams combines the simplicity of writing and deploying standard Java and Scala applications on the client side with the benefits of Kafka's server-side cluster technology to make these applications highly scalable, elastic, fault-tolerant, and distributed. The library supports exactly-once processing, stateful operations and aggregations, windowing, joins, processing based on event-time, and much more.
To give you a first taste, here's how one would implement the popular WordCount
algorithm:
KStream<String, String> textLines = builder.stream("quickstart-events");
KTable<String, Long> wordCounts = textLines
.flatMapValues(line -> Arrays.asList(line.toLowerCase().split(" ")))
.groupBy((keyIgnored, word) -> word)
.count();
wordCounts.toStream().to("output-topic", Produced.with(Serdes.String(), Serdes.Long()));
The Kafka Streams demo and the app development tutorial demonstrate how to code and run such a streaming application from start to finish.
Step 8: Terminate the Kafka environment
Now that you reached the end of the quickstart, feel free to tear down the Kafka environment—or continue playing around.
-
Stop the producer and consumer clients with
Ctrl-C
, if you haven't done so already. -
Stop the Kafka broker with
Ctrl-C
. -
Lastly, if the Kafka with ZooKeeper section was followed, stop the ZooKeeper server with
Ctrl-C
.
If you also want to delete any data of your local Kafka environment including any events you have created along the way, run the command:
$ rm -rf /tmp/kafka-logs /tmp/zookeeper /tmp/kraft-combined-logs
Congratulations!
You have successfully finished the Apache Kafka quickstart.
To learn more, we suggest the following next steps:
- Read through the brief Introduction to learn how Kafka works at a high level, its main concepts, and how it compares to other technologies. To understand Kafka in more detail, head over to the Documentation.
- Browse through the Use Cases to learn how other users in our world-wide community are getting value out of Kafka.
- Join a local Kafka meetup group and watch talks from Kafka Summit, the main conference of the Kafka community.
1.4 Ecosystem
There are a plethora of tools that integrate with Kafka outside the main distribution. The ecosystem page lists many of these, including stream processing systems, Hadoop integration, monitoring, and deployment tools.1.5 Upgrading From Previous Versions
Upgrading to 3.9.0 from any version 0.8.x through 3.8.x
Upgrading ZooKeeper-based clusters
If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.8
,3.7
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.8
,3.7
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to3.9
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.8 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Upgrading KRaft-based clusters
If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.
For a rolling upgrade:
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
- Once the cluster's behavior and performance has been verified, bump the metadata.version by running
bin/kafka-features.sh upgrade --metadata 3.9
- Note that cluster metadata downgrade is not supported in this software version.
Every MetadataVersion
after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e.
IBP_3_3_IV3(7, "3.3", "IV3", true)
means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.9.0
- In case you run your Kafka clusters with no execution permission for the
/tmp
partition, Kafka will not work properly. It might either refuse to start or fail when producing and consuming messages. This is due to the compression librarieszstd-jni
andsnappy
. To remediate this problem you need to pass the following JVM flags to KafkaZstdTempFolder
andorg.xerial.snappy.tempdir
pointing to a directory with execution permissions. For example, this could be done via theKAFKA_OPTS
environment variable like follows:export KAFKA_OPTS="-DZstdTempFolder=/opt/kafka/tmp -Dorg.xerial.snappy.tempdir=/opt/kafka/tmp"
. This is a known issue for version 3.8.0 as well. unclean.leader.election.enable
config is supported in KRaft. Compared with ZK mode, there is one behavior change in KRaft mode when dynamically enablingunclean.leader.election.enable
config. Please check here for more details.- Tiered storage is now a production ready feature. You can check
Kafka Tiered Storage GA Release Notes for more details.
The below enhancements are added in this release.
- In KRaft mode, the tiered storage feature can be dynamically disabled and then re-enabled on topic level. See KIP-950 for more details.
- With the tiered storage quota feature, users can define a maximum limit on the rate at which log segments are transferred to or retrieved from the remote storage. See KIP-956 for more details.
- Controller membership change in KRaft is now supported when formatting with `--standalone` or `--initial-controllers` option. See here for more details.
Upgrading to 3.8.1 from any version 0.8.x through 3.7.x
Upgrading ZooKeeper-based clusters
If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.7
,3.6
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.7
,3.6
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to3.8
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.8 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Upgrading KRaft-based clusters
If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.
For a rolling upgrade:
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
- Once the cluster's behavior and performance has been verified, bump the metadata.version by running
bin/kafka-features.sh upgrade --metadata 3.8
- Note that cluster metadata downgrade is not supported in this version since it has metadata changes.
Every MetadataVersion
after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e.
IBP_3_3_IV3(7, "3.3", "IV3", true)
means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.8.1
- In case you run your Kafka clusters with no execution permission for the
/tmp
partition, Kafka will not work properly. It might either refuse to start or fail when producing and consuming messages. This is due to the compression librarieszstd-jni
andsnappy
. To remediate this problem you need to pass the following JVM flags to KafkaZstdTempFolder
andorg.xerial.snappy.tempdir
pointing to a directory with execution permissions. For example, this could be done via theKAFKA_OPTS
environment variable like follows:export KAFKA_OPTS="-DZstdTempFolder=/opt/kafka/tmp -Dorg.xerial.snappy.tempdir=/opt/kafka/tmp"
. This is a known issue for version 3.8.0. - In 3.8.0 the
kafka.utils.Thottler
metric was accidentally renamed toorg.apache.kafka.storage.internals.utils.Throttler
. This change has been reverted and the metric is now namedkafka.utils.Thottler
again.
Upgrading to 3.8.0 from any version 0.8.x through 3.7.x
Notable changes in 3.8.0
- MirrorMaker 2 can now emit checkpoints for offsets mirrored before the start of the Checkpoint task for improved offset translation. This requires MirrorMaker 2 to have READ authorization to the Checkpoint topic. If READ is not authorized, checkpointing is limited to offsets mirrorred after the start of the task. See KAFKA-15905 for more details.
- JBOD in KRaft is no longer in early access.
- Tiered Storage, which is still in early access, now supports clusters configured with multiple log directories (i.e. JBOD feature).
Upgrading to 3.7.1 from any version 0.8.x through 3.6.x
Upgrading ZooKeeper-based clusters
If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.6
,3.5
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.6
,3.5
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to3.7
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.7 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Upgrading KRaft-based clusters
If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.
For a rolling upgrade:
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
- Once the cluster's behavior and performance has been verified, bump the metadata.version by running
bin/kafka-features.sh upgrade --metadata 3.7
- Note that cluster metadata downgrade is not supported in this version since it has metadata changes.
Every MetadataVersion
after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e.
IBP_3_3_IV3(7, "3.3", "IV3", true)
means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.7.1
- MirrorMaker 2 can now emit checkpoints for offsets mirrored before the start of the Checkpoint task for improved offset translation. This requires MirrorMaker 2 to have READ authorization to the Checkpoint topic. If READ is not authorized, checkpointing is limited to offsets mirrorred after the start of the task. See KAFKA-15905 for more details.
- JBOD support in KRaft was introduced from Metadata Version (MV) 3.7-IV2. Configuring Brokers with multiple log directories can lead to indefinite unavailability. Brokers will now detect this situation and log an error. See KAFKA-16606 for more details.
Notable changes in 3.7.0
- Java 11 support for the broker and tools has been deprecated and will be removed in Apache Kafka 4.0. This complements the previous deprecation of Java 8 for all components. Please refer to KIP-1013 for more details.
- Client APIs released prior to Apache Kafka 2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. See KIP-896 for details and RPC versions that are now deprecated.
- Early access of the new simplified Consumer Rebalance Protocol is available, and it is not recommended for use in production environments. You are encouraged to test it and provide feedback! For more information about the early access feature, please check KIP-848 and the Early Access Release Notes.
- More metrics related to Tiered Storage have been introduced. They should improve the operational experience of running Tiered Storage in production. For more detailed information, please refer to KIP-963.
- Kafka Streams ships multiple KIPs for IQv2 support. See the Kafka Streams upgrade section for more details.
- In versions 3.5.0, 3.5.1, 3.5.2, 3.6.0, and 3.6.1, MirrorMaker 2 offset translation may not reach the end of a replicated topic after the upstream consumers commit at the end of the source topic. This was addressed in KAFKA-15906.
- All the notable changes are present in the blog post announcing the 3.7.0 release.
Upgrading to 3.6.2 from any version 0.8.x through 3.5.x
Upgrading ZooKeeper-based clusters
If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.5
,3.4
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.5
,3.4
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to3.6
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.6 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Upgrading KRaft-based clusters
If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.
For a rolling upgrade:
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
- Once the cluster's behavior and performance has been verified, bump the metadata.version by running
bin/kafka-features.sh upgrade --metadata 3.6
- Note that cluster metadata downgrade is not supported in this version since it has metadata changes.
Every MetadataVersion
after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e.
IBP_3_3_IV3(7, "3.3", "IV3", true)
means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.6.0
- Apache Kafka now supports having both an IPv4 and an IPv6 listener on the same port. This change only applies to non advertised listeners (advertised listeners already have this feature)
- The Apache Zookeeper dependency has been upgraded to 3.8.1 due to 3.6 reaching end-of-life. To bring both your
Kafka and Zookeeper clusters to the latest versions:
- >=2.4 Kafka clusters can be updated directly. Zookeeper clusters which are running binaries bundled with Kafka versions 2.4 or above can be updated directly.
- <2.4 Kafka clusters first need to be updated to a version greater than 2.4 and smaller than 3.6. Zookeeper clusters which are running binaries bundled with Kafka versions below 2.4 need to be updated to any binaries bundled with Kafka versions greater than 2.4 and smaller than 3.6. You can then follow the first bullet-point.
- The configuration
log.message.timestamp.difference.max.ms
is deprecated. Two new configurations,log.message.timestamp.before.max.ms
andlog.message.timestamp.after.max.ms
, have been added. For more detailed information, please refer to KIP-937. -
Kafka Streams has introduced a new task assignor,
RackAwareTaskAssignor
, for computing task assignments which can minimize cross rack traffic under certain conditions. It works with existingStickyTaskAssignor
andHighAvailabilityTaskAssignor
. See KIP-925 and Kafka Streams Developer Guide for more details. - To account for a break in compatibility introduced in version 3.1.0, MirrorMaker 2 has added a new
replication.policy.internal.topic.separator.enabled
property. If upgrading from 3.0.x or earlier, it may be necessary to set this property tofalse
; see the property's documentation for more details. - Early access of tiered storage feature is available, and it is not recommended for use in production environments. Welcome to test it and provide any feedback to us. For more information about the early access tiered storage feature, please check KIP-405 and Tiered Storage Early Access Release Note.
- Transaction partition verification (KIP-890)
has been added to data partitions to prevent hanging transactions. This feature is enabled by default and can be disabled by setting
transaction.partition.verification.enable
to false. The configuration can also be updated dynamically and is applied to the broker. Workloads running on version 3.6.0 with compression can experience InvalidRecordExceptions and UnknownServerExceptions. Upgrading to 3.6.1 or newer or disabling the feature fixes the issue.
Upgrading to 3.5.2 from any version 0.8.x through 3.4.x
All upgrade steps remain same as upgrading to 3.5.0Notable changes in 3.5.2
- When migrating producer ID blocks from ZK to KRaft, there could be duplicate producer IDs being given to transactional or idempotent producers. This can cause long term problems since the producer IDs are persisted and reused for a long time. See KAFKA-15552 for more details.
- In 3.5.0 and 3.5.1, there could be an issue that the empty ISR is returned from controller after AlterPartition request during rolling upgrade. This issue will impact the availability of the topic partition. See KAFKA-15353 for more details.
- In 3.5.0 and 3.5.1, there was an issue where MirrorMaker 2 offset translation produced an earlier offset than needed, substantially increasing the re-delivery of data when starting a consumer from the downstream consumer offsets. See KAFKA-15202 for more details.
Upgrading to 3.5.1 from any version 0.8.x through 3.4.x
All upgrade steps remain same as upgrading to 3.5.0Notable changes in 3.5.1
- Upgraded the dependency, snappy-java, to a version which is not vulnerable to CVE-2023-34455. You can find more information about the CVE at Kafka CVE list.
-
Fixed a regression introduced in 3.3.0, which caused
security.protocol
configuration values to be restricted to upper case only. After the fix,security.protocol
values are case insensitive. See KAFKA-15053 for details.
Upgrading to 3.5.0 from any version 0.8.x through 3.4.x
Upgrading ZooKeeper-based clusters
If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.4
,3.3
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.4
,3.3
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to3.5
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.5 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Upgrading KRaft-based clusters
If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.
For a rolling upgrade:
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
- Once the cluster's behavior and performance has been verified, bump the metadata.version by running
bin/kafka-features.sh upgrade --metadata 3.5
- Note that cluster metadata downgrade is not supported in this version since it has metadata changes.
Every MetadataVersion
after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e.
IBP_3_3_IV3(7, "3.3", "IV3", true)
means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.5.0
- Kafka Streams has introduced a new state store type, versioned key-value stores, for storing multiple record versions per key, thereby enabling timestamped retrieval operations to return the latest record (per key) as of a specified timestamp. See KIP-889 and KIP-914 for more details. If the new store typed is used in the DSL, improved processing semantics are applied as described in KIP-914.
- KTable aggregation semantics got further improved via KIP-904, now avoiding spurious intermediate results.
- Kafka Streams'
ProductionExceptionHandler
is improved via KIP-399, now also covering serialization errors. - MirrorMaker now uses incrementalAlterConfigs API by default to synchronize topic configurations instead of the deprecated alterConfigs API.
A new settings called
use.incremental.alter.configs
is introduced to allow users to control which API to use. This new setting is marked deprecated and will be removed in the next major release when incrementalAlterConfigs API is always used. See KIP-894 for more details. - The JmxTool, EndToEndLatency, StreamsResetter, ConsumerPerformance and ClusterTool have been migrated to the tools module. The 'kafka.tools' package is deprecated and will change to 'org.apache.kafka.tools' in the next major release. See KAFKA-14525 for more details.
- In versions earlier than 3.5.0 and 3.4.1, MirrorMaker 2 offset translation could incorrectly translate offsets for topics using compaction, transactional producers, and filter SMTs. In 3.5.0 and 3.4.1, offset translation has changed for all topics in order to ensure at-least-once delivery when a consumer is failed-over to the translated offsets. Translated offsets will be earlier than in previous versions, so consumers using downstream offsets may initially have more lag, and re-deliver more data after failing-over. See KAFKA-12468 and linked tickets, and PR #13178 for more details. Further improvements to the offset translation are included in later releases to reduce the lag introduced by this change, so consider upgrading MM2 to the latest version available.
Upgrading to 3.4.0 from any version 0.8.x through 3.3.x
If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.3
,3.2
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.3
,3.2
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to3.4
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.4 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Upgrading a KRaft-based cluster to 3.4.0 from any version 3.0.x through 3.3.x
If you are upgrading from a version prior to 3.3.0, please see the note below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.
For a rolling upgrade:
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
- Once the cluster's behavior and performance has been verified, bump the metadata.version by running
bin/kafka-features.sh upgrade --metadata 3.4
- Note that cluster metadata downgrade is not supported in this version since it has metadata changes.
Every MetadataVersion
after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e.
IBP_3_3_IV3(7, "3.3", "IV3", true)
means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.4.0
- Since Apache Kafka 3.4.0, we have added a system property ("org.apache.kafka.disallowed.login.modules") to disable the problematic login modules usage in SASL JAAS configuration. Also by default "com.sun.security.auth.module.JndiLoginModule" is disabled from Apache Kafka 3.4.0.
Upgrading to 3.3.1 from any version 0.8.x through 3.2.x
If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.2
,3.1
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.2
,3.1
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to3.3
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.3 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Upgrading a KRaft-based cluster to 3.3.1 from any version 3.0.x through 3.2.x
If you are upgrading from a version prior to 3.3.1, please see the note below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.
For a rolling upgrade:
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
- Once the cluster's behavior and performance has been verified, bump the metadata.version by running
bin/kafka-features.sh upgrade --metadata 3.3
- Note that cluster metadata downgrade is not supported in this version since it has metadata changes.
Every MetadataVersion
after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e.
IBP_3_3_IV3(7, "3.3", "IV3", true)
means this version has metadata changes). Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.
Notable changes in 3.3.1
- KRaft mode is production ready for new clusters. See KIP-833 for more details (including limitations).
- The partitioner used by default for records with no keys has been improved to avoid pathological behavior when one or more brokers are slow.
The new logic may affect the batching behavior, which can be tuned using the
batch.size
and/orlinger.ms
configuration settings. The previous behavior can be restored by settingpartitioner.class=org.apache.kafka.clients.producer.internals.DefaultPartitioner
. See KIP-794 for more details. - There is now a slightly different upgrade process for KRaft clusters than for ZK-based clusters, as described above.
- Introduced a new API
addMetricIfAbsent
toMetrics
which would create a new Metric if not existing or return the same metric if already registered. Note that this behaviour is different fromaddMetric
API which throws anIllegalArgumentException
when trying to create an already existing metric. (See KIP-843 for more details).
Upgrading to 3.2.0 from any version 0.8.x through 3.1.x
If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.1
,3.0
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.1
,3.0
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to3.2
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.2 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 3.2.0
- Idempotence for the producer is enabled by default if no conflicting configurations are set. When producing to brokers older than 2.8.0,
the
IDEMPOTENT_WRITE
permission is required. Check the compatibility section of KIP-679 for details. In 3.0.0 and 3.1.0, a bug prevented this default from being applied, which meant that idempotence remained disabled unless the user had explicitly setenable.idempotence
to true (See KAFKA-13598 for more details). This issue was fixed and the default is properly applied in 3.0.1, 3.1.1, and 3.2.0. - A notable exception is Connect that by default disables idempotent behavior for all of its producers in order to uniformly support using a wide range of Kafka broker versions. Users can change this behavior to enable idempotence for some or all producers via Connect worker and/or connector configuration. Connect may enable idempotent producers by default in a future major release.
- Kafka has replaced log4j with reload4j due to security concerns.
This only affects modules that specify a logging backend (
connect-runtime
andkafka-tools
are two such examples). A number of modules, includingkafka-clients
, leave it to the application to specify the logging backend. More information can be found at reload4j. Projects that depend on the affected modules from the Kafka project should use slf4j-log4j12 version 1.7.35 or above or slf4j-reload4j to avoid possible compatibility issues originating from the logging framework. - The example connectors,
FileStreamSourceConnector
andFileStreamSinkConnector
, have been removed from the default classpath. To use them in Kafka Connect standalone or distributed mode they need to be explicitly added, for exampleCLASSPATH=./libs/connect-file-3.2.0.jar bin/connect-distributed.sh
.
Upgrading to 3.1.0 from any version 0.8.x through 3.0.x
If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.0
,2.8
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
3.0
,2.8
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to3.1
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.1 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 3.1.1
- Idempotence for the producer is enabled by default if no conflicting configurations are set. When producing to brokers older than 2.8.0,
the
IDEMPOTENT_WRITE
permission is required. Check the compatibility section of KIP-679 for details. A bug prevented the producer idempotence default from being applied which meant that it remained disabled unless the user had explicitly setenable.idempotence
to true. See KAFKA-13598 for more details. This issue was fixed and the default is properly applied. - A notable exception is Connect that by default disables idempotent behavior for all of its producers in order to uniformly support using a wide range of Kafka broker versions. Users can change this behavior to enable idempotence for some or all producers via Connect worker and/or connector configuration. Connect may enable idempotent producers by default in a future major release.
- Kafka has replaced log4j with reload4j due to security concerns.
This only affects modules that specify a logging backend (
connect-runtime
andkafka-tools
are two such examples). A number of modules, includingkafka-clients
, leave it to the application to specify the logging backend. More information can be found at reload4j. Projects that depend on the affected modules from the Kafka project should use slf4j-log4j12 version 1.7.35 or above or slf4j-reload4j to avoid possible compatibility issues originating from the logging framework.
Notable changes in 3.1.0
- Apache Kafka supports Java 17.
- The following metrics have been deprecated:
bufferpool-wait-time-total
,io-waittime-total
, andiotime-total
. Please usebufferpool-wait-time-ns-total
,io-wait-time-ns-total
, andio-time-ns-total
instead. See KIP-773 for more details. - IBP 3.1 introduces topic IDs to FetchRequest as a part of KIP-516.
Upgrading to 3.0.1 from any version 0.8.x through 2.8.x
If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
2.8
,2.7
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
2.8
,2.7
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to3.0
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.0 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 3.0.1
- Idempotence for the producer is enabled by default if no conflicting configurations are set. When producing to brokers older than 2.8.0,
the
IDEMPOTENT_WRITE
permission is required. Check the compatibility section of KIP-679 for details. A bug prevented the producer idempotence default from being applied which meant that it remained disabled unless the user had explicitly setenable.idempotence
to true. See KAFKA-13598 for more details. This issue was fixed and the default is properly applied.
Notable changes in 3.0.0
- The producer has stronger delivery guarantees by default:
idempotence
is enabled andacks
is set toall
instead of1
. See KIP-679 for details. In 3.0.0 and 3.1.0, a bug prevented the idempotence default from being applied which meant that it remained disabled unless the user had explicitly setenable.idempotence
to true. Note that the bug did not affect theacks=all
change. See KAFKA-13598 for more details. This issue was fixed and the default is properly applied in 3.0.1, 3.1.1, and 3.2.0. - Java 8 and Scala 2.12 support have been deprecated since Apache Kafka 3.0 and will be removed in Apache Kafka 4.0. See KIP-750 and KIP-751 for more details.
- ZooKeeper has been upgraded to version 3.6.3.
- A preview of KRaft mode is available, though upgrading to it from the 2.8 Early Access release is not possible. See the KRaft section for details.
- The release tarball no longer includes test, sources, javadoc and test sources jars. These are still published to the Maven Central repository.
- A number of implementation dependency jars are now available in the runtime classpath instead of compile and runtime classpaths. Compilation errors after the upgrade can be fixed by adding the missing dependency jar(s) explicitly or updating the application not to use internal classes.
- The default value for the consumer configuration
session.timeout.ms
was increased from 10s to 45s. See KIP-735 for more details. - The broker configuration
log.message.format.version
and topic configurationmessage.format.version
have been deprecated. The value of both configurations is always assumed to be3.0
ifinter.broker.protocol.version
is3.0
or higher. Iflog.message.format.version
ormessage.format.version
are set, we recommend clearing them at the same time as theinter.broker.protocol.version
upgrade to 3.0. This will avoid potential compatibility issues if theinter.broker.protocol.version
is downgraded. See KIP-724 for more details. - The Streams API removed all deprecated APIs that were deprecated in version 2.5.0 or earlier. For a complete list of removed APIs compare the detailed Kafka Streams upgrade notes.
- Kafka Streams no longer has a compile time dependency on "connect:json" module (KAFKA-5146). Projects that were relying on this transitive dependency will have to explicitly declare it.
- Custom principal builder implementations specified through
principal.builder.class
must now implement theKafkaPrincipalSerde
interface to allow for forwarding between brokers. See KIP-590 for more details about the usage of KafkaPrincipalSerde. - A number of deprecated classes, methods and tools have been removed from the
clients
,connect
,core
andtools
modules: - The Scala
Authorizer
,SimpleAclAuthorizer
and related classes have been removed. Please use the JavaAuthorizer
andAclAuthorizer
instead. - The
Metric#value()
method was removed (KAFKA-12573). - The
Sum
andTotal
classes were removed (KAFKA-12584). Please useWindowedSum
andCumulativeSum
instead. - The
Count
andSampledTotal
classes were removed. Please useWindowedCount
andWindowedSum
respectively instead. - The
PrincipalBuilder
,DefaultPrincipalBuilder
andResourceFilter
classes were removed. - Various constants and constructors were removed from
SslConfigs
,SaslConfigs
,AclBinding
andAclBindingFilter
. - The
Admin.electedPreferredLeaders()
methods were removed. Please useAdmin.electLeaders
instead. - The
kafka-preferred-replica-election
command line tool was removed. Please usekafka-leader-election
instead. - The
--zookeeper
option was removed from thekafka-topics
andkafka-reassign-partitions
command line tools. Please use--bootstrap-server
instead. - In the
kafka-configs
command line tool, the--zookeeper
option is only supported for updating SCRAM Credentials configuration and describing/updating dynamic broker configs when brokers are not running. Please use--bootstrap-server
for other configuration operations. - The
ConfigEntry
constructor was removed (KAFKA-12577). Please use the remaining public constructor instead. - The config value
default
for the client configclient.dns.lookup
has been removed. In the unlikely event that you set this config explicitly, we recommend leaving the config unset (use_all_dns_ips
is used by default). - The
ExtendedDeserializer
andExtendedSerializer
classes have been removed. Please useDeserializer
andSerializer
instead. - The
close(long, TimeUnit)
method was removed from the producer, consumer and admin client. Please useclose(Duration)
. - The
ConsumerConfig.addDeserializerToConfig
andProducerConfig.addSerializerToConfig
methods were removed. These methods were not intended to be public API and there is no replacement. - The
NoOffsetForPartitionException.partition()
method was removed. Please usepartitions()
instead. - The default
partition.assignment.strategy
is changed to "[RangeAssignor, CooperativeStickyAssignor]", which will use the RangeAssignor by default, but allows upgrading to the CooperativeStickyAssignor with just a single rolling bounce that removes the RangeAssignor from the list. Please check the client upgrade path guide here for more detail. - The Scala
kafka.common.MessageFormatter
was removed. Please use the Javaorg.apache.kafka.common.MessageFormatter
. - The
MessageFormatter.init(Properties)
method was removed. Please useconfigure(Map)
instead. - The
checksum()
method has been removed fromConsumerRecord
andRecordMetadata
. The message format v2, which has been the default since 0.11, moved the checksum from the record to the record batch. As such, these methods don't make sense and no replacements exist. - The
ChecksumMessageFormatter
class was removed. It is not part of the public API, but it may have been used withkafka-console-consumer.sh
. It reported the checksum of each record, which has not been supported since message format v2. - The
org.apache.kafka.clients.consumer.internals.PartitionAssignor
class has been removed. Please useorg.apache.kafka.clients.consumer.ConsumerPartitionAssignor
instead. - The
quota.producer.default
andquota.consumer.default
configurations were removed (KAFKA-12591). Dynamic quota defaults must be used instead. - The
port
andhost.name
configurations were removed. Please uselisteners
instead. - The
advertised.port
andadvertised.host.name
configurations were removed. Please useadvertised.listeners
instead. - The deprecated worker configurations
rest.host.name
andrest.port
were removed (KAFKA-12482) from the Kafka Connect worker configuration. Please uselisteners
instead. - The
Producer#sendOffsetsToTransaction(Map offsets, String consumerGroupId)
method has been deprecated. Please useProducer#sendOffsetsToTransaction(Map offsets, ConsumerGroupMetadata metadata)
instead, where theConsumerGroupMetadata
can be retrieved viaKafkaConsumer#groupMetadata()
for stronger semantics. Note that the full set of consumer group metadata is only understood by brokers or version 2.5 or higher, so you must upgrade your kafka cluster to get the stronger semantics. Otherwise, you can just pass innew ConsumerGroupMetadata(consumerGroupId)
to work with older brokers. See KIP-732 for more details. -
The Connect
internal.key.converter
andinternal.value.converter
properties have been completely removed. The use of these Connect worker properties has been deprecated since version 2.0.0. Workers are now hardcoded to use the JSON converter withschemas.enable
set tofalse
. If your cluster has been using a different internal key or value converter, you can follow the migration steps outlined in KIP-738 to safely upgrade your Connect cluster to 3.0. - The Connect-based MirrorMaker (MM2) includes changes to support
IdentityReplicationPolicy
, enabling replication without renaming topics. The existingDefaultReplicationPolicy
is still used by default, but identity replication can be enabled via thereplication.policy
configuration property. This is especially useful for users migrating from the older MirrorMaker (MM1), or for use-cases with simple one-way replication topologies where topic renaming is undesirable. Note thatIdentityReplicationPolicy
, unlikeDefaultReplicationPolicy
, cannot prevent replication cycles based on topic names, so take care to avoid cycles when constructing your replication topology. - The original MirrorMaker (MM1) and related classes have been deprecated. Please use the Connect-based MirrorMaker (MM2), as described in the Geo-Replication section.
Upgrading to 2.8.1 from any version 0.8.x through 2.7.x
If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
2.7
,2.6
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
2.7
,2.6
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to2.8
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.8 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 2.8.0
- The 2.8.0 release added a new method to the Authorizer Interface introduced in KIP-679. The motivation is to unblock our future plan to enable the strongest message delivery guarantee by default. Custom authorizer should consider providing a more efficient implementation that supports audit logging and any custom configs or access rules.
- IBP 2.8 introduces topic IDs to topics as a part of KIP-516. When using ZooKeeper, this information is stored in the TopicZNode. If the cluster is downgraded to a previous IBP or version, future topics will not get topic IDs and it is not guaranteed that topics will retain their topic IDs in ZooKeeper. This means that upon upgrading again, some topics or all topics will be assigned new IDs.
- Kafka Streams introduce a type-safe
split()
operator as a substitution for deprecatedKStream#branch()
method (cf. KIP-418).
Upgrading to 2.7.0 from any version 0.8.x through 2.6.x
If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
2.6
,2.5
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
2.6
,2.5
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to2.7
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.7 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 2.7.0
- The 2.7.0 release includes the core Raft implementation specified in KIP-595. There is a separate "raft" module containing most of the logic. Until integration with the controller is complete, there is a standalone server that users can use for testing the performance of the Raft implementation. See the README.md in the raft module for details
- KIP-651 adds support for using PEM files for key and trust stores.
- KIP-612 adds support for enforcing broker-wide and per-listener connection create rates. The 2.7.0 release contains the first part of KIP-612 with dynamic configuration coming in the 2.8.0 release.
- The ability to throttle topic and partition creations or topics deletions to prevent a cluster from being harmed via KIP-599
-
When new features become available in Kafka there are two main issues:
- How do Kafka clients become aware of broker capabilities?
- How does the broker decide which features to enable?
-
The ability to print record offsets and headers with the
ConsoleConsumer
is now possible via KIP-431 - The addition of KIP-554 continues progress towards the goal of Zookeeper removal from Kafka. The addition of KIP-554 means you don't have to connect directly to ZooKeeper anymore for managing SCRAM credentials.
- Altering non-reconfigurable configs of existent listeners causes
InvalidRequestException
. By contrast, the previous (unintended) behavior would have caused the updated configuration to be persisted, but it wouldn't take effect until the broker was restarted. See KAFKA-10479 for more discussion. SeeDynamicBrokerConfig.DynamicSecurityConfigs
andSocketServer.ListenerReconfigurableConfigs
for the supported reconfigurable configs of existent listeners. - Kafka Streams adds support for Sliding Windows Aggregations in the KStreams DSL.
- Reverse iteration over state stores enabling more efficient most recent update searches with KIP-617
- End-to-End latency metrics in Kafka Steams see KIP-613 for more details
- Kafka Streams added metrics reporting default RocksDB properties with KIP-607
- Better Scala implicit Serdes support from KIP-616
Upgrading to 2.6.0 from any version 0.8.x through 2.5.x
If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
2.5
,2.4
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
2.5
,2.4
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to2.6
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.6 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 2.6.0
- Kafka Streams adds a new processing mode (requires broker 2.5 or newer) that improves application scalability using exactly-once guarantees (cf. KIP-447)
- TLSv1.3 has been enabled by default for Java 11 or newer. The client and server will negotiate TLSv1.3 if both support it and fallback to TLSv1.2 otherwise. See KIP-573 for more details.
- The default value for the
client.dns.lookup
configuration has been changed fromdefault
touse_all_dns_ips
. If a hostname resolves to multiple IP addresses, clients and brokers will now attempt to connect to each IP in sequence until the connection is successfully established. See KIP-602 for more details. NotLeaderForPartitionException
has been deprecated and replaced withNotLeaderOrFollowerException
. Fetch requests and other requests intended only for the leader or follower return NOT_LEADER_OR_FOLLOWER(6) instead of REPLICA_NOT_AVAILABLE(9) if the broker is not a replica, ensuring that this transient error during reassignments is handled by all clients as a retriable exception.
Upgrading to 2.5.0 from any version 0.8.x through 2.4.x
If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
2.4
,2.3
, etc.) - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
2.4
,2.3
, etc.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g.
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to2.5
. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.5 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
- There are several notable changes to the reassignment tool
kafka-reassign-partitions.sh
following the completion of KIP-455. This tool now requires the--additional
flag to be provided when changing the throttle of an active reassignment. Reassignment cancellation is now possible using the--cancel
command. Finally, reassignment with--zookeeper
has been deprecated in favor of--bootstrap-server
. See the KIP for more detail.
Notable changes in 2.5.0
- When
RebalanceProtocol#COOPERATIVE
is used,Consumer#poll
can still return data while it is in the middle of a rebalance for those partitions still owned by the consumer; in additionConsumer#commitSync
now may throw a non-fatalRebalanceInProgressException
to notify users of such an event, in order to distinguish from the fatalCommitFailedException
and allow users to complete the ongoing rebalance and then reattempt committing offsets for those still-owned partitions. - For improved resiliency in typical network environments, the default value of
zookeeper.session.timeout.ms
has been increased from 6s to 18s andreplica.lag.time.max.ms
from 10s to 30s. - New DSL operator
cogroup()
has been added for aggregating multiple streams together at once. - Added a new
KStream.toTable()
API to translate an input event stream into a KTable. - Added a new Serde type
Void
to represent null keys or null values from input topic. - Deprecated
UsePreviousTimeOnInvalidTimestamp
and replaced it withUsePartitionTimeOnInvalidTimeStamp
. - Improved exactly-once semantics by adding a pending offset fencing mechanism and stronger transactional commit consistency check, which greatly simplifies the implementation of a scalable exactly-once application. We also added a new exactly-once semantics code example under examples folder. Check out KIP-447 for the full details.
- Added a new public api
KafkaStreams.queryMetadataForKey(String, K, Serializer) to get detailed information on the key being queried. It provides information about the partition number where the key resides in addition to hosts containing the active and standby partitions for the key.
- Provided support to query stale stores (for high availability) and the stores belonging to a specific partition by deprecating
KafkaStreams.store(String, QueryableStoreType)
and replacing it withKafkaStreams.store(StoreQueryParameters)
. - Added a new public api to access lag information for stores local to an instance with
KafkaStreams.allLocalStorePartitionLags()
. - Scala 2.11 is no longer supported. See KIP-531 for details.
- All Scala classes from the package
kafka.security.auth
have been deprecated. See KIP-504 for details of the new Java authorizer API added in 2.4.0. Note thatkafka.security.auth.Authorizer
andkafka.security.auth.SimpleAclAuthorizer
were deprecated in 2.4.0. - TLSv1 and TLSv1.1 have been disabled by default since these have known security vulnerabilities. Only TLSv1.2 is now
enabled by default. You can continue to use TLSv1 and TLSv1.1 by explicitly enabling these in the configuration options
ssl.protocol
andssl.enabled.protocols
. - ZooKeeper has been upgraded to 3.5.7, and a ZooKeeper upgrade from 3.4.X to 3.5.7 can fail if there are no snapshot files in the 3.4 data directory.
This usually happens in test upgrades where ZooKeeper 3.5.7 is trying to load an existing 3.4 data dir in which no snapshot file has been created.
For more details about the issue please refer to ZOOKEEPER-3056.
A fix is given in ZOOKEEPER-3056, which is to set
snapshot.trust.empty=true
config inzookeeper.properties
before the upgrade. - ZooKeeper version 3.5.7 supports TLS-encrypted connectivity to ZooKeeper both with or without client certificates, and additional Kafka configurations are available to take advantage of this. See KIP-515 for details.
Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, 1.1.x, 2.0.x or 2.1.x or 2.2.x or 2.3.x to 2.4.0
If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.10.0, 0.11.0, 1.0, 2.0, 2.2).
- log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0, 1.0, 1.1, 2.0, 2.1, 2.2, 2.3).
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to 2.4. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.4 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Additional Upgrade Notes:
- ZooKeeper has been upgraded to 3.5.6. ZooKeeper upgrade from 3.4.X to 3.5.6 can fail if there are no snapshot files in 3.4 data directory.
This usually happens in test upgrades where ZooKeeper 3.5.6 is trying to load an existing 3.4 data dir in which no snapshot file has been created.
For more details about the issue please refer to ZOOKEEPER-3056.
A fix is given in ZOOKEEPER-3056, which is to set
snapshot.trust.empty=true
config inzookeeper.properties
before the upgrade. But we have observed data loss in standalone cluster upgrades when usingsnapshot.trust.empty=true
config. For more details about the issue please refer to ZOOKEEPER-3644. So we recommend the safe workaround of copying empty snapshot file to the 3.4 data directory, if there are no snapshot files in 3.4 data directory. For more details about the workaround please refer to ZooKeeper Upgrade FAQ. -
An embedded Jetty based AdminServer added in ZooKeeper 3.5.
AdminServer is enabled by default in ZooKeeper and is started on port 8080.
AdminServer is disabled by default in the ZooKeeper config (
zookeeper.properties
) provided by the Apache Kafka distribution. Make sure to update your localzookeeper.properties
file withadmin.enableServer=false
if you wish to disable the AdminServer. Please refer AdminServer config to configure the AdminServer.
Notable changes in 2.4.0
- A new Admin API has been added for partition reassignments. Due to changing the way Kafka propagates reassignment information, it is possible to lose reassignment state in failure edge cases while upgrading to the new version. It is not recommended to start reassignments while upgrading.
- ZooKeeper has been upgraded from 3.4.14 to 3.5.6. TLS and dynamic reconfiguration are supported by the new version.
- The
bin/kafka-preferred-replica-election.sh
command line tool has been deprecated. It has been replaced bybin/kafka-leader-election.sh
. - The methods
electPreferredLeaders
in the JavaAdminClient
class have been deprecated in favor of the methodselectLeaders
. - Scala code leveraging the
NewTopic(String, int, short)
constructor with literal values will need to explicitly calltoShort
on the second literal. - The argument in the constructor
GroupAuthorizationException(String)
is now used to specify an exception message. Previously it referred to the group that failed authorization. This was done for consistency with other exception types and to avoid potential misuse. The constructorTopicAuthorizationException(String)
which was previously used for a single unauthorized topic was changed similarly. - The internal
PartitionAssignor
interface has been deprecated and replaced with a newConsumerPartitionAssignor
in the public API. Some methods/signatures are slightly different between the two interfaces. Users implementing a custom PartitionAssignor should migrate to the new interface as soon as possible. - The
DefaultPartitioner
now uses a sticky partitioning strategy. This means that records for specific topic with null keys and no assigned partition will be sent to the same partition until the batch is ready to be sent. When a new batch is created, a new partition is chosen. This decreases latency to produce, but it may result in uneven distribution of records across partitions in edge cases. Generally users will not be impacted, but this difference may be noticeable in tests and other situations producing records for a very short amount of time. - The blocking
KafkaConsumer#committed
methods have been extended to allow a list of partitions as input parameters rather than a single partition. It enables fewer request/response iterations between clients and brokers fetching for the committed offsets for the consumer group. The old overloaded functions are deprecated and we would recommend users to make their code changes to leverage the new methods (details can be found in KIP-520). - We've introduced a new
INVALID_RECORD
error in the produce response to distinguish from theCORRUPT_MESSAGE
error. To be more concrete, previously when a batch of records was sent as part of a single request to the broker and one or more of the records failed the validation due to various causes (mismatch magic bytes, crc checksum errors, null key for log compacted topics, etc), the whole batch would be rejected with the same and misleadingCORRUPT_MESSAGE
, and the caller of the producer client would see the corresponding exception from either the future object ofRecordMetadata
returned from thesend
call as well as in theCallback#onCompletion(RecordMetadata metadata, Exception exception)
Now with the new error code and improved error messages of the exception, producer callers would be better informed about the root cause why their sent records were failed. - We are introducing incremental cooperative rebalancing to the clients' group protocol, which allows consumers to keep all of their assigned partitions during a rebalance
and at the end revoke only those which must be migrated to another consumer for overall cluster balance. The
ConsumerCoordinator
will choose the latestRebalanceProtocol
that is commonly supported by all of the consumer's supported assignors. You can use the new built-inCooperativeStickyAssignor
or plug in your own custom cooperative assignor. To do so you must implement theConsumerPartitionAssignor
interface and includeRebalanceProtocol.COOPERATIVE
in the list returned byConsumerPartitionAssignor#supportedProtocols
. Your custom assignor can then leverage theownedPartitions
field in each consumer'sSubscription
to give partitions back to their previous owners whenever possible. Note that when a partition is to be reassigned to another consumer, it must be removed from the new assignment until it has been revoked from its original owner. Any consumer that has to revoke a partition will trigger a followup rebalance to allow the revoked partition to safely be assigned to its new owner. See the ConsumerPartitionAssignor RebalanceProtocol javadocs for more information.
To upgrade from the old (eager) protocol, which always revokes all partitions before rebalancing, to cooperative rebalancing, you must follow a specific upgrade path to get all clients on the sameConsumerPartitionAssignor
that supports the cooperative protocol. This can be done with two rolling bounces, using theCooperativeStickyAssignor
for the example: during the first one, add "cooperative-sticky" to the list of supported assignors for each member (without removing the previous assignor -- note that if previously using the default, you must include that explicitly as well). You then bounce and/or upgrade it. Once the entire group is on 2.4+ and all members have the "cooperative-sticky" among their supported assignors, remove the other assignor(s) and perform a second rolling bounce so that by the end all members support only the cooperative protocol. For further details on the cooperative rebalancing protocol and upgrade path, see KIP-429. - There are some behavioral changes to the
ConsumerRebalanceListener
, as well as a new API. Exceptions thrown during any of the listener's three callbacks will no longer be swallowed, and will instead be re-thrown all the way up to theConsumer.poll()
call. TheonPartitionsLost
method has been added to allow users to react to abnormal circumstances where a consumer may have lost ownership of its partitions (such as a missed rebalance) and cannot commit offsets. By default, this will simply call the existingonPartitionsRevoked
API to align with previous behavior. Note however thatonPartitionsLost
will not be called when the set of lost partitions is empty. This means that no callback will be invoked at the beginning of the first rebalance of a new consumer joining the group.
The semantics of theConsumerRebalanceListener's
callbacks are further changed when following the cooperative rebalancing protocol described above. In addition toonPartitionsLost
,onPartitionsRevoked
will also never be called when the set of revoked partitions is empty. The callback will generally be invoked only at the end of a rebalance, and only on the set of partitions that are being moved to another consumer. TheonPartitionsAssigned
callback will however always be called, even with an empty set of partitions, as a way to notify users of a rebalance event (this is true for both cooperative and eager). For details on the new callback semantics, see the ConsumerRebalanceListener javadocs. - The Scala trait
kafka.security.auth.Authorizer
has been deprecated and replaced with a new Java APIorg.apache.kafka.server.authorizer.Authorizer
. The authorizer implementation classkafka.security.auth.SimpleAclAuthorizer
has also been deprecated and replaced with a new implementationkafka.security.authorizer.AclAuthorizer
.AclAuthorizer
uses features supported by the new API to improve authorization logging and is compatible withSimpleAclAuthorizer
. For more details, see KIP-504.
Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, 1.1.x, 2.0.x or 2.1.x or 2.2.x to 2.3.0
If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0, 1.1).
- log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0, 1.0, 1.1, 2.0, 2.1, 2.2).
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to 2.3. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.3 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 2.3.0
- We are introducing a new rebalancing protocol for Kafka Connect based on incremental cooperative rebalancing. The new protocol does not require stopping all the tasks during a rebalancing phase between Connect workers. Instead, only the tasks that need to be exchanged between workers are stopped and they are started in a follow up rebalance. The new Connect protocol is enabled by default beginning with 2.3.0. For more details on how it works and how to enable the old behavior of eager rebalancing, checkout incremental cooperative rebalancing design.
- We are introducing static membership towards consumer user. This feature reduces unnecessary rebalances during normal application upgrades or rolling bounces. For more details on how to use it, checkout static membership design.
- Kafka Streams DSL switches its used store types. While this change is mainly transparent to users, there are some corner cases that may require code changes. See the Kafka Streams upgrade section for more details.
- Kafka Streams 2.3.0 requires 0.11 message format or higher and does not work with older message format.
Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, 1.1.x, 2.0.x or 2.1.x to 2.2.0
If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets. Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0, 1.1).
- log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0, 1.0, 1.1, 2.0).
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to 2.2. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.2 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Notable changes in 2.2.1
- Kafka Streams 2.2.1 requires 0.11 message format or higher and does not work with older message format.
Notable changes in 2.2.0
- The default consumer group id has been changed from the empty string (
""
) tonull
. Consumers who use the new default group id will not be able to subscribe to topics, and fetch or commit offsets. The empty string as consumer group id is deprecated but will be supported until a future major release. Old clients that rely on the empty string group id will now have to explicitly provide it as part of their consumer config. For more information see KIP-289. - The
bin/kafka-topics.sh
command line tool is now able to connect directly to brokers with--bootstrap-server
instead of zookeeper. The old--zookeeper
option is still available for now. Please read KIP-377 for more information. - Kafka Streams depends on a newer version of RocksDBs that requires MacOS 10.13 or higher.
Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, 1.1.x, or 2.0.0 to 2.1.0
Note that 2.1.x contains a change to the internal schema used to store consumer offsets. Once the upgrade is complete, it will not be possible to downgrade to previous versions. See the rolling upgrade notes below for more detail.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0, 1.1).
- log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0, 1.0, 1.1, 2.0).
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
- Once the cluster's behavior and performance has been verified, bump the protocol version by editing
inter.broker.protocol.version
and setting it to 2.1. - Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.1 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.
Additional Upgrade Notes:
- Offset expiration semantics has slightly changed in this version. According to the new semantics, offsets of partitions in a group will not be removed while the group is subscribed to the corresponding topic and is still active (has active consumers). If group becomes empty all its offsets will be removed after default offset retention period (or the one set by broker) has passed (unless the group becomes active again). Offsets associated with standalone (simple) consumers, that do not use Kafka group management, will be removed after default offset retention period (or the one set by broker) has passed since their last commit.
- The default for console consumer's
enable.auto.commit
property when nogroup.id
is provided is now set tofalse
. This is to avoid polluting the consumer coordinator cache as the auto-generated group is not likely to be used by other consumers. - The default value for the producer's
retries
config was changed toInteger.MAX_VALUE
, as we introduceddelivery.timeout.ms
in KIP-91, which sets an upper bound on the total time between sending a record and receiving acknowledgement from the broker. By default, the delivery timeout is set to 2 minutes. - By default, MirrorMaker now overrides
delivery.timeout.ms
toInteger.MAX_VALUE
when configuring the producer. If you have overridden the value ofretries
in order to fail faster, you will instead need to overridedelivery.timeout.ms
. - The
ListGroup
API now expects, as a recommended alternative,Describe Group
access to the groups a user should be able to list. Even though the oldDescribe Cluster
access is still supported for backward compatibility, using it for this API is not advised. - KIP-336 deprecates the ExtendedSerializer and ExtendedDeserializer interfaces and propagates the usage of Serializer and Deserializer. ExtendedSerializer and ExtendedDeserializer were introduced with KIP-82 to provide record headers for serializers and deserializers in a Java 7 compatible fashion. Now we consolidated these interfaces as Java 7 support has been dropped since.
Notable changes in 2.1.0
- Jetty has been upgraded to 9.4.12, which excludes TLS_RSA_* ciphers by default because they do not support forward secrecy, see https://github.com/eclipse/jetty.project/issues/2807 for more information.
- Unclean leader election is automatically enabled by the controller when
unclean.leader.election.enable
config is dynamically updated by using per-topic config override. - The
AdminClient
has added a methodAdminClient#metrics()
. Now any application using theAdminClient
can gain more information and insight by viewing the metrics captured from theAdminClient
. For more information see KIP-324 - Kafka now supports Zstandard compression from KIP-110. You must upgrade the broker as well as clients to make use of it. Consumers prior to 2.1.0 will not be able to read from topics which use Zstandard compression, so you should not enable it for a topic until all downstream consumers are upgraded. See the KIP for more detail.
Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, or 1.1.x to 2.0.0
Kafka 2.0.0 introduces wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please review the notable changes in 2.0.0 before upgrading.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0, 1.1).
- log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0, 1.0, 1.1).
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.
- Once the entire cluster is upgraded, bump the protocol version by editing
inter.broker.protocol.version
and setting it to 2.0. - Restart the brokers one by one for the new protocol version to take effect.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 2.0 on each broker and restart them one by one. Note that the older Scala consumer does not support the new message format introduced in 0.11, so to avoid the performance cost of down-conversion (or to take advantage of exactly once semantics), the newer Java consumer must be used.
Additional Upgrade Notes:
- If you are willing to accept downtime, you can simply take all the brokers down, update the code and start them back up. They will start with the new protocol by default.
- Bumping the protocol version and restarting can be done any time after the brokers are upgraded. It does not have to be immediately after. Similarly for the message format version.
- If you are using Java8 method references in your Kafka Streams code you might need to update your code to resolve method ambiguities. Hot-swapping the jar-file only might not work.
- ACLs should not be added to prefixed resources,
(added in KIP-290),
until all brokers in the cluster have been updated.
NOTE: any prefixed ACLs added to a cluster, even after the cluster is fully upgraded, will be ignored should the cluster be downgraded again.
Notable changes in 2.0.0
- KIP-186 increases the default offset retention time from 1 day to 7 days. This makes it less likely to "lose" offsets in an application that commits infrequently. It also increases the active set of offsets and therefore can increase memory usage on the broker. Note that the console consumer currently enables offset commit by default and can be the source of a large number of offsets which this change will now preserve for 7 days instead of 1. You can preserve the existing behavior by setting the broker config
offsets.retention.minutes
to 1440. - Support for Java 7 has been dropped, Java 8 is now the minimum version required.
- The default value for
ssl.endpoint.identification.algorithm
was changed tohttps
, which performs hostname verification (man-in-the-middle attacks are possible otherwise). Setssl.endpoint.identification.algorithm
to an empty string to restore the previous behaviour. - KAFKA-5674 extends the lower interval of
max.connections.per.ip
minimum to zero and therefore allows IP-based filtering of inbound connections. - KIP-272
added API version tag to the metric
kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower|...}
. This metric now becomeskafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower|...},version={0|1|2|3|...}
. This will impact JMX monitoring tools that do not automatically aggregate. To get the total count for a specific request type, the tool needs to be updated to aggregate across different versions. - KIP-225 changed the metric "records.lag" to use tags for topic and partition. The original version with the name format "{topic}-{partition}.records-lag" has been removed.
- The Scala consumers, which have been deprecated since 0.11.0.0, have been removed. The Java consumer has been the recommended option since 0.10.0.0. Note that the Scala consumers in 1.1.0 (and older) will continue to work even if the brokers are upgraded to 2.0.0.
- The Scala producers, which have been deprecated since 0.10.0.0, have been removed. The Java producer has been the recommended option since 0.9.0.0. Note that the behaviour of the default partitioner in the Java producer differs from the default partitioner in the Scala producers. Users migrating should consider configuring a custom partitioner that retains the previous behaviour. Note that the Scala producers in 1.1.0 (and older) will continue to work even if the brokers are upgraded to 2.0.0.
- MirrorMaker and ConsoleConsumer no longer support the Scala consumer, they always use the Java consumer.
- The ConsoleProducer no longer supports the Scala producer, it always uses the Java producer.
- A number of deprecated tools that rely on the Scala clients have been removed: ReplayLogProducer, SimpleConsumerPerformance, SimpleConsumerShell, ExportZkOffsets, ImportZkOffsets, UpdateOffsetsInZK, VerifyConsumerRebalance.
- The deprecated kafka.tools.ProducerPerformance has been removed, please use org.apache.kafka.tools.ProducerPerformance.
- New Kafka Streams configuration parameter
upgrade.from
added that allows rolling bounce upgrade from older version. - KIP-284 changed the retention time for Kafka Streams repartition topics by setting its default value to
Long.MAX_VALUE
. - Updated
ProcessorStateManager
APIs in Kafka Streams for registering state stores to the processor topology. For more details please read the Streams Upgrade Guide. -
In earlier releases, Connect's worker configuration required the
internal.key.converter
andinternal.value.converter
properties. In 2.0, these are no longer required and default to the JSON converter. You may safely remove these properties from your Connect standalone and distributed worker configurations:
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter.schemas.enable=false
- KIP-266 adds a new consumer configuration
default.api.timeout.ms
to specify the default timeout to use forKafkaConsumer
APIs that could block. The KIP also adds overloads for such blocking APIs to support specifying a specific timeout to use for each of them instead of using the default timeout set bydefault.api.timeout.ms
. In particular, a newpoll(Duration)
API has been added which does not block for dynamic partition assignment. The oldpoll(long)
API has been deprecated and will be removed in a future version. Overloads have also been added for otherKafkaConsumer
methods likepartitionsFor
,listTopics
,offsetsForTimes
,beginningOffsets
,endOffsets
andclose
that take in aDuration
. - Also as part of KIP-266, the default value of
request.timeout.ms
has been changed to 30 seconds. The previous value was a little higher than 5 minutes to account for maximum time that a rebalance would take. Now we treat the JoinGroup request in the rebalance as a special case and use a value derived frommax.poll.interval.ms
for the request timeout. All other request types use the timeout defined byrequest.timeout.ms
- The internal method
kafka.admin.AdminClient.deleteRecordsBefore
has been removed. Users are encouraged to migrate toorg.apache.kafka.clients.admin.AdminClient.deleteRecords
. - The AclCommand tool
--producer
convenience option uses the KIP-277 finer grained ACL on the given topic. - KIP-176 removes
the
--new-consumer
option for all consumer based tools. This option is redundant since the new consumer is automatically used if --bootstrap-server is defined. - KIP-290 adds the ability to define ACLs on prefixed resources, e.g. any topic starting with 'foo'.
- KIP-283 improves message down-conversion
handling on Kafka broker, which has typically been a memory-intensive operation. The KIP adds a mechanism by which the operation becomes less memory intensive
by down-converting chunks of partition data at a time which helps put an upper bound on memory consumption. With this improvement, there is a change in
FetchResponse
protocol behavior where the broker could send an oversized message batch towards the end of the response with an invalid offset. Such oversized messages must be ignored by consumer clients, as is done byKafkaConsumer
.KIP-283 also adds new topic and broker configurations
message.downconversion.enable
andlog.message.downconversion.enable
respectively to control whether down-conversion is enabled. When disabled, broker does not perform any down-conversion and instead sends anUNSUPPORTED_VERSION
error to the client. - Dynamic broker configuration options can be stored in ZooKeeper using kafka-configs.sh before brokers are started. This option can be used to avoid storing clear passwords in server.properties as all password configs may be stored encrypted in ZooKeeper.
- ZooKeeper hosts are now re-resolved if connection attempt fails. But if your ZooKeeper host names resolve
to multiple addresses and some of them are not reachable, then you may need to increase the connection timeout
zookeeper.connection.timeout.ms
.
New Protocol Versions
- KIP-279: OffsetsForLeaderEpochResponse v1 introduces a partition-level
leader_epoch
field. - KIP-219: Bump up the protocol versions of non-cluster action requests and responses that are throttled on quota violation.
- KIP-290: Bump up the protocol versions ACL create, describe and delete requests and responses.
Upgrading a 1.1 Kafka Streams Application
- Upgrading your Streams application from 1.1 to 2.0 does not require a broker upgrade. A Kafka Streams 2.0 application can connect to 2.0, 1.1, 1.0, 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
- Note that in 2.0 we have removed the public APIs that are deprecated prior to 1.0; users leveraging on those deprecated APIs need to make code changes accordingly. See Streams API changes in 2.0.0 for more details.
Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, or 1.0.x to 1.1.x
Kafka 1.1.0 introduces wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please review the notable changes in 1.1.0 before upgrading.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0).
- log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0 or 1.0).
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.
- Once the entire cluster is upgraded, bump the protocol version by editing
inter.broker.protocol.version
and setting it to 1.1. - Restart the brokers one by one for the new protocol version to take effect.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 1.1 on each broker and restart them one by one. Note that the older Scala consumer does not support the new message format introduced in 0.11, so to avoid the performance cost of down-conversion (or to take advantage of exactly once semantics), the newer Java consumer must be used.
Additional Upgrade Notes:
- If you are willing to accept downtime, you can simply take all the brokers down, update the code and start them back up. They will start with the new protocol by default.
- Bumping the protocol version and restarting can be done any time after the brokers are upgraded. It does not have to be immediately after. Similarly for the message format version.
- If you are using Java8 method references in your Kafka Streams code you might need to update your code to resolve method ambiguties. Hot-swapping the jar-file only might not work.
Notable changes in 1.1.1
- New Kafka Streams configuration parameter
upgrade.from
added that allows rolling bounce upgrade from version 0.10.0.x - See the Kafka Streams upgrade guide for details about this new config.
Notable changes in 1.1.0
- The kafka artifact in Maven no longer depends on log4j or slf4j-log4j12. Similarly to the kafka-clients artifact, users can now choose the logging back-end by including the appropriate slf4j module (slf4j-log4j12, logback, etc.). The release tarball still includes log4j and slf4j-log4j12.
- KIP-225 changed the metric "records.lag" to use tags for topic and partition. The original version with the name format "{topic}-{partition}.records-lag" is deprecated and will be removed in 2.0.0.
- Kafka Streams is more robust against broker communication errors. Instead of stopping the Kafka Streams client with a fatal exception,
Kafka Streams tries to self-heal and reconnect to the cluster. Using the new
AdminClient
you have better control of how often Kafka Streams retries and can configure fine-grained timeouts (instead of hard coded retries as in older version). - Kafka Streams rebalance time was reduced further making Kafka Streams more responsive.
- Kafka Connect now supports message headers in both sink and source connectors, and to manipulate them via simple message transforms. Connectors must be changed to explicitly use them. A new
HeaderConverter
is introduced to control how headers are (de)serialized, and the new "SimpleHeaderConverter" is used by default to use string representations of values. - kafka.tools.DumpLogSegments now automatically sets deep-iteration option if print-data-log is enabled explicitly or implicitly due to any of the other options like decoder.
New Protocol Versions
- KIP-226 introduced DescribeConfigs Request/Response v1.
- KIP-227 introduced Fetch Request/Response v7.
Upgrading a 1.0 Kafka Streams Application
- Upgrading your Streams application from 1.0 to 1.1 does not require a broker upgrade. A Kafka Streams 1.1 application can connect to 1.0, 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
- See Streams API changes in 1.1.0 for more details.
Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x or 0.11.0.x to 1.0.0
Kafka 1.0.0 introduces wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please review the notable changes in 1.0.0 before upgrading.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0).
- log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- inter.broker.protocol.version=0.11.0
- log.message.format.version=0.11.0
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.
- Once the entire cluster is upgraded, bump the protocol version by editing
inter.broker.protocol.version
and setting it to 1.0. - Restart the brokers one by one for the new protocol version to take effect.
- If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 1.0 on each broker and restart them one by one. If you are upgrading from 0.11.0 and log.message.format.version is set to 0.11.0, you can update the config and skip the rolling restart. Note that the older Scala consumer does not support the new message format introduced in 0.11, so to avoid the performance cost of down-conversion (or to take advantage of exactly once semantics), the newer Java consumer must be used.
Additional Upgrade Notes:
- If you are willing to accept downtime, you can simply take all the brokers down, update the code and start them back up. They will start with the new protocol by default.
- Bumping the protocol version and restarting can be done any time after the brokers are upgraded. It does not have to be immediately after. Similarly for the message format version.
Notable changes in 1.0.2
- New Kafka Streams configuration parameter
upgrade.from
added that allows rolling bounce upgrade from version 0.10.0.x - See the Kafka Streams upgrade guide for details about this new config.
Notable changes in 1.0.1
- Restored binary compatibility of AdminClient's Options classes (e.g. CreateTopicsOptions, DeleteTopicsOptions, etc.) with 0.11.0.x. Binary (but not source) compatibility had been broken inadvertently in 1.0.0.
Notable changes in 1.0.0
- Topic deletion is now enabled by default, since the functionality is now stable. Users who wish to
to retain the previous behavior should set the broker config
delete.topic.enable
tofalse
. Keep in mind that topic deletion removes data and the operation is not reversible (i.e. there is no "undelete" operation) - For topics that support timestamp search if no offset can be found for a partition, that partition is now included in the search result with a null offset value. Previously, the partition was not included in the map. This change was made to make the search behavior consistent with the case of topics not supporting timestamp search.
- If the
inter.broker.protocol.version
is 1.0 or later, a broker will now stay online to serve replicas on live log directories even if there are offline log directories. A log directory may become offline due to IOException caused by hardware failure. Users need to monitor the per-broker metricofflineLogDirectoryCount
to check whether there is offline log directory. - Added KafkaStorageException which is a retriable exception. KafkaStorageException will be converted to NotLeaderForPartitionException in the response if the version of the client's FetchRequest or ProducerRequest does not support KafkaStorageException.
- -XX:+DisableExplicitGC was replaced by -XX:+ExplicitGCInvokesConcurrent in the default JVM settings. This helps avoid out of memory exceptions during allocation of native memory by direct buffers in some cases.
- The overridden
handleError
method implementations have been removed from the following deprecated classes in thekafka.api
package:FetchRequest
,GroupCoordinatorRequest
,OffsetCommitRequest
,OffsetFetchRequest
,OffsetRequest
,ProducerRequest
, andTopicMetadataRequest
. This was only intended for use on the broker, but it is no longer in use and the implementations have not been maintained. A stub implementation has been retained for binary compatibility. - The Java clients and tools now accept any string as a client-id.
- The deprecated tool
kafka-consumer-offset-checker.sh
has been removed. Usekafka-consumer-groups.sh
to get consumer group details. - SimpleAclAuthorizer now logs access denials to the authorizer log by default.
- Authentication failures are now reported to clients as one of the subclasses of
AuthenticationException
. No retries will be performed if a client connection fails authentication. - Custom
SaslServer
implementations may throwSaslAuthenticationException
to provide an error message to return to clients indicating the reason for authentication failure. Implementors should take care not to include any security-critical information in the exception message that should not be leaked to unauthenticated clients. - The
app-info
mbean registered with JMX to provide version and commit id will be deprecated and replaced with metrics providing these attributes. - Kafka metrics may now contain non-numeric values.
org.apache.kafka.common.Metric#value()
has been deprecated and will return0.0
in such cases to minimise the probability of breaking users who read the value of every client metric (via aMetricsReporter
implementation or by calling themetrics()
method).org.apache.kafka.common.Metric#metricValue()
can be used to retrieve numeric and non-numeric metric values. - Every Kafka rate metric now has a corresponding cumulative count metric with the suffix
-total
to simplify downstream processing. For example,records-consumed-rate
has a corresponding metric namedrecords-consumed-total
. - Mx4j will only be enabled if the system property
kafka_mx4jenable
is set totrue
. Due to a logic inversion bug, it was previously enabled by default and disabled ifkafka_mx4jenable
was set totrue
. - The package
org.apache.kafka.common.security.auth
in the clients jar has been made public and added to the javadocs. Internal classes which had previously been located in this package have been moved elsewhere. - When using an Authorizer and a user doesn't have required permissions on a topic, the broker will return TOPIC_AUTHORIZATION_FAILED errors to requests irrespective of topic existence on broker. If the user have required permissions and the topic doesn't exists, then the UNKNOWN_TOPIC_OR_PARTITION error code will be returned.
- config/consumer.properties file updated to use new consumer config properties.
New Protocol Versions
- KIP-112: LeaderAndIsrRequest v1 introduces a partition-level
is_new
field. - KIP-112: UpdateMetadataRequest v4 introduces a partition-level
offline_replicas
field. - KIP-112: MetadataResponse v5 introduces a partition-level
offline_replicas
field. - KIP-112: ProduceResponse v4 introduces error code for KafkaStorageException.
- KIP-112: FetchResponse v6 introduces error code for KafkaStorageException.
- KIP-152: SaslAuthenticate request has been added to enable reporting of authentication failures. This request will be used if the SaslHandshake request version is greater than 0.
Upgrading a 0.11.0 Kafka Streams Application
- Upgrading your Streams application from 0.11.0 to 1.0 does not require a broker upgrade. A Kafka Streams 1.0 application can connect to 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though). However, Kafka Streams 1.0 requires 0.10 message format or newer and does not work with older message formats.
- If you are monitoring on streams metrics, you will need make some changes to the metrics names in your reporting and monitoring code, because the metrics sensor hierarchy was changed.
- There are a few public APIs including
ProcessorContext#schedule()
,Processor#punctuate()
andKStreamBuilder
,TopologyBuilder
are being deprecated by new APIs. We recommend making corresponding code changes, which should be very minor since the new APIs look quite similar, when you upgrade. - See Streams API changes in 1.0.0 for more details.
Upgrading a 0.10.2 Kafka Streams Application
- Upgrading your Streams application from 0.10.2 to 1.0 does not require a broker upgrade. A Kafka Streams 1.0 application can connect to 1.0, 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
- If you are monitoring on streams metrics, you will need make some changes to the metrics names in your reporting and monitoring code, because the metrics sensor hierarchy was changed.
- There are a few public APIs including
ProcessorContext#schedule()
,Processor#punctuate()
andKStreamBuilder
,TopologyBuilder
are being deprecated by new APIs. We recommend making corresponding code changes, which should be very minor since the new APIs look quite similar, when you upgrade. - If you specify customized
key.serde
,value.serde
andtimestamp.extractor
in configs, it is recommended to use their replaced configure parameter as these configs are deprecated. - See Streams API changes in 0.11.0 for more details.
Upgrading a 0.10.1 Kafka Streams Application
- Upgrading your Streams application from 0.10.1 to 1.0 does not require a broker upgrade. A Kafka Streams 1.0 application can connect to 1.0, 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
- You need to recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
- If you are monitoring on streams metrics, you will need make some changes to the metrics names in your reporting and monitoring code, because the metrics sensor hierarchy was changed.
- There are a few public APIs including
ProcessorContext#schedule()
,Processor#punctuate()
andKStreamBuilder
,TopologyBuilder
are being deprecated by new APIs. We recommend making corresponding code changes, which should be very minor since the new APIs look quite similar, when you upgrade. - If you specify customized
key.serde
,value.serde
andtimestamp.extractor
in configs, it is recommended to use their replaced configure parameter as these configs are deprecated. - If you use a custom (i.e., user implemented) timestamp extractor, you will need to update this code, because the
TimestampExtractor
interface was changed. - If you register custom metrics, you will need to update this code, because the
StreamsMetric
interface was changed. - See Streams API changes in 1.0.0, Streams API changes in 0.11.0 and Streams API changes in 0.10.2 for more details.
Upgrading a 0.10.0 Kafka Streams Application
- Upgrading your Streams application from 0.10.0 to 1.0 does require a broker upgrade because a Kafka Streams 1.0 application can only connect to 0.1, 0.11.0, 0.10.2, or 0.10.1 brokers.
- There are couple of API changes, that are not backward compatible (cf. Streams API changes in 1.0.0, Streams API changes in 0.11.0, Streams API changes in 0.10.2, and Streams API changes in 0.10.1 for more details). Thus, you need to update and recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
- Upgrading from 0.10.0.x to 1.0.2 requires two rolling bounces with config
upgrade.from="0.10.0"
set for first upgrade phase (cf. KIP-268). As an alternative, an offline upgrade is also possible.- prepare your application instances for a rolling bounce and make sure that config
upgrade.from
is set to"0.10.0"
for new version 0.11.0.3 - bounce each instance of your application once
- prepare your newly deployed 1.0.2 application instances for a second round of rolling bounces; make sure to remove the value for config
upgrade.from
- bounce each instance of your application once more to complete the upgrade
- prepare your application instances for a rolling bounce and make sure that config
- Upgrading from 0.10.0.x to 1.0.0 or 1.0.1 requires an offline upgrade (rolling bounce upgrade is not supported)
- stop all old (0.10.0.x) application instances
- update your code and swap old code and jar file with new code and new jar file
- restart all new (1.0.0 or 1.0.1) application instances
Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x or 0.10.2.x to 0.11.0.0
Kafka 0.11.0.0 introduces a new message format version as well as wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please review the notable changes in 0.11.0.0 before upgrading.
Starting with version 0.10.2, Java clients (producer and consumer) have acquired the ability to communicate with older brokers. Version 0.11.0 clients can talk to version 0.10.0 or newer brokers. However, if your brokers are older than 0.10.0, you must upgrade all the brokers in the Kafka cluster before upgrading your clients. Version 0.11.0 brokers support 0.8.x and newer clients.
For a rolling upgrade:
- Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the current message format version currently in use. If you have
not overridden the message format previously, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1 or 0.10.2).
- log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.
- Once the entire cluster is upgraded, bump the protocol version by editing
inter.broker.protocol.version
and setting it to 0.11.0, but do not changelog.message.format.version
yet. - Restart the brokers one by one for the new protocol version to take effect.
- Once all (or most) consumers have been upgraded to 0.11.0 or later, then change log.message.format.version to 0.11.0 on each broker and restart them one by one. Note that the older Scala consumer does not support the new message format, so to avoid the performance cost of down-conversion (or to take advantage of exactly once semantics), the new Java consumer must be used.
Additional Upgrade Notes:
- If you are willing to accept downtime, you can simply take all the brokers down, update the code and start them back up. They will start with the new protocol by default.
- Bumping the protocol version and restarting can be done any time after the brokers are upgraded. It does not have to be immediately after. Similarly for the message format version.
- It is also possible to enable the 0.11.0 message format on individual topics using the topic admin tool (
bin/kafka-topics.sh
) prior to updating the global settinglog.message.format.version
. - If you are upgrading from a version prior to 0.10.0, it is NOT necessary to first update the message format to 0.10.0 before you switch to 0.11.0.
Upgrading a 0.10.2 Kafka Streams Application
- Upgrading your Streams application from 0.10.2 to 0.11.0 does not require a broker upgrade. A Kafka Streams 0.11.0 application can connect to 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
- If you specify customized
key.serde
,value.serde
andtimestamp.extractor
in configs, it is recommended to use their replaced configure parameter as these configs are deprecated. - See Streams API changes in 0.11.0 for more details.
Upgrading a 0.10.1 Kafka Streams Application
- Upgrading your Streams application from 0.10.1 to 0.11.0 does not require a broker upgrade. A Kafka Streams 0.11.0 application can connect to 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
- You need to recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
- If you specify customized
key.serde
,value.serde
andtimestamp.extractor
in configs, it is recommended to use their replaced configure parameter as these configs are deprecated. - If you use a custom (i.e., user implemented) timestamp extractor, you will need to update this code, because the
TimestampExtractor
interface was changed. - If you register custom metrics, you will need to update this code, because the
StreamsMetric
interface was changed. - See Streams API changes in 0.11.0 and Streams API changes in 0.10.2 for more details.
Upgrading a 0.10.0 Kafka Streams Application
- Upgrading your Streams application from 0.10.0 to 0.11.0 does require a broker upgrade because a Kafka Streams 0.11.0 application can only connect to 0.11.0, 0.10.2, or 0.10.1 brokers.
- There are couple of API changes, that are not backward compatible (cf. Streams API changes in 0.11.0, Streams API changes in 0.10.2, and Streams API changes in 0.10.1 for more details). Thus, you need to update and recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
- Upgrading from 0.10.0.x to 0.11.0.3 requires two rolling bounces with config
upgrade.from="0.10.0"
set for first upgrade phase (cf. KIP-268). As an alternative, an offline upgrade is also possible.- prepare your application instances for a rolling bounce and make sure that config
upgrade.from
is set to"0.10.0"
for new version 0.11.0.3 - bounce each instance of your application once
- prepare your newly deployed 0.11.0.3 application instances for a second round of rolling bounces; make sure to remove the value for config
upgrade.from
- bounce each instance of your application once more to complete the upgrade
- prepare your application instances for a rolling bounce and make sure that config
- Upgrading from 0.10.0.x to 0.11.0.0, 0.11.0.1, or 0.11.0.2 requires an offline upgrade (rolling bounce upgrade is not supported)
- stop all old (0.10.0.x) application instances
- update your code and swap old code and jar file with new code and new jar file
- restart all new (0.11.0.0 , 0.11.0.1, or 0.11.0.2) application instances
Notable changes in 0.11.0.3
- New Kafka Streams configuration parameter
upgrade.from
added that allows rolling bounce upgrade from version 0.10.0.x - See the Kafka Streams upgrade guide for details about this new config.
Notable changes in 0.11.0.0
- Unclean leader election is now disabled by default. The new default favors durability over availability. Users who wish to
to retain the previous behavior should set the broker config
unclean.leader.election.enable
totrue
. - Producer configs
block.on.buffer.full
,metadata.fetch.timeout.ms
andtimeout.ms
have been removed. They were initially deprecated in Kafka 0.9.0.0. - The
offsets.topic.replication.factor
broker config is now enforced upon auto topic creation. Internal auto topic creation will fail with a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets this replication factor requirement. - When compressing data with snappy, the producer and broker will use the compression scheme's default block size (2 x 32 KB) instead of 1 KB in order to improve the compression ratio. There have been reports of data compressed with the smaller block size being 50% larger than when compressed with the larger block size. For the snappy case, a producer with 5000 partitions will require an additional 315 MB of JVM heap.
- Similarly, when compressing data with gzip, the producer and broker will use 8 KB instead of 1 KB as the buffer size. The default for gzip is excessively low (512 bytes).
- The broker configuration
max.message.bytes
now applies to the total size of a batch of messages. Previously the setting applied to batches of compressed messages, or to non-compressed messages individually. A message batch may consist of only a single message, so in most cases, the limitation on the size of individual messages is only reduced by the overhead of the batch format. However, there are some subtle implications for message format conversion (see below for more detail). Note also that while previously the broker would ensure that at least one message is returned in each fetch request (regardless of the total and partition-level fetch sizes), the same behavior now applies to one message batch. - GC log rotation is enabled by default, see KAFKA-3754 for details.
- Deprecated constructors of RecordMetadata, MetricName and Cluster classes have been removed.
- Added user headers support through a new Headers interface providing user headers read and write access.
- ProducerRecord and ConsumerRecord expose the new Headers API via
Headers headers()
method call. - ExtendedSerializer and ExtendedDeserializer interfaces are introduced to support serialization and deserialization for headers. Headers will be ignored if the configured serializer and deserializer are not the above classes.
- A new config,
group.initial.rebalance.delay.ms
, was introduced. This config specifies the time, in milliseconds, that theGroupCoordinator
will delay the initial consumer rebalance. The rebalance will be further delayed by the value ofgroup.initial.rebalance.delay.ms
as new members join the group, up to a maximum ofmax.poll.interval.ms
. The default value for this is 3 seconds. During development and testing it might be desirable to set this to 0 in order to not delay test execution time. org.apache.kafka.common.Cluster#partitionsForTopic
,partitionsForNode
andavailablePartitionsForTopic
methods will return an empty list instead ofnull
(which is considered a bad practice) in case the metadata for the required topic does not exist.- Streams API configuration parameters
timestamp.extractor
,key.serde
, andvalue.serde
were deprecated and replaced bydefault.timestamp.extractor
,default.key.serde
, anddefault.value.serde
, respectively. - For offset commit failures in the Java consumer's
commitAsync
APIs, we no longer expose the underlying cause when instances ofRetriableCommitFailedException
are passed to the commit callback. See KAFKA-5052 for more detail.
New Protocol Versions
- KIP-107: FetchRequest v5 introduces a partition-level
log_start_offset
field. - KIP-107: FetchResponse v5 introduces a partition-level
log_start_offset
field. - KIP-82: ProduceRequest v3 introduces an array of
header
in the message protocol, containingkey
field andvalue
field. - KIP-82: FetchResponse v5 introduces an array of
header
in the message protocol, containingkey
field andvalue
field.
Notes on Exactly Once Semantics
Kafka 0.11.0 includes support for idempotent and transactional capabilities in the producer. Idempotent delivery ensures that messages are delivered exactly once to a particular topic partition during the lifetime of a single producer. Transactional delivery allows producers to send data to multiple partitions such that either all messages are successfully delivered, or none of them are. Together, these capabilities enable "exactly once semantics" in Kafka. More details on these features are available in the user guide, but below we add a few specific notes on enabling them in an upgraded cluster. Note that enabling EoS is not required and there is no impact on the broker's behavior if unused.
- Only the new Java producer and consumer support exactly once semantics.
- These features depend crucially on the 0.11.0 message format. Attempting to use them on an older format will result in unsupported version errors.
- Transaction state is stored in a new internal topic
__transaction_state
. This topic is not created until the the first attempt to use a transactional request API. Similar to the consumer offsets topic, there are several settings to control the topic's configuration. For example,transaction.state.log.min.isr
controls the minimum ISR for this topic. See the configuration section in the user guide for a full list of options. - For secure clusters, the transactional APIs require new ACLs which can be turned on with the
bin/kafka-acls.sh
. tool. - EoS in Kafka introduces new request APIs and modifies several existing ones. See KIP-98 for the full details
Notes on the new message format in 0.11.0
The 0.11.0 message format includes several major enhancements in order to support better delivery semantics for the producer (see KIP-98) and improved replication fault tolerance (see KIP-101). Although the new format contains more information to make these improvements possible, we have made the batch format much more efficient. As long as the number of messages per batch is more than 2, you can expect lower overall overhead. For smaller batches, however, there may be a small performance impact. See here for the results of our initial performance analysis of the new message format. You can also find more detail on the message format in the KIP-98 proposal.
One of the notable differences in the new message format is that even uncompressed messages are stored together as a single batch.
This has a few implications for the broker configuration max.message.bytes
, which limits the size of a single batch. First,
if an older client produces messages to a topic partition using the old format, and the messages are individually smaller than
max.message.bytes
, the broker may still reject them after they are merged into a single batch during the up-conversion process.
Generally this can happen when the aggregate size of the individual messages is larger than max.message.bytes
. There is a similar
effect for older consumers reading messages down-converted from the new format: if the fetch size is not set at least as large as
max.message.bytes
, the consumer may not be able to make progress even if the individual uncompressed messages are smaller
than the configured fetch size. This behavior does not impact the Java client for 0.10.1.0 and later since it uses an updated fetch protocol
which ensures that at least one message can be returned even if it exceeds the fetch size. To get around these problems, you should ensure
1) that the producer's batch size is not set larger than max.message.bytes
, and 2) that the consumer's fetch size is set at
least as large as max.message.bytes
.
Most of the discussion on the performance impact of upgrading to the 0.10.0 message format remains pertinent to the 0.11.0 upgrade. This mainly affects clusters that are not secured with TLS since "zero-copy" transfer is already not possible in that case. In order to avoid the cost of down-conversion, you should ensure that consumer applications are upgraded to the latest 0.11.0 client. Significantly, since the old consumer has been deprecated in 0.11.0.0, it does not support the new message format. You must upgrade to use the new consumer to use the new message format without the cost of down-conversion. Note that 0.11.0 consumers support backwards compatibility with 0.10.0 brokers and upward, so it is possible to upgrade the clients first before the brokers.
Upgrading from 0.8.x, 0.9.x, 0.10.0.x or 0.10.1.x to 0.10.2.0
0.10.2.0 has wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please review the notable changes in 0.10.2.0 before upgrading.
Starting with version 0.10.2, Java clients (producer and consumer) have acquired the ability to communicate with older brokers. Version 0.10.2 clients can talk to version 0.10.0 or newer brokers. However, if your brokers are older than 0.10.0, you must upgrade all the brokers in the Kafka cluster before upgrading your clients. Version 0.10.2 brokers support 0.8.x and newer clients.
For a rolling upgrade:
- Update server.properties file on all brokers and add the following properties:
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0 or 0.10.1).
- log.message.format.version=CURRENT_KAFKA_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.
- Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 0.10.2.
- If your previous message format is 0.10.0, change log.message.format.version to 0.10.2 (this is a no-op as the message format is the same for 0.10.0, 0.10.1 and 0.10.2). If your previous message format version is lower than 0.10.0, do not change log.message.format.version yet - this parameter should only change once all consumers have been upgraded to 0.10.0.0 or later.
- Restart the brokers one by one for the new protocol version to take effect.
- If log.message.format.version is still lower than 0.10.0 at this point, wait until all consumers have been upgraded to 0.10.0 or later, then change log.message.format.version to 0.10.2 on each broker and restart them one by one.
Note: If you are willing to accept downtime, you can simply take all the brokers down, update the code and start all of them. They will start with the new protocol by default.
Note: Bumping the protocol version and restarting can be done any time after the brokers were upgraded. It does not have to be immediately after.
Upgrading a 0.10.1 Kafka Streams Application
- Upgrading your Streams application from 0.10.1 to 0.10.2 does not require a broker upgrade. A Kafka Streams 0.10.2 application can connect to 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
- You need to recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
- If you use a custom (i.e., user implemented) timestamp extractor, you will need to update this code, because the
TimestampExtractor
interface was changed. - If you register custom metrics, you will need to update this code, because the
StreamsMetric
interface was changed. - See Streams API changes in 0.10.2 for more details.
Upgrading a 0.10.0 Kafka Streams Application
- Upgrading your Streams application from 0.10.0 to 0.10.2 does require a broker upgrade because a Kafka Streams 0.10.2 application can only connect to 0.10.2 or 0.10.1 brokers.
- There are couple of API changes, that are not backward compatible (cf. Streams API changes in 0.10.2 for more details). Thus, you need to update and recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
- Upgrading from 0.10.0.x to 0.10.2.2 requires two rolling bounces with config
upgrade.from="0.10.0"
set for first upgrade phase (cf. KIP-268). As an alternative, an offline upgrade is also possible.- prepare your application instances for a rolling bounce and make sure that config
upgrade.from
is set to"0.10.0"
for new version 0.10.2.2 - bounce each instance of your application once
- prepare your newly deployed 0.10.2.2 application instances for a second round of rolling bounces; make sure to remove the value for config
upgrade.from
- bounce each instance of your application once more to complete the upgrade
- prepare your application instances for a rolling bounce and make sure that config
- Upgrading from 0.10.0.x to 0.10.2.0 or 0.10.2.1 requires an offline upgrade (rolling bounce upgrade is not supported)
- stop all old (0.10.0.x) application instances
- update your code and swap old code and jar file with new code and new jar file
- restart all new (0.10.2.0 or 0.10.2.1) application instances
Notable changes in 0.10.2.2
- New configuration parameter
upgrade.from
added that allows rolling bounce upgrade from version 0.10.0.x
Notable changes in 0.10.2.1
- The default values for two configurations of the StreamsConfig class were changed to improve the resiliency of Kafka Streams applications. The internal Kafka Streams producer
retries
default value was changed from 0 to 10. The internal Kafka Streams consumermax.poll.interval.ms
default value was changed from 300000 toInteger.MAX_VALUE
.
Notable changes in 0.10.2.0
- The Java clients (producer and consumer) have acquired the ability to communicate with older brokers. Version 0.10.2 clients can talk to version 0.10.0 or newer brokers. Note that some features are not available or are limited when older brokers are used.
- Several methods on the Java consumer may now throw
InterruptException
if the calling thread is interrupted. Please refer to theKafkaConsumer
Javadoc for a more in-depth explanation of this change. - Java consumer now shuts down gracefully. By default, the consumer waits up to 30 seconds to complete pending requests.
A new close API with timeout has been added to
KafkaConsumer
to control the maximum wait time. - Multiple regular expressions separated by commas can be passed to MirrorMaker with the new Java consumer via the --whitelist option. This makes the behaviour consistent with MirrorMaker when used the old Scala consumer.
- Upgrading your Streams application from 0.10.1 to 0.10.2 does not require a broker upgrade. A Kafka Streams 0.10.2 application can connect to 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).
- The Zookeeper dependency was removed from the Streams API. The Streams API now uses the Kafka protocol to manage internal topics instead of modifying Zookeeper directly. This eliminates the need for privileges to access Zookeeper directly and "StreamsConfig.ZOOKEEPER_CONFIG" should not be set in the Streams app any more. If the Kafka cluster is secured, Streams apps must have the required security privileges to create new topics.
- Several new fields including "security.protocol", "connections.max.idle.ms", "retry.backoff.ms", "reconnect.backoff.ms" and "request.timeout.ms" were added to StreamsConfig class. User should pay attention to the default values and set these if needed. For more details please refer to 3.5 Kafka Streams Configs.
New Protocol Versions
- KIP-88: OffsetFetchRequest v2 supports retrieval of offsets for all topics if the
topics
array is set tonull
. - KIP-88: OffsetFetchResponse v2 introduces a top-level
error_code
field. - KIP-103: UpdateMetadataRequest v3 introduces a
listener_name
field to the elements of theend_points
array. - KIP-108: CreateTopicsRequest v1 introduces a
validate_only
field. - KIP-108: CreateTopicsResponse v1 introduces an
error_message
field to the elements of thetopic_errors
array.
Upgrading from 0.8.x, 0.9.x or 0.10.0.X to 0.10.1.0
0.10.1.0 has wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please notice the Potential breaking changes in 0.10.1.0 before upgrade.Note: Because new protocols are introduced, it is important to upgrade your Kafka clusters before upgrading your clients (i.e. 0.10.1.x clients only support 0.10.1.x or later brokers while 0.10.1.x brokers also support older clients).
For a rolling upgrade:
- Update server.properties file on all brokers and add the following properties:
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2.0, 0.9.0.0 or 0.10.0.0).
- log.message.format.version=CURRENT_KAFKA_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.
- Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 0.10.1.0.
- If your previous message format is 0.10.0, change log.message.format.version to 0.10.1 (this is a no-op as the message format is the same for both 0.10.0 and 0.10.1). If your previous message format version is lower than 0.10.0, do not change log.message.format.version yet - this parameter should only change once all consumers have been upgraded to 0.10.0.0 or later.
- Restart the brokers one by one for the new protocol version to take effect.
- If log.message.format.version is still lower than 0.10.0 at this point, wait until all consumers have been upgraded to 0.10.0 or later, then change log.message.format.version to 0.10.1 on each broker and restart them one by one.
Note: If you are willing to accept downtime, you can simply take all the brokers down, update the code and start all of them. They will start with the new protocol by default.
Note: Bumping the protocol version and restarting can be done any time after the brokers were upgraded. It does not have to be immediately after.
Notable changes in 0.10.1.2
- New configuration parameter
upgrade.from
added that allows rolling bounce upgrade from version 0.10.0.x
Potential breaking changes in 0.10.1.0
- The log retention time is no longer based on last modified time of the log segments. Instead it will be based on the largest timestamp of the messages in a log segment.
- The log rolling time is no longer depending on log segment create time. Instead it is now based on the timestamp in the messages. More specifically. if the timestamp of the first message in the segment is T, the log will be rolled out when a new message has a timestamp greater than or equal to T + log.roll.ms
- The open file handlers of 0.10.0 will increase by ~33% because of the addition of time index files for each segment.
- The time index and offset index share the same index size configuration. Since each time index entry is 1.5x the size of offset index entry. User may need to increase log.index.size.max.bytes to avoid potential frequent log rolling.
- Due to the increased number of index files, on some brokers with large amount the log segments (e.g. >15K), the log loading process during the broker startup could be longer. Based on our experiment, setting the num.recovery.threads.per.data.dir to one may reduce the log loading time.
Upgrading a 0.10.0 Kafka Streams Application
- Upgrading your Streams application from 0.10.0 to 0.10.1 does require a broker upgrade because a Kafka Streams 0.10.1 application can only connect to 0.10.1 brokers.
- There are couple of API changes, that are not backward compatible (cf. Streams API changes in 0.10.1 for more details). Thus, you need to update and recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application.
- Upgrading from 0.10.0.x to 0.10.1.2 requires two rolling bounces with config
upgrade.from="0.10.0"
set for first upgrade phase (cf. KIP-268). As an alternative, an offline upgrade is also possible.- prepare your application instances for a rolling bounce and make sure that config
upgrade.from
is set to"0.10.0"
for new version 0.10.1.2 - bounce each instance of your application once
- prepare your newly deployed 0.10.1.2 application instances for a second round of rolling bounces; make sure to remove the value for config
upgrade.from
- bounce each instance of your application once more to complete the upgrade
- prepare your application instances for a rolling bounce and make sure that config
- Upgrading from 0.10.0.x to 0.10.1.0 or 0.10.1.1 requires an offline upgrade (rolling bounce upgrade is not supported)
- stop all old (0.10.0.x) application instances
- update your code and swap old code and jar file with new code and new jar file
- restart all new (0.10.1.0 or 0.10.1.1) application instances
Notable changes in 0.10.1.0
- The new Java consumer is no longer in beta and we recommend it for all new development. The old Scala consumers are still supported, but they will be deprecated in the next release and will be removed in a future major release.
- The
--new-consumer
/--new.consumer
switch is no longer required to use tools like MirrorMaker and the Console Consumer with the new consumer; one simply needs to pass a Kafka broker to connect to instead of the ZooKeeper ensemble. In addition, usage of the Console Consumer with the old consumer has been deprecated and it will be removed in a future major release. - Kafka clusters can now be uniquely identified by a cluster id. It will be automatically generated when a broker is upgraded to 0.10.1.0. The cluster id is available via the kafka.server:type=KafkaServer,name=ClusterId metric and it is part of the Metadata response. Serializers, client interceptors and metric reporters can receive the cluster id by implementing the ClusterResourceListener interface.
- The BrokerState "RunningAsController" (value 4) has been removed. Due to a bug, a broker would only be in this state briefly before transitioning out of it and hence the impact of the removal should be minimal. The recommended way to detect if a given broker is the controller is via the kafka.controller:type=KafkaController,name=ActiveControllerCount metric.
- The new Java Consumer now allows users to search offsets by timestamp on partitions.
- The new Java Consumer now supports heartbeating from a background thread. There is a new configuration
max.poll.interval.ms
which controls the maximum time between poll invocations before the consumer will proactively leave the group (5 minutes by default). The value of the configurationrequest.timeout.ms
(default to 30 seconds) must always be smaller thanmax.poll.interval.ms
(default to 5 minutes), since that is the maximum time that a JoinGroup request can block on the server while the consumer is rebalance. Finally, the default value ofsession.timeout.ms
has been adjusted down to 10 seconds, and the default value ofmax.poll.records
has been changed to 500. - When using an Authorizer and a user doesn't have Describe authorization on a topic, the broker will no longer return TOPIC_AUTHORIZATION_FAILED errors to requests since this leaks topic names. Instead, the UNKNOWN_TOPIC_OR_PARTITION error code will be returned. This may cause unexpected timeouts or delays when using the producer and consumer since Kafka clients will typically retry automatically on unknown topic errors. You should consult the client logs if you suspect this could be happening.
- Fetch responses have a size limit by default (50 MB for consumers and 10 MB for replication). The existing per partition limits also apply (1 MB for consumers and replication). Note that neither of these limits is an absolute maximum as explained in the next point.
- Consumers and replicas can make progress if a message larger than the response/partition size limit is found. More concretely, if the first message in the first non-empty partition of the fetch is larger than either or both limits, the message will still be returned.
- Overloaded constructors were added to
kafka.api.FetchRequest
andkafka.javaapi.FetchRequest
to allow the caller to specify the order of the partitions (since order is significant in v3). The previously existing constructors were deprecated and the partitions are shuffled before the request is sent to avoid starvation issues.
New Protocol Versions
- ListOffsetRequest v1 supports accurate offset search based on timestamps.
- MetadataResponse v2 introduces a new field: "cluster_id".
- FetchRequest v3 supports limiting the response size (in addition to the existing per partition limit), it returns messages bigger than the limits if required to make progress and the order of partitions in the request is now significant.
- JoinGroup v1 introduces a new field: "rebalance_timeout".
Upgrading from 0.8.x or 0.9.x to 0.10.0.0
0.10.0.0 has potential breaking changes (please review before upgrading) and possible performance impact following the upgrade. By following the recommended rolling upgrade plan below, you guarantee no downtime and no performance impact during and following the upgrade.
Note: Because new protocols are introduced, it is important to upgrade your Kafka clusters before upgrading your clients.
Notes to clients with version 0.9.0.0: Due to a bug introduced in 0.9.0.0, clients that depend on ZooKeeper (old Scala high-level Consumer and MirrorMaker if used with the old consumer) will not work with 0.10.0.x brokers. Therefore, 0.9.0.0 clients should be upgraded to 0.9.0.1 before brokers are upgraded to 0.10.0.x. This step is not necessary for 0.8.X or 0.9.0.1 clients.
For a rolling upgrade:
- Update server.properties file on all brokers and add the following properties:
- inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2 or 0.9.0.0).
- log.message.format.version=CURRENT_KAFKA_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)
- Upgrade the brokers. This can be done a broker at a time by simply bringing it down, updating the code, and restarting it.
- Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 0.10.0.0. NOTE: You shouldn't touch log.message.format.version yet - this parameter should only change once all consumers have been upgraded to 0.10.0.0
- Restart the brokers one by one for the new protocol version to take effect.
- Once all consumers have been upgraded to 0.10.0, change log.message.format.version to 0.10.0 on each broker and restart them one by one.
Note: If you are willing to accept downtime, you can simply take all the brokers down, update the code and start all of them. They will start with the new protocol by default.
Note: Bumping the protocol version and restarting can be done any time after the brokers were upgraded. It does not have to be immediately after.
Potential performance impact following upgrade to 0.10.0.0
The message format in 0.10.0 includes a new timestamp field and uses relative offsets for compressed messages. The on disk message format can be configured through log.message.format.version in the server.properties file. The default on-disk message format is 0.10.0. If a consumer client is on a version before 0.10.0.0, it only understands message formats before 0.10.0. In this case, the broker is able to convert messages from the 0.10.0 format to an earlier format before sending the response to the consumer on an older version. However, the broker can't use zero-copy transfer in this case. Reports from the Kafka community on the performance impact have shown CPU utilization going from 20% before to 100% after an upgrade, which forced an immediate upgrade of all clients to bring performance back to normal. To avoid such message conversion before consumers are upgraded to 0.10.0.0, one can set log.message.format.version to 0.8.2 or 0.9.0 when upgrading the broker to 0.10.0.0. This way, the broker can still use zero-copy transfer to send the data to the old consumers. Once consumers are upgraded, one can change the message format to 0.10.0 on the broker and enjoy the new message format that includes new timestamp and improved compression. The conversion is supported to ensure compatibility and can be useful to support a few apps that have not updated to newer clients yet, but is impractical to support all consumer traffic on even an overprovisioned cluster. Therefore, it is critical to avoid the message conversion as much as possible when brokers have been upgraded but the majority of clients have not.
For clients that are upgraded to 0.10.0.0, there is no performance impact.
Note: By setting the message format version, one certifies that all existing messages are on or below that message format version. Otherwise consumers before 0.10.0.0 might break. In particular, after the message format is set to 0.10.0, one should not change it back to an earlier format as it may break consumers on versions before 0.10.0.0.
Note: Due to the additional timestamp introduced in each message, producers sending small messages may see a message throughput degradation because of the increased overhead. Likewise, replication now transmits an additional 8 bytes per message. If you're running close to the network capacity of your cluster, it's possible that you'll overwhelm the network cards and see failures and performance issues due to the overload.
Note: If you have enabled compression on producers, you may notice reduced producer throughput and/or lower compression rate on the broker in some cases. When receiving compressed messages, 0.10.0 brokers avoid recompressing the messages, which in general reduces the latency and improves the throughput. In certain cases, however, this may reduce the batching size on the producer, which could lead to worse throughput. If this happens, users can tune linger.ms and batch.size of the producer for better throughput. In addition, the producer buffer used for compressing messages with snappy is smaller than the one used by the broker, which may have a negative impact on the compression ratio for the messages on disk. We intend to make this configurable in a future Kafka release.
Potential breaking changes in 0.10.0.0
- Starting from Kafka 0.10.0.0, the message format version in Kafka is represented as the Kafka version. For example, message format 0.9.0 refers to the highest message version supported by Kafka 0.9.0.
- Message format 0.10.0 has been introduced and it is used by default. It includes a timestamp field in the messages and relative offsets are used for compressed messages.
- ProduceRequest/Response v2 has been introduced and it is used by default to support message format 0.10.0
- FetchRequest/Response v2 has been introduced and it is used by default to support message format 0.10.0
- MessageFormatter interface was changed from
def writeTo(key: Array[Byte], value: Array[Byte], output: PrintStream)
todef writeTo(consumerRecord: ConsumerRecord[Array[Byte], Array[Byte]], output: PrintStream)
- MessageReader interface was changed from
def readMessage(): KeyedMessage[Array[Byte], Array[Byte]]
todef readMessage(): ProducerRecord[Array[Byte], Array[Byte]]
- MessageFormatter's package was changed from
kafka.tools
tokafka.common
- MessageReader's package was changed from
kafka.tools
tokafka.common
- MirrorMakerMessageHandler no longer exposes the
handle(record: MessageAndMetadata[Array[Byte], Array[Byte]])
method as it was never called. - The 0.7 KafkaMigrationTool is no longer packaged with Kafka. If you need to migrate from 0.7 to 0.10.0, please migrate to 0.8 first and then follow the documented upgrade process to upgrade from 0.8 to 0.10.0.
- The new consumer has standardized its APIs to accept
java.util.Collection
as the sequence type for method parameters. Existing code may have to be updated to work with the 0.10.0 client library. - LZ4-compressed message handling was changed to use an interoperable framing specification (LZ4f v1.5.1). To maintain compatibility with old clients, this change only applies to Message format 0.10.0 and later. Clients that Produce/Fetch LZ4-compressed messages using v0/v1 (Message format 0.9.0) should continue to use the 0.9.0 framing implementation. Clients that use Produce/Fetch protocols v2 or later should use interoperable LZ4f framing. A list of interoperable LZ4 libraries is available at https://www.lz4.org/
Notable changes in 0.10.0.0
- Starting from Kafka 0.10.0.0, a new client library named Kafka Streams is available for stream processing on data stored in Kafka topics. This new client library only works with 0.10.x and upward versioned brokers due to message format changes mentioned above. For more information please read Streams documentation.
- The default value of the configuration parameter
receive.buffer.bytes
is now 64K for the new consumer. - The new consumer now exposes the configuration parameter
exclude.internal.topics
to restrict internal topics (such as the consumer offsets topic) from accidentally being included in regular expression subscriptions. By default, it is enabled. - The old Scala producer has been deprecated. Users should migrate their code to the Java producer included in the kafka-clients JAR as soon as possible.
- The new consumer API has been marked stable.
Upgrading from 0.8.0, 0.8.1.X, or 0.8.2.X to 0.9.0.0
0.9.0.0 has potential breaking changes (please review before upgrading) and an inter-broker protocol change from previous versions. This means that upgraded brokers and clients may not be compatible with older versions. It is important that you upgrade your Kafka cluster before upgrading your clients. If you are using MirrorMaker downstream clusters should be upgraded first as well.For a rolling upgrade:
- Update server.properties file on all brokers and add the following property: inter.broker.protocol.version=0.8.2.X
- Upgrade the brokers. This can be done a broker at a time by simply bringing it down, updating the code, and restarting it.
- Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 0.9.0.0.
- Restart the brokers one by one for the new protocol version to take effect
Note: If you are willing to accept downtime, you can simply take all the brokers down, update the code and start all of them. They will start with the new protocol by default.
Note: Bumping the protocol version and restarting can be done any time after the brokers were upgraded. It does not have to be immediately after.
Potential breaking changes in 0.9.0.0
- Java 1.6 is no longer supported.
- Scala 2.9 is no longer supported.
- Broker IDs above 1000 are now reserved by default to automatically assigned broker IDs. If your cluster has existing broker IDs above that threshold make sure to increase the reserved.broker.max.id broker configuration property accordingly.
- Configuration parameter replica.lag.max.messages was removed. Partition leaders will no longer consider the number of lagging messages when deciding which replicas are in sync.
- Configuration parameter replica.lag.time.max.ms now refers not just to the time passed since last fetch request from replica, but also to time since the replica last caught up. Replicas that are still fetching messages from leaders but did not catch up to the latest messages in replica.lag.time.max.ms will be considered out of sync.
- Compacted topics no longer accept messages without key and an exception is thrown by the producer if this is attempted. In 0.8.x, a message without key would cause the log compaction thread to subsequently complain and quit (and stop compacting all compacted topics).
- MirrorMaker no longer supports multiple target clusters. As a result it will only accept a single --consumer.config parameter. To mirror multiple source clusters, you will need at least one MirrorMaker instance per source cluster, each with its own consumer configuration.
- Tools packaged under org.apache.kafka.clients.tools.* have been moved to org.apache.kafka.tools.*. All included scripts will still function as usual, only custom code directly importing these classes will be affected.
- The default Kafka JVM performance options (KAFKA_JVM_PERFORMANCE_OPTS) have been changed in kafka-run-class.sh.
- The kafka-topics.sh script (kafka.admin.TopicCommand) now exits with non-zero exit code on failure.
- The kafka-topics.sh script (kafka.admin.TopicCommand) will now print a warning when topic names risk metric collisions due to the use of a '.' or '_' in the topic name, and error in the case of an actual collision.
- The kafka-console-producer.sh script (kafka.tools.ConsoleProducer) will use the Java producer instead of the old Scala producer be default, and users have to specify 'old-producer' to use the old producer.
- By default, all command line tools will print all logging messages to stderr instead of stdout.
Notable changes in 0.9.0.1
- The new broker id generation feature can be disabled by setting broker.id.generation.enable to false.
- Configuration parameter log.cleaner.enable is now true by default. This means topics with a cleanup.policy=compact will now be compacted by default, and 128 MB of heap will be allocated to the cleaner process via log.cleaner.dedupe.buffer.size. You may want to review log.cleaner.dedupe.buffer.size and the other log.cleaner configuration values based on your usage of compacted topics.
- Default value of configuration parameter fetch.min.bytes for the new consumer is now 1 by default.
Deprecations in 0.9.0.0
- Altering topic configuration from the kafka-topics.sh script (kafka.admin.TopicCommand) has been deprecated. Going forward, please use the kafka-configs.sh script (kafka.admin.ConfigCommand) for this functionality.
- The kafka-consumer-offset-checker.sh (kafka.tools.ConsumerOffsetChecker) has been deprecated. Going forward, please use kafka-consumer-groups.sh (kafka.admin.ConsumerGroupCommand) for this functionality.
- The kafka.tools.ProducerPerformance class has been deprecated. Going forward, please use org.apache.kafka.tools.ProducerPerformance for this functionality (kafka-producer-perf-test.sh will also be changed to use the new class).
- The producer config block.on.buffer.full has been deprecated and will be removed in future release. Currently its default value has been changed to false. The KafkaProducer will no longer throw BufferExhaustedException but instead will use max.block.ms value to block, after which it will throw a TimeoutException. If block.on.buffer.full property is set to true explicitly, it will set the max.block.ms to Long.MAX_VALUE and metadata.fetch.timeout.ms will not be honoured
Upgrading from 0.8.1 to 0.8.2
0.8.2 is fully compatible with 0.8.1. The upgrade can be done one broker at a time by simply bringing it down, updating the code, and restarting it.Upgrading from 0.8.0 to 0.8.1
0.8.1 is fully compatible with 0.8. The upgrade can be done one broker at a time by simply bringing it down, updating the code, and restarting it.Upgrading from 0.7
Release 0.7 is incompatible with newer releases. Major changes were made to the API, ZooKeeper data structures, and protocol, and configuration in order to add replication (Which was missing in 0.7). The upgrade from 0.7 to later versions requires a special tool for migration. This migration can be done without downtime.1.6 Docker
JVM Based Apache Kafka Docker Image
Docker is a popular container runtime. Docker images for the JVM based Apache Kafka can be found on Docker Hub and are available from version 3.7.0.
Docker image can be pulled from Docker Hub using the following command:
$ docker pull apache/kafka:3.9.0
If you want to fetch the latest version of the Docker image use following command:
$ docker pull apache/kafka:latest
To start the Kafka container using this Docker image with default configs and on default port 9092:
$ docker run -p 9092:9092 apache/kafka:3.9.0
GraalVM Based Native Apache Kafka Docker Image
Docker images for the GraalVM Based Native Apache Kafka can be found on Docker Hub and are available from version 3.8.0.
NOTE: This image is experimental and intended for local development and testing purposes only; it is not recommended for production use.
Docker image can be pulled from Docker Hub using the following command:
$ docker pull apache/kafka-native:3.9.0
If you want to fetch the latest version of the Docker image use following command:
$ docker pull apache/kafka-native:latest
To start the Kafka container using this Docker image with default configs and on default port 9092:
$ docker run -p 9092:9092 apache/kafka-native:3.9.0
Usage guide
Detailed instructions for using the Docker image are mentioned here.
2. APIs
- The Producer API allows applications to send streams of data to topics in the Kafka cluster.
- The Consumer API allows applications to read streams of data from topics in the Kafka cluster.
- The Streams API allows transforming streams of data from input topics to output topics.
- The Connect API allows implementing connectors that continually pull from some source system or application into Kafka or push from Kafka into some sink system or application.
- The Admin API allows managing and inspecting topics, brokers, and other Kafka objects.
2.1 Producer API
The Producer API allows applications to send streams of data to topics in the Kafka cluster.Examples showing how to use the producer are given in the javadocs.
To use the producer, you can use the following maven dependency:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>3.9.0</version>
</dependency>
2.2 Consumer API
The Consumer API allows applications to read streams of data from topics in the Kafka cluster.Examples showing how to use the consumer are given in the javadocs.
To use the consumer, you can use the following maven dependency:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>3.9.0</version>
</dependency>
2.3 Streams API
The Streams API allows transforming streams of data from input topics to output topics.Examples showing how to use this library are given in the javadocs
Additional documentation on using the Streams API is available here.
To use Kafka Streams you can use the following maven dependency:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
<version>3.9.0</version>
</dependency>
When using Scala you may optionally include the kafka-streams-scala
library. Additional documentation on using the Kafka Streams DSL for Scala is available in the developer guide.
To use Kafka Streams DSL for Scala for Scala 2.13 you can use the following maven dependency:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams-scala_2.13</artifactId>
<version>3.9.0</version>
</dependency>
2.4 Connect API
The Connect API allows implementing connectors that continually pull from some source data system into Kafka or push from Kafka into some sink data system.Many users of Connect won't need to use this API directly, though, they can use pre-built connectors without needing to write any code. Additional information on using Connect is available here.
Those who want to implement custom connectors can see the javadoc.
2.5 Admin API
The Admin API supports managing and inspecting topics, brokers, acls, and other Kafka objects.To use the Admin API, add the following Maven dependency:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>3.9.0</version>
</dependency>
For more information about the Admin APIs, see the javadoc.
3. Configuration
3.1 Broker Configs
The essential configurations are the following:broker.id
log.dirs
zookeeper.connect
-
advertised.listeners
Listeners to publish to ZooKeeper for clients to use, if different than the
listeners
config property. In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, the value forlisteners
will be used. Unlikelisteners
, it is not valid to advertise the 0.0.0.0 meta-address.
Also unlikelisteners
, there can be duplicated ports in this property, so that one listener can be configured to advertise another listener's address. This can be useful in some cases where external load balancers are used.Type: string Default: null Valid Values: Importance: high Update Mode: per-broker -
auto.create.topics.enable
Enable auto creation of topic on the server.
Type: boolean Default: true Valid Values: Importance: high Update Mode: read-only -
auto.leader.rebalance.enable
Enables auto leader balancing. A background thread checks the distribution of partition leaders at regular intervals, configurable by leader.imbalance.check.interval.seconds. If the leader imbalance exceeds leader.imbalance.per.broker.percentage, leader rebalance to the preferred leader for partitions is triggered.
Type: boolean Default: true Valid Values: Importance: high Update Mode: read-only -
background.threads
The number of threads to use for various background processing tasks
Type: int Default: 10 Valid Values: [1,...] Importance: high Update Mode: cluster-wide -
broker.id
The broker id for this server. If unset, a unique broker id will be generated.To avoid conflicts between ZooKeeper generated broker id's and user configured broker id's, generated broker ids start from reserved.broker.max.id + 1.
Type: int Default: -1 Valid Values: Importance: high Update Mode: read-only -
compression.type
Specify the final compression type for a given topic. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'uncompressed' which is equivalent to no compression; and 'producer' which means retain the original compression codec set by the producer.
Type: string Default: producer Valid Values: [uncompressed, zstd, lz4, snappy, gzip, producer] Importance: high Update Mode: cluster-wide -
control.plane.listener.name
Name of listener used for communication between controller and brokers. A broker will use the
control.plane.listener.name
to locate the endpoint in listeners list, to listen for connections from the controller. For example, if a broker's config is:listeners = INTERNAL://192.1.1.8:9092, EXTERNAL://10.1.1.5:9093, CONTROLLER://192.1.1.8:9094listener.security.protocol.map = INTERNAL:PLAINTEXT, EXTERNAL:SSL, CONTROLLER:SSLcontrol.plane.listener.name = CONTROLLER
On startup, the broker will start listening on "192.1.1.8:9094" with security protocol "SSL".
On the controller side, when it discovers a broker's published endpoints through ZooKeeper, it will use thecontrol.plane.listener.name
to find the endpoint, which it will use to establish connection to the broker.
For example, if the broker's published endpoints on ZooKeeper are:
"endpoints" : ["INTERNAL://broker1.example.com:9092","EXTERNAL://broker1.example.com:9093","CONTROLLER://broker1.example.com:9094"]
and the controller's config is:listener.security.protocol.map = INTERNAL:PLAINTEXT, EXTERNAL:SSL, CONTROLLER:SSLcontrol.plane.listener.name = CONTROLLER
then the controller will use "broker1.example.com:9094" with security protocol "SSL" to connect to the broker.
If not explicitly configured, the default value will be null and there will be no dedicated endpoints for controller connections.
If explicitly configured, the value cannot be the same as the value ofinter.broker.listener.name
.Type: string Default: null Valid Values: Importance: high Update Mode: read-only -
controller.listener.names
A comma-separated list of the names of the listeners used by the controller. This is required if running in KRaft mode. When communicating with the controller quorum, the broker will always use the first listener in this list.
Note: The ZooKeeper-based controller should not set this configuration.Type: string Default: null Valid Values: Importance: high Update Mode: read-only -
controller.quorum.bootstrap.servers
List of endpoints to use for bootstrapping the cluster metadata. The endpoints are specified in comma-separated list of
{host}:{port}
entries. For example:localhost:9092,localhost:9093,localhost:9094
.Type: list Default: "" Valid Values: non-empty list Importance: high Update Mode: read-only -
controller.quorum.election.backoff.max.ms
Maximum time in milliseconds before starting new elections. This is used in the binary exponential backoff mechanism that helps prevent gridlocked elections
Type: int Default: 1000 (1 second) Valid Values: Importance: high Update Mode: read-only -
controller.quorum.election.timeout.ms
Maximum time in milliseconds to wait without being able to fetch from the leader before triggering a new election
Type: int Default: 1000 (1 second) Valid Values: Importance: high Update Mode: read-only -
controller.quorum.fetch.timeout.ms
Maximum time without a successful fetch from the current leader before becoming a candidate and triggering an election for voters; Maximum time a leader can go without receiving valid fetch or fetchSnapshot request from a majority of the quorum before resigning.
Type: int Default: 2000 (2 seconds) Valid Values: Importance: high Update Mode: read-only -
controller.quorum.voters
Map of id/endpoint information for the set of voters in a comma-separated list of
{id}@{host}:{port}
entries. For example:1@localhost:9092,2@localhost:9093,3@localhost:9094
Type: list Default: "" Valid Values: non-empty list Importance: high Update Mode: read-only -
delete.topic.enable
Enables delete topic. Delete topic through the admin tool will have no effect if this config is turned off
Type: boolean Default: true Valid Values: Importance: high Update Mode: read-only -
early.start.listeners
A comma-separated list of listener names which may be started before the authorizer has finished initialization. This is useful when the authorizer is dependent on the cluster itself for bootstrapping, as is the case for the StandardAuthorizer (which stores ACLs in the metadata log.) By default, all listeners included in controller.listener.names will also be early start listeners. A listener should not appear in this list if it accepts external traffic.
Type: string Default: null Valid Values: Importance: high Update Mode: read-only -
eligible.leader.replicas.enable
Enable the Eligible leader replicas
Type: boolean Default: false Valid Values: Importance: high Update Mode: read-only -
leader.imbalance.check.interval.seconds
The frequency with which the partition rebalance check is triggered by the controller
Type: long Default: 300 Valid Values: [1,...] Importance: high Update Mode: read-only -
leader.imbalance.per.broker.percentage
The ratio of leader imbalance allowed per broker. The controller would trigger a leader balance if it goes above this value per broker. The value is specified in percentage.
Type: int Default: 10 Valid Values: Importance: high Update Mode: read-only -
listeners
Listener List - Comma-separated list of URIs we will listen on and the listener names. If the listener name is not a security protocol,
listener.security.protocol.map
must also be set.
Listener names and port numbers must be unique unless %n one listener is an IPv4 address and the other listener is %n an IPv6 address (for the same port).%n Specify hostname as 0.0.0.0 to bind to all interfaces.%n Leave hostname empty to bind to default interface.%n Examples of legal listener lists:%nPLAINTEXT://myhost:9092,SSL://:9091
%nCLIENT://0.0.0.0:9092,REPLICATION://localhost:9093
%nPLAINTEXT://127.0.0.1:9092,SSL://[::1]:9092
%nType: string Default: PLAINTEXT://:9092 Valid Values: Importance: high Update Mode: per-broker -
log.dir
The directory in which the log data is kept (supplemental for log.dirs property)
Type: string Default: /tmp/kafka-logs Valid Values: Importance: high Update Mode: read-only -
log.dirs
A comma-separated list of the directories where the log data is stored. If not set, the value in log.dir is used.
Type: string Default: null Valid Values: Importance: high Update Mode: read-only -
log.flush.interval.messages
The number of messages accumulated on a log partition before messages are flushed to disk.
Type: long Default: 9223372036854775807 Valid Values: [1,...] Importance: high Update Mode: cluster-wide -
log.flush.interval.ms
The maximum time in ms that a message in any topic is kept in memory before flushed to disk. If not set, the value in log.flush.scheduler.interval.ms is used
Type: long Default: null Valid Values: Importance: high Update Mode: cluster-wide -
log.flush.offset.checkpoint.interval.ms
The frequency with which we update the persistent record of the last flush which acts as the log recovery point.
Type: int Default: 60000 (1 minute) Valid Values: [0,...] Importance: high Update Mode: read-only -
log.flush.scheduler.interval.ms
The frequency in ms that the log flusher checks whether any log needs to be flushed to disk
Type: long Default: 9223372036854775807 Valid Values: Importance: high Update Mode: read-only -
log.flush.start.offset.checkpoint.interval.ms
The frequency with which we update the persistent record of log start offset
Type: int Default: 60000 (1 minute) Valid Values: [0,...] Importance: high Update Mode: read-only -
log.retention.bytes
The maximum size of the log before deleting it
Type: long Default: -1 Valid Values: Importance: high Update Mode: cluster-wide -
log.retention.hours
The number of hours to keep a log file before deleting it (in hours), tertiary to log.retention.ms property
Type: int Default: 168 Valid Values: Importance: high Update Mode: read-only -
log.retention.minutes
The number of minutes to keep a log file before deleting it (in minutes), secondary to log.retention.ms property. If not set, the value in log.retention.hours is used
Type: int Default: null Valid Values: Importance: high Update Mode: read-only -
log.retention.ms
The number of milliseconds to keep a log file before deleting it (in milliseconds), If not set, the value in log.retention.minutes is used. If set to -1, no time limit is applied.
Type: long Default: null Valid Values: Importance: high Update Mode: cluster-wide -
log.roll.hours
The maximum time before a new log segment is rolled out (in hours), secondary to log.roll.ms property
Type: int Default: 168 Valid Values: [1,...] Importance: high Update Mode: read-only -
log.roll.jitter.hours
The maximum jitter to subtract from logRollTimeMillis (in hours), secondary to log.roll.jitter.ms property
Type: int Default: 0 Valid Values: [0,...] Importance: high Update Mode: read-only -
log.roll.jitter.ms
The maximum jitter to subtract from logRollTimeMillis (in milliseconds). If not set, the value in log.roll.jitter.hours is used
Type: long Default: null Valid Values: Importance: high Update Mode: cluster-wide -
log.roll.ms
The maximum time before a new log segment is rolled out (in milliseconds). If not set, the value in log.roll.hours is used
Type: long Default: null Valid Values: Importance: high Update Mode: cluster-wide -
log.segment.bytes
The maximum size of a single log file
Type: int Default: 1073741824 (1 gibibyte) Valid Values: [14,...] Importance: high Update Mode: cluster-wide -
log.segment.delete.delay.ms
The amount of time to wait before deleting a file from the filesystem. If the value is 0 and there is no file to delete, the system will wait 1 millisecond. Low value will cause busy waiting
Type: long Default: 60000 (1 minute) Valid Values: [0,...] Importance: high Update Mode: cluster-wide -
message.max.bytes
The largest record batch size allowed by Kafka (after compression if compression is enabled). If this is increased and there are consumers older than 0.10.2, the consumers' fetch size must also be increased so that they can fetch record batches this large. In the latest message format version, records are always grouped into batches for efficiency. In previous message format versions, uncompressed records are not grouped into batches and this limit only applies to a single record in that case.This can be set per topic with the topic level
max.message.bytes
config.Type: int Default: 1048588 Valid Values: [0,...] Importance: high Update Mode: cluster-wide -
metadata.log.dir
This configuration determines where we put the metadata log for clusters in KRaft mode. If it is not set, the metadata log is placed in the first log directory from log.dirs.
Type: string Default: null Valid Values: Importance: high Update Mode: read-only -
metadata.log.max.record.bytes.between.snapshots
This is the maximum number of bytes in the log between the latest snapshot and the high-watermark needed before generating a new snapshot. The default value is 20971520. To generate snapshots based on the time elapsed, see the
metadata.log.max.snapshot.interval.ms
configuration. The Kafka node will generate a snapshot when either the maximum time interval is reached or the maximum bytes limit is reached.Type: long Default: 20971520 Valid Values: [1,...] Importance: high Update Mode: read-only -
metadata.log.max.snapshot.interval.ms
This is the maximum number of milliseconds to wait to generate a snapshot if there are committed records in the log that are not included in the latest snapshot. A value of zero disables time based snapshot generation. The default value is 3600000. To generate snapshots based on the number of metadata bytes, see the
metadata.log.max.record.bytes.between.snapshots
configuration. The Kafka node will generate a snapshot when either the maximum time interval is reached or the maximum bytes limit is reached.Type: long Default: 3600000 (1 hour) Valid Values: [0,...] Importance: high Update Mode: read-only -
metadata.log.segment.bytes
The maximum size of a single metadata log file.
Type: int Default: 1073741824 (1 gibibyte) Valid Values: [12,...] Importance: high Update Mode: read-only -
metadata.log.segment.ms
The maximum time before a new metadata log file is rolled out (in milliseconds).
Type: long Default: 604800000 (7 days) Valid Values: Importance: high Update Mode: read-only -
metadata.max.retention.bytes
The maximum combined size of the metadata log and snapshots before deleting old snapshots and log files. Since at least one snapshot must exist before any logs can be deleted, this is a soft limit.
Type: long Default: 104857600 (100 mebibytes) Valid Values: Importance: high Update Mode: read-only -
metadata.max.retention.ms
The number of milliseconds to keep a metadata log file or snapshot before deleting it. Since at least one snapshot must exist before any logs can be deleted, this is a soft limit.
Type: long Default: 604800000 (7 days) Valid Values: Importance: high Update Mode: read-only -
min.insync.replicas
When a producer sets acks to "all" (or "-1"),
min.insync.replicas
specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this minimum cannot be met, then the producer will raise an exception (eitherNotEnoughReplicas
orNotEnoughReplicasAfterAppend
).
When used together,min.insync.replicas
and acks allow you to enforce greater durability guarantees. A typical scenario would be to create a topic with a replication factor of 3, setmin.insync.replicas
to 2, and produce with acks of "all". This will ensure that the producer raises an exception if a majority of replicas do not receive a write.Type: int Default: 1 Valid Values: [1,...] Importance: high Update Mode: cluster-wide -
node.id
The node ID associated with the roles this process is playing when
process.roles
is non-empty. This is required configuration when running in KRaft mode.Type: int Default: -1 Valid Values: Importance: high Update Mode: read-only -
num.io.threads
The number of threads that the server uses for processing requests, which may include disk I/O
Type: int Default: 8 Valid Values: [1,...] Importance: high Update Mode: cluster-wide -
num.network.threads
The number of threads that the server uses for receiving requests from the network and sending responses to the network. Noted: each listener (except for controller listener) creates its own thread pool.
Type: int Default: 3 Valid Values: [1,...] Importance: high Update Mode: cluster-wide -
num.recovery.threads.per.data.dir
The number of threads per data directory to be used for log recovery at startup and flushing at shutdown
Type: int Default: 1 Valid Values: [1,...] Importance: high Update Mode: cluster-wide -
num.replica.alter.log.dirs.threads
The number of threads that can move replicas between log directories, which may include disk I/O
Type: int Default: null Valid Values: Importance: high Update Mode: read-only -
num.replica.fetchers
Number of fetcher threads used to replicate records from each source broker. The total number of fetchers on each broker is bound by
num.replica.fetchers
multiplied by the number of brokers in the cluster.Increasing this value can increase the degree of I/O parallelism in the follower and leader broker at the cost of higher CPU and memory utilization.Type: int Default: 1 Valid Values: Importance: high Update Mode: cluster-wide -
offset.metadata.max.bytes
The maximum size for a metadata entry associated with an offset commit.
Type: int Default: 4096 (4 kibibytes) Valid Values: Importance: high Update Mode: read-only -
offsets.commit.required.acks
DEPRECATED: The required acks before the commit can be accepted. In general, the default (-1) should not be overridden.
Type: short Default: -1 Valid Values: Importance: high Update Mode: read-only -
offsets.commit.timeout.ms
Offset commit will be delayed until all replicas for the offsets topic receive the commit or this timeout is reached. This is similar to the producer request timeout.
Type: int Default: 5000 (5 seconds) Valid Values: [1,...] Importance: high Update Mode: read-only -
offsets.load.buffer.size
Batch size for reading from the offsets segments when loading offsets into the cache (soft-limit, overridden if records are too large).
Type: int Default: 5242880 Valid Values: [1,...] Importance: high Update Mode: read-only -
offsets.retention.check.interval.ms
Frequency at which to check for stale offsets
Type: long Default: 600000 (10 minutes) Valid Values: [1,...] Importance: high Update Mode: read-only -
offsets.retention.minutes
For subscribed consumers, committed offset of a specific partition will be expired and discarded when 1) this retention period has elapsed after the consumer group loses all its consumers (i.e. becomes empty); 2) this retention period has elapsed since the last time an offset is committed for the partition and the group is no longer subscribed to the corresponding topic. For standalone consumers (using manual assignment), offsets will be expired after this retention period has elapsed since the time of last commit. Note that when a group is deleted via the delete-group request, its committed offsets will also be deleted without extra retention period; also when a topic is deleted via the delete-topic request, upon propagated metadata update any group's committed offsets for that topic will also be deleted without extra retention period.
Type: int Default: 10080 Valid Values: [1,...] Importance: high Update Mode: read-only -
offsets.topic.compression.codec
Compression codec for the offsets topic - compression may be used to achieve "atomic" commits.
Type: int Default: 0 Valid Values: Importance: high Update Mode: read-only -
offsets.topic.num.partitions
The number of partitions for the offset commit topic (should not change after deployment).
Type: int Default: 50 Valid Values: [1,...] Importance: high Update Mode: read-only -
offsets.topic.replication.factor
The replication factor for the offsets topic (set higher to ensure availability). Internal topic creation will fail until the cluster size meets this replication factor requirement.
Type: short Default: 3 Valid Values: [1,...] Importance: high Update Mode: read-only -
offsets.topic.segment.bytes
The offsets topic segment bytes should be kept relatively small in order to facilitate faster log compaction and cache loads.
Type: int Default: 104857600 (100 mebibytes) Valid Values: [1,...] Importance: high Update Mode: read-only -
process.roles
The roles that this process plays: 'broker', 'controller', or 'broker,controller' if it is both. This configuration is only applicable for clusters in KRaft (Kafka Raft) mode (instead of ZooKeeper). Leave this config undefined or empty for ZooKeeper clusters.
Type: list Default: "" Valid Values: [broker, controller] Importance: high Update Mode: read-only -
queued.max.requests
The number of queued requests allowed for data-plane, before blocking the network threads
Type: int Default: 500 Valid Values: [1,...] Importance: high Update Mode: read-only -
replica.fetch.min.bytes
Minimum bytes expected for each fetch response. If not enough bytes, wait up to
replica.fetch.wait.max.ms
(broker config).Type: int Default: 1 Valid Values: Importance: high Update Mode: read-only -
replica.fetch.wait.max.ms
The maximum wait time for each fetcher request issued by follower replicas. This value should always be less than the replica.lag.time.max.ms at all times to prevent frequent shrinking of ISR for low throughput topics
Type: int Default: 500 Valid Values: Importance: high Update Mode: read-only -
replica.high.watermark.checkpoint.interval.ms
The frequency with which the high watermark is saved out to disk
Type: long Default: 5000 (5 seconds) Valid Values: Importance: high Update Mode: read-only -
replica.lag.time.max.ms
If a follower hasn't sent any fetch requests or hasn't consumed up to the leaders log end offset for at least this time, the leader will remove the follower from isr
Type: long Default: 30000 (30 seconds) Valid Values: Importance: high Update Mode: read-only -
replica.socket.receive.buffer.bytes
The socket receive buffer for network requests to the leader for replicating data
Type: int Default: 65536 (64 kibibytes) Valid Values: Importance: high Update Mode: read-only -
replica.socket.timeout.ms
The socket timeout for network requests. Its value should be at least replica.fetch.wait.max.ms
Type: int Default: 30000 (30 seconds) Valid Values: Importance: high Update Mode: read-only -
request.timeout.ms
The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.
Type: int Default: 30000 (30 seconds) Valid Values: Importance: high Update Mode: read-only -
sasl.mechanism.controller.protocol
SASL mechanism used for communication with controllers. Default is GSSAPI.
Type: string Default: GSSAPI Valid Values: Importance: high Update Mode: read-only -
socket.receive.buffer.bytes
The SO_RCVBUF buffer of the socket server sockets. If the value is -1, the OS default will be used.
Type: int Default: 102400 (100 kibibytes) Valid Values: Importance: high Update Mode: read-only -
socket.request.max.bytes
The maximum number of bytes in a socket request
Type: int Default: 104857600 (100 mebibytes) Valid Values: [1,...] Importance: high Update Mode: read-only -
socket.send.buffer.bytes
The SO_SNDBUF buffer of the socket server sockets. If the value is -1, the OS default will be used.
Type: int Default: 102400 (100 kibibytes) Valid Values: Importance: high Update Mode: read-only -
transaction.max.timeout.ms
The maximum allowed timeout for transactions. If a client’s requested transaction time exceed this, then the broker will return an error in InitProducerIdRequest. This prevents a client from too large of a timeout, which can stall consumers reading from topics included in the transaction.
Type: int Default: 900000 (15 minutes) Valid Values: [1,...] Importance: high Update Mode: read-only -
transaction.state.log.load.buffer.size
Batch size for reading from the transaction log segments when loading producer ids and transactions into the cache (soft-limit, overridden if records are too large).
Type: int Default: 5242880 Valid Values: [1,...] Importance: high Update Mode: read-only -
transaction.state.log.min.isr
The minimum number of replicas that must acknowledge a write to transaction topic in order to be considered successful.
Type: int Default: 2 Valid Values: [1,...] Importance: high Update Mode: read-only -
transaction.state.log.num.partitions
The number of partitions for the transaction topic (should not change after deployment).
Type: int Default: 50 Valid Values: [1,...] Importance: high Update Mode: read-only -
transaction.state.log.replication.factor
The replication factor for the transaction topic (set higher to ensure availability). Internal topic creation will fail until the cluster size meets this replication factor requirement.
Type: short Default: 3 Valid Values: [1,...] Importance: high Update Mode: read-only -
transaction.state.log.segment.bytes
The transaction topic segment bytes should be kept relatively small in order to facilitate faster log compaction and cache loads
Type: int Default: 104857600 (100 mebibytes) Valid Values: [1,...] Importance: high Update Mode: read-only -
transactional.id.expiration.ms
The time in ms that the transaction coordinator will wait without receiving any transaction status updates for the current transaction before expiring its transactional id. Transactional IDs will not expire while a the transaction is still ongoing.
Type: int Default: 604800000 (7 days) Valid Values: [1,...] Importance: high Update Mode: read-only -
unclean.leader.election.enable
Indicates whether to enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss
Note: In KRaft mode, when enabling this config dynamically, it needs to wait for the unclean leader election thread to trigger election periodically (default is 5 minutes). Please run `kafka-leader-election.sh` with `unclean` option to trigger the unclean leader election immediately if needed.
Type: boolean Default: false Valid Values: Importance: high Update Mode: cluster-wide -
zookeeper.connect
Specifies the ZooKeeper connection string in the form
hostname:port
where host and port are the host and port of a ZooKeeper server. To allow connecting through other ZooKeeper nodes when that ZooKeeper machine is down you can also specify multiple hosts in the formhostname1:port1,hostname2:port2,hostname3:port3
.
The server can also have a ZooKeeper chroot path as part of its ZooKeeper connection string which puts its data under some path in the global ZooKeeper namespace. For example to give a chroot path of/chroot/path
you would give the connection string ashostname1:port1,hostname2:port2,hostname3:port3/chroot/path
.Type: string Default: null Valid Values: Importance: high Update Mode: read-only -
zookeeper.connection.timeout.ms
The max time that the client waits to establish a connection to ZooKeeper. If not set, the value in zookeeper.session.timeout.ms is used
Type: int Default: null Valid Values: Importance: high Update Mode: read-only -
zookeeper.max.in.flight.requests
The maximum number of unacknowledged requests the client will send to ZooKeeper before blocking.
Type: int Default: 10 Valid Values: [1,...] Importance: high Update Mode: read-only -
zookeeper.metadata.migration.enable
Enable ZK to KRaft migration
Type: boolean Default: false Valid Values: Importance: high Update Mode: read-only -
zookeeper.session.timeout.ms
Zookeeper session timeout
Type: int Default: 18000 (18 seconds) Valid Values: Importance: high Update Mode: read-only -
zookeeper.set.acl
Set client to use secure ACLs
Type: boolean Default: false Valid Values: Importance: high Update Mode: read-only -
broker.heartbeat.interval.ms
The length of time in milliseconds between broker heartbeats. Used when running in KRaft mode.
Type: int Default: 2000 (2 seconds) Valid Values: Importance: medium Update Mode: read-only -
broker.id.generation.enable
Enable automatic broker id generation on the server. When enabled the value configured for reserved.broker.max.id should be reviewed.
Type: boolean Default: true Valid Values: Importance: medium Update Mode: read-only -
broker.rack
Rack of the broker. This will be used in rack aware replication assignment for fault tolerance. Examples:
RACK1
,us-east-1d
Type: string Default: null Valid Values: Importance: medium Update Mode: read-only -
broker.session.timeout.ms
The length of time in milliseconds that a broker lease lasts if no heartbeats are made. Used when running in KRaft mode.
Type: int Default: 9000 (9 seconds) Valid Values: Importance: medium Update Mode: read-only -
compression.gzip.level
The compression level to use if compression.type is set to 'gzip'.
Type: int Default: -1 Valid Values: [1,...,9] or -1 Importance: medium Update Mode: cluster-wide -
compression.lz4.level
The compression level to use if compression.type is set to 'lz4'.
Type: int Default: 9 Valid Values: [1,...,17] Importance: medium Update Mode: cluster-wide -
compression.zstd.level
The compression level to use if compression.type is set to 'zstd'.
Type: int Default: 3 Valid Values: [-131072,...,22] Importance: medium Update Mode: cluster-wide -
connections.max.idle.ms
Idle connections timeout: the server socket processor threads close the connections that idle more than this
Type: long Default: 600000 (10 minutes) Valid Values: Importance: medium Update Mode: read-only -
connections.max.reauth.ms
When explicitly set to a positive number (the default is 0, not a positive number), a session lifetime that will not exceed the configured value will be communicated to v2.2.0 or later clients when they authenticate. The broker will disconnect any such connection that is not re-authenticated within the session lifetime and that is then subsequently used for any purpose other than re-authentication. Configuration names can optionally be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.oauthbearer.connections.max.reauth.ms=3600000
Type: long Default: 0 Valid Values: Importance: medium Update Mode: read-only -
controlled.shutdown.enable
Enable controlled shutdown of the server.
Type: boolean Default: true Valid Values: Importance: medium Update Mode: read-only -
controlled.shutdown.max.retries
Controlled shutdown can fail for multiple reasons. This determines the number of retries when such failure happens
Type: int Default: 3 Valid Values: Importance: medium Update Mode: read-only -
controlled.shutdown.retry.backoff.ms
Before each retry, the system needs time to recover from the state that caused the previous failure (Controller fail over, replica lag etc). This config determines the amount of time to wait before retrying.
Type: long Default: 5000 (5 seconds) Valid Values: Importance: medium Update Mode: read-only -
controller.quorum.append.linger.ms
The duration in milliseconds that the leader will wait for writes to accumulate before flushing them to disk.
Type: int Default: 25 Valid Values: Importance: medium Update Mode: read-only -
controller.quorum.request.timeout.ms
The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.
Type: int Default: 2000 (2 seconds) Valid Values: Importance: medium Update Mode: read-only -
controller.socket.timeout.ms
The socket timeout for controller-to-broker channels.
Type: int Default: 30000 (30 seconds) Valid Values: Importance: medium Update Mode: read-only -
default.replication.factor
The replication factor for automatically created topics, and for topics created with -1 as the replication factor
Type: int Default: 1 Valid Values: Importance: medium Update Mode: read-only -
delegation.token.expiry.time.ms
The token validity time in milliseconds before the token needs to be renewed. Default value 1 day.
Type: long Default: 86400000 (1 day) Valid Values: [1,...] Importance: medium Update Mode: read-only -
delegation.token.master.key
DEPRECATED: An alias for delegation.token.secret.key, which should be used instead of this config.
Type: password Default: null Valid Values: Importance: medium Update Mode: read-only -
delegation.token.max.lifetime.ms
The token has a maximum lifetime beyond which it cannot be renewed anymore. Default value 7 days.
Type: long Default: 604800000 (7 days) Valid Values: [1,...] Importance: medium Update Mode: read-only -
delegation.token.secret.key
Secret key to generate and verify delegation tokens. The same key must be configured across all the brokers. If using Kafka with KRaft, the key must also be set across all controllers. If the key is not set or set to empty string, brokers will disable the delegation token support.
Type: password Default: null Valid Values: Importance: medium Update Mode: read-only -
delete.records.purgatory.purge.interval.requests
The purge interval (in number of requests) of the delete records request purgatory
Type: int Default: 1 Valid Values: Importance: medium Update Mode: read-only -
fetch.max.bytes
The maximum number of bytes we will return for a fetch request. Must be at least 1024.
Type: int Default: 57671680 (55 mebibytes) Valid Values: [1024,...] Importance: medium Update Mode: read-only -
fetch.purgatory.purge.interval.requests
The purge interval (in number of requests) of the fetch request purgatory
Type: int Default: 1000 Valid Values: Importance: medium Update Mode: read-only -
group.consumer.assignors
The server side assignors as a list of full class names. The first one in the list is considered as the default assignor to be used in the case where the consumer does not specify an assignor.
Type: list Default: org.apache.kafka.coordinator.group.assignor.UniformAssignor,org.apache.kafka.coordinator.group.assignor.RangeAssignor Valid Values: Importance: medium Update Mode: read-only -
group.consumer.heartbeat.interval.ms
The heartbeat interval given to the members of a consumer group.
Type: int Default: 5000 (5 seconds) Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.consumer.max.heartbeat.interval.ms
The maximum heartbeat interval for registered consumers.
Type: int Default: 15000 (15 seconds) Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.consumer.max.session.timeout.ms
The maximum allowed session timeout for registered consumers.
Type: int Default: 60000 (1 minute) Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.consumer.max.size
The maximum number of consumers that a single consumer group can accommodate. This value will only impact the new consumer coordinator. To configure the classic consumer coordinator check group.max.size instead.
Type: int Default: 2147483647 Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.consumer.min.heartbeat.interval.ms
The minimum heartbeat interval for registered consumers.
Type: int Default: 5000 (5 seconds) Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.consumer.min.session.timeout.ms
The minimum allowed session timeout for registered consumers.
Type: int Default: 45000 (45 seconds) Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.consumer.session.timeout.ms
The timeout to detect client failures when using the consumer group protocol.
Type: int Default: 45000 (45 seconds) Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.coordinator.append.linger.ms
The duration in milliseconds that the coordinator will wait for writes to accumulate before flushing them to disk. Transactional writes are not accumulated.
Type: int Default: 10 Valid Values: [0,...] Importance: medium Update Mode: read-only -
group.coordinator.rebalance.protocols
The list of enabled rebalance protocols. The consumer and share rebalance protocol are in early access and therefore must not be used in production.
Type: list Default: classic Valid Values: [consumer, classic, share] Importance: medium Update Mode: read-only -
group.coordinator.threads
The number of threads used by the group coordinator.
Type: int Default: 1 Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.initial.rebalance.delay.ms
The amount of time the group coordinator will wait for more consumers to join a new group before performing the first rebalance. A longer delay means potentially fewer rebalances, but increases the time until processing begins.
Type: int Default: 3000 (3 seconds) Valid Values: Importance: medium Update Mode: read-only -
group.max.session.timeout.ms
The maximum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures.
Type: int Default: 1800000 (30 minutes) Valid Values: Importance: medium Update Mode: read-only -
group.max.size
The maximum number of consumers that a single consumer group can accommodate.
Type: int Default: 2147483647 Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.min.session.timeout.ms
The minimum allowed session timeout for registered consumers. Shorter timeouts result in quicker failure detection at the cost of more frequent consumer heartbeating, which can overwhelm broker resources.
Type: int Default: 6000 (6 seconds) Valid Values: Importance: medium Update Mode: read-only -
group.share.delivery.count.limit
The maximum number of delivery attempts for a record delivered to a share group.
Type: int Default: 5 Valid Values: [2,...,10] Importance: medium Update Mode: read-only -
group.share.heartbeat.interval.ms
The heartbeat interval given to the members of a share group.
Type: int Default: 5000 (5 seconds) Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.share.max.groups
The maximum number of share groups.
Type: short Default: 10 Valid Values: [1,...,100] Importance: medium Update Mode: read-only -
group.share.max.heartbeat.interval.ms
The maximum heartbeat interval for share group members.
Type: int Default: 15000 (15 seconds) Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.share.max.record.lock.duration.ms
The record acquisition lock maximum duration in milliseconds for share groups.
Type: int Default: 60000 (1 minute) Valid Values: [30000,...,3600000] Importance: medium Update Mode: read-only -
group.share.max.session.timeout.ms
The maximum allowed session timeout for share group members.
Type: int Default: 60000 (1 minute) Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.share.max.size
The maximum number of members that a single share group can accommodate.
Type: short Default: 200 Valid Values: [10,...,1000] Importance: medium Update Mode: read-only -
group.share.min.heartbeat.interval.ms
The minimum heartbeat interval for share group members.
Type: int Default: 5000 (5 seconds) Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.share.min.record.lock.duration.ms
The record acquisition lock minimum duration in milliseconds for share groups.
Type: int Default: 15000 (15 seconds) Valid Values: [1000,...,30000] Importance: medium Update Mode: read-only -
group.share.min.session.timeout.ms
The minimum allowed session timeout for share group members.
Type: int Default: 45000 (45 seconds) Valid Values: [1,...] Importance: medium Update Mode: read-only -
group.share.partition.max.record.locks
Share-group record lock limit per share-partition.
Type: int Default: 200 Valid Values: [100,...,10000] Importance: medium Update Mode: read-only -
group.share.record.lock.duration.ms
The record acquisition lock duration in milliseconds for share groups.
Type: int Default: 30000 (30 seconds) Valid Values: [1000,...,60000] Importance: medium Update Mode: read-only -
group.share.session.timeout.ms
The timeout to detect client failures when using the share group protocol.
Type: int Default: 45000 (45 seconds) Valid Values: [1,...] Importance: medium Update Mode: read-only -
initial.broker.registration.timeout.ms
When initially registering with the controller quorum, the number of milliseconds to wait before declaring failure and exiting the broker process.
Type: int Default: 60000 (1 minute) Valid Values: Importance: medium Update Mode: read-only -
inter.broker.listener.name
Name of listener used for communication between brokers. If this is unset, the listener name is defined by security.inter.broker.protocolIt is an error to set this and security.inter.broker.protocol properties at the same time.
Type: string Default: null Valid Values: Importance: medium Update Mode: read-only -
inter.broker.protocol.version
Specify which version of the inter-broker protocol will be used.
. This is typically bumped after all brokers were upgraded to a new version.
Example of some valid values are: 0.8.0, 0.8.1, 0.8.1.1, 0.8.2, 0.8.2.0, 0.8.2.1, 0.9.0.0, 0.9.0.1 Check MetadataVersion for the full list.Type: string Default: 3.9-IV0 Valid Values: [0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.10.0-IV0, 0.10.0-IV1, 0.10.1-IV0, 0.10.1-IV1, 0.10.1-IV2, 0.10.2-IV0, 0.11.0-IV0, 0.11.0-IV1, 0.11.0-IV2, 1.0-IV0, 1.1-IV0, 2.0-IV0, 2.0-IV1, 2.1-IV0, 2.1-IV1, 2.1-IV2, 2.2-IV0, 2.2-IV1, 2.3-IV0, 2.3-IV1, 2.4-IV0, 2.4-IV1, 2.5-IV0, 2.6-IV0, 2.7-IV0, 2.7-IV1, 2.7-IV2, 2.8-IV0, 2.8-IV1, 3.0-IV0, 3.0-IV1, 3.1-IV0, 3.2-IV0, 3.3-IV0, 3.3-IV1, 3.3-IV2, 3.3-IV3, 3.4-IV0, 3.5-IV0, 3.5-IV1, 3.5-IV2, 3.6-IV0, 3.6-IV1, 3.6-IV2, 3.7-IV0, 3.7-IV1, 3.7-IV2, 3.7-IV3, 3.7-IV4, 3.8-IV0, 3.9-IV0, 4.0-IV0, 4.0-IV1] Importance: medium Update Mode: read-only -
log.cleaner.backoff.ms
The amount of time to sleep when there are no logs to clean
Type: long Default: 15000 (15 seconds) Valid Values: [0,...] Importance: medium Update Mode: cluster-wide -
log.cleaner.dedupe.buffer.size
The total memory used for log deduplication across all cleaner threads
Type: long Default: 134217728 Valid Values: Importance: medium Update Mode: cluster-wide -
log.cleaner.delete.retention.ms
The amount of time to retain tombstone message markers for log compacted topics. This setting also gives a bound on the time in which a consumer must complete a read if they begin from offset 0 to ensure that they get a valid snapshot of the final stage (otherwise tombstones messages may be collected before a consumer completes their scan).
Type: long Default: 86400000 (1 day) Valid Values: [0,...] Importance: medium Update Mode: cluster-wide -
log.cleaner.enable
Enable the log cleaner process to run on the server. Should be enabled if using any topics with a cleanup.policy=compact including the internal offsets topic. If disabled those topics will not be compacted and continually grow in size.
Type: boolean Default: true Valid Values: Importance: medium Update Mode: read-only -
log.cleaner.io.buffer.load.factor
Log cleaner dedupe buffer load factor. The percentage full the dedupe buffer can become. A higher value will allow more log to be cleaned at once but will lead to more hash collisions
Type: double Default: 0.9 Valid Values: Importance: medium Update Mode: cluster-wide -
log.cleaner.io.buffer.size
The total memory used for log cleaner I/O buffers across all cleaner threads
Type: int Default: 524288 Valid Values: [0,...] Importance: medium Update Mode: cluster-wide -
log.cleaner.io.max.bytes.per.second
The log cleaner will be throttled so that the sum of its read and write i/o will be less than this value on average
Type: double Default: 1.7976931348623157E308 Valid Values: Importance: medium Update Mode: cluster-wide -
log.cleaner.max.compaction.lag.ms
The maximum time a message will remain ineligible for compaction in the log. Only applicable for logs that are being compacted.
Type: long Default: 9223372036854775807 Valid Values: [1,...] Importance: medium Update Mode: cluster-wide -
log.cleaner.min.cleanable.ratio
The minimum ratio of dirty log to total log for a log to eligible for cleaning. If the log.cleaner.max.compaction.lag.ms or the log.cleaner.min.compaction.lag.ms configurations are also specified, then the log compactor considers the log eligible for compaction as soon as either: (i) the dirty ratio threshold has been met and the log has had dirty (uncompacted) records for at least the log.cleaner.min.compaction.lag.ms duration, or (ii) if the log has had dirty (uncompacted) records for at most the log.cleaner.max.compaction.lag.ms period.
Type: double Default: 0.5 Valid Values: [0,...,1] Importance: medium Update Mode: cluster-wide -
log.cleaner.min.compaction.lag.ms
The minimum time a message will remain uncompacted in the log. Only applicable for logs that are being compacted.
Type: long Default: 0 Valid Values: [0,...] Importance: medium Update Mode: cluster-wide -
log.cleaner.threads
The number of background threads to use for log cleaning
Type: int Default: 1 Valid Values: [0,...] Importance: medium Update Mode: cluster-wide -
log.cleanup.policy
The default cleanup policy for segments beyond the retention window. A comma separated list of valid policies. Valid policies are: "delete" and "compact"
Type: list Default: delete Valid Values: [compact, delete] Importance: medium Update Mode: cluster-wide -
log.index.interval.bytes
The interval with which we add an entry to the offset index.
Type: int Default: 4096 (4 kibibytes) Valid Values: [0,...] Importance: medium Update Mode: cluster-wide -
log.index.size.max.bytes
The maximum size in bytes of the offset index
Type: int Default: 10485760 (10 mebibytes) Valid Values: [4,...] Importance: medium Update Mode: cluster-wide -
log.local.retention.bytes
The maximum size of local log segments that can grow for a partition before it gets eligible for deletion. Default value is -2, it represents `log.retention.bytes` value to be used. The effective value should always be less than or equal to `log.retention.bytes` value.
Type: long Default: -2 Valid Values: [-2,...] Importance: medium Update Mode: cluster-wide -
log.local.retention.ms
The number of milliseconds to keep the local log segments before it gets eligible for deletion. Default value is -2, it represents `log.retention.ms` value is to be used. The effective value should always be less than or equal to `log.retention.ms` value.
Type: long Default: -2 Valid Values: [-2,...] Importance: medium Update Mode: cluster-wide -
log.message.format.version
Specify the message format version the broker will use to append messages to the logs. The value should be a valid MetadataVersion. Some examples are: 0.8.2, 0.9.0.0, 0.10.0, check MetadataVersion for more details. By setting a particular message format version, the user is certifying that all the existing messages on disk are smaller or equal than the specified version. Setting this value incorrectly will cause consumers with older versions to break as they will receive messages with a format that they don't understand.
Type: string Default: 3.0-IV1 Valid Values: [0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.10.0-IV0, 0.10.0-IV1, 0.10.1-IV0, 0.10.1-IV1, 0.10.1-IV2, 0.10.2-IV0, 0.11.0-IV0, 0.11.0-IV1, 0.11.0-IV2, 1.0-IV0, 1.1-IV0, 2.0-IV0, 2.0-IV1, 2.1-IV0, 2.1-IV1, 2.1-IV2, 2.2-IV0, 2.2-IV1, 2.3-IV0, 2.3-IV1, 2.4-IV0, 2.4-IV1, 2.5-IV0, 2.6-IV0, 2.7-IV0, 2.7-IV1, 2.7-IV2, 2.8-IV0, 2.8-IV1, 3.0-IV0, 3.0-IV1, 3.1-IV0, 3.2-IV0, 3.3-IV0, 3.3-IV1, 3.3-IV2, 3.3-IV3, 3.4-IV0, 3.5-IV0, 3.5-IV1, 3.5-IV2, 3.6-IV0, 3.6-IV1, 3.6-IV2, 3.7-IV0, 3.7-IV1, 3.7-IV2, 3.7-IV3, 3.7-IV4, 3.8-IV0, 3.9-IV0, 4.0-IV0, 4.0-IV1] Importance: medium Update Mode: read-only -
log.message.timestamp.after.max.ms
This configuration sets the allowable timestamp difference between the message timestamp and the broker's timestamp. The message timestamp can be later than or equal to the broker's timestamp, with the maximum allowable difference determined by the value set in this configuration. If log.message.timestamp.type=CreateTime, the message will be rejected if the difference in timestamps exceeds this specified threshold. This configuration is ignored if log.message.timestamp.type=LogAppendTime.
Type: long Default: 9223372036854775807 Valid Values: [0,...] Importance: medium Update Mode: cluster-wide -
log.message.timestamp.before.max.ms
This configuration sets the allowable timestamp difference between the broker's timestamp and the message timestamp. The message timestamp can be earlier than or equal to the broker's timestamp, with the maximum allowable difference determined by the value set in this configuration. If log.message.timestamp.type=CreateTime, the message will be rejected if the difference in timestamps exceeds this specified threshold. This configuration is ignored if log.message.timestamp.type=LogAppendTime.
Type: long Default: 9223372036854775807 Valid Values: [0,...] Importance: medium Update Mode: cluster-wide -
log.message.timestamp.difference.max.ms
[DEPRECATED] The maximum difference allowed between the timestamp when a broker receives a message and the timestamp specified in the message. If log.message.timestamp.type=CreateTime, a message will be rejected if the difference in timestamp exceeds this threshold. This configuration is ignored if log.message.timestamp.type=LogAppendTime.The maximum timestamp difference allowed should be no greater than log.retention.ms to avoid unnecessarily frequent log rolling.
Type: long Default: 9223372036854775807 Valid Values: [0,...] Importance: medium Update Mode: cluster-wide -
log.message.timestamp.type
Define whether the timestamp in the message is message create time or log append time. The value should be either
CreateTime
orLogAppendTime
.Type: string Default: CreateTime Valid Values: [CreateTime, LogAppendTime] Importance: medium Update Mode: cluster-wide -
log.preallocate
Should pre allocate file when create new segment? If you are using Kafka on Windows, you probably need to set it to true.
Type: boolean Default: false Valid Values: Importance: medium Update Mode: cluster-wide -
log.retention.check.interval.ms
The frequency in milliseconds that the log cleaner checks whether any log is eligible for deletion
Type: long Default: 300000 (5 minutes) Valid Values: [1,...] Importance: medium Update Mode: read-only -
max.connection.creation.rate
The maximum connection creation rate we allow in the broker at any time. Listener-level limits may also be configured by prefixing the config name with the listener prefix, for example,
listener.name.internal.max.connection.creation.rate
.Broker-wide connection rate limit should be configured based on broker capacity while listener limits should be configured based on application requirements. New connections will be throttled if either the listener or the broker limit is reached, with the exception of inter-broker listener. Connections on the inter-broker listener will be throttled only when the listener-level rate limit is reached.Type: int Default: 2147483647 Valid Values: [0,...] Importance: medium Update Mode: cluster-wide -
max.connections
The maximum number of connections we allow in the broker at any time. This limit is applied in addition to any per-ip limits configured using max.connections.per.ip. Listener-level limits may also be configured by prefixing the config name with the listener prefix, for example,
listener.name.internal.max.connections.per.ip
. Broker-wide limit should be configured based on broker capacity while listener limits should be configured based on application requirements. New connections are blocked if either the listener or broker limit is reached. Connections on the inter-broker listener are permitted even if broker-wide limit is reached. The least recently used connection on another listener will be closed in this case.Type: int Default: 2147483647 Valid Values: [0,...] Importance: medium Update Mode: cluster-wide -
max.connections.per.ip
The maximum number of connections we allow from each ip address. This can be set to 0 if there are overrides configured using max.connections.per.ip.overrides property. New connections from the ip address are dropped if the limit is reached.
Type: int Default: 2147483647 Valid Values: [0,...] Importance: medium Update Mode: cluster-wide -
max.connections.per.ip.overrides
A comma-separated list of per-ip or hostname overrides to the default maximum number of connections. An example value is "hostName:100,127.0.0.1:200"
Type: string Default: "" Valid Values: Importance: medium Update Mode: cluster-wide -
max.incremental.fetch.session.cache.slots
The maximum number of total incremental fetch sessions that we will maintain. FetchSessionCache is sharded into 8 shards and the limit is equally divided among all shards. Sessions are allocated to each shard in round-robin. Only entries within a shard are considered eligible for eviction.
Type: int Default: 1000 Valid Values: [0,...] Importance: medium Update Mode: read-only -
max.request.partition.size.limit
The maximum number of partitions can be served in one request.
Type: int Default: 2000 Valid Values: [1,...] Importance: medium Update Mode: read-only -
num.partitions
The default number of log partitions per topic
Type: int Default: 1 Valid Values: [1,...] Importance: medium Update Mode: read-only -
password.encoder.old.secret
The old secret that was used for encoding dynamically configured passwords. This is required only when the secret is updated. If specified, all dynamically encoded passwords are decoded using this old secret and re-encoded using password.encoder.secret when broker starts up.
Type: password Default: null Valid Values: Importance: medium Update Mode: read-only -
password.encoder.secret
The secret used for encoding dynamically configured passwords for this broker.
Type: password Default: null Valid Values: Importance: medium Update Mode: read-only -
principal.builder.class
The fully qualified name of a class that implements the KafkaPrincipalBuilder interface, which is used to build the KafkaPrincipal object used during authorization. If no principal builder is defined, the default behavior depends on the security protocol in use. For SSL authentication, the principal will be derived using the rules defined by
ssl.principal.mapping.rules
applied on the distinguished name from the client certificate if one is provided; otherwise, if client authentication is not required, the principal name will be ANONYMOUS. For SASL authentication, the principal will be derived using the rules defined bysasl.kerberos.principal.to.local.rules
if GSSAPI is in use, and the SASL authentication ID for other mechanisms. For PLAINTEXT, the principal will be ANONYMOUS.Type: class Default: org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder Valid Values: Importance: medium Update Mode: per-broker -
producer.purgatory.purge.interval.requests
The purge interval (in number of requests) of the producer request purgatory
Type: int Default: 1000 Valid Values: Importance: medium Update Mode: read-only -
queued.max.request.bytes
The number of queued bytes allowed before no more requests are read
Type: long Default: -1 Valid Values: Importance: medium Update Mode: read-only -
remote.fetch.max.wait.ms
The maximum amount of time the server will wait before answering the remote fetch request
Type: int Default: 500 Valid Values: [1,...] Importance: medium Update Mode: cluster-wide -
remote.log.manager.copier.thread.pool.size
Size of the thread pool used in scheduling tasks to copy segments. The default value of -1 means that this will be set to the configured value of remote.log.manager.thread.pool.size, if available; otherwise, it defaults to 10.
Type: int Default: -1 Valid Values: The default value of -1 means that this will be set to the configured value of remote.log.manager.thread.pool.size, if available; otherwise, it defaults to 10. Importance: medium Update Mode: read-only -
remote.log.manager.copy.max.bytes.per.second
The maximum number of bytes that can be copied from local storage to remote storage per second. This is a global limit for all the partitions that are being copied from local storage to remote storage. The default value is Long.MAX_VALUE, which means there is no limit on the number of bytes that can be copied per second.
Type: long Default: 9223372036854775807 Valid Values: [1,...] Importance: medium Update Mode: cluster-wide -
remote.log.manager.copy.quota.window.num
The number of samples to retain in memory for remote copy quota management. The default value is 11, which means there are 10 whole windows + 1 current window.
Type: int Default: 11 Valid Values: [1,...] Importance: medium Update Mode: read-only -
remote.log.manager.copy.quota.window.size.seconds
The time span of each sample for remote copy quota management. The default value is 1 second.
Type: int Default: 1 Valid Values: [1,...] Importance: medium Update Mode: read-only -
remote.log.manager.expiration.thread.pool.size
Size of the thread pool used in scheduling tasks to clean up remote log segments. The default value of -1 means that this will be set to the configured value of remote.log.manager.thread.pool.size, if available; otherwise, it defaults to 10.
Type: int Default: -1 Valid Values: The default value of -1 means that this will be set to the configured value of remote.log.manager.thread.pool.size, if available; otherwise, it defaults to 10. Importance: medium Update Mode: read-only -
remote.log.manager.fetch.max.bytes.per.second
The maximum number of bytes that can be fetched from remote storage to local storage per second. This is a global limit for all the partitions that are being fetched from remote storage to local storage. The default value is Long.MAX_VALUE, which means there is no limit on the number of bytes that can be fetched per second.
Type: long Default: 9223372036854775807 Valid Values: [1,...] Importance: medium Update Mode: cluster-wide -
remote.log.manager.fetch.quota.window.num
The number of samples to retain in memory for remote fetch quota management. The default value is 11, which means there are 10 whole windows + 1 current window.
Type: int Default: 11 Valid Values: [1,...] Importance: medium Update Mode: read-only -
remote.log.manager.fetch.quota.window.size.seconds
The time span of each sample for remote fetch quota management. The default value is 1 second.
Type: int Default: 1 Valid Values: [1,...] Importance: medium Update Mode: read-only -
remote.log.manager.thread.pool.size
Deprecated. Size of the thread pool used in scheduling tasks to copy segments, fetch remote log indexes and clean up remote log segments.
Type: int Default: 10 Valid Values: [1,...] Importance: medium Update Mode: read-only -
remote.log.metadata.manager.class.name
Fully qualified class name of `RemoteLogMetadataManager` implementation.
Type: string Default: org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager Valid Values: non-empty string Importance: medium Update Mode: read-only -
remote.log.metadata.manager.class.path
Class path of the `RemoteLogMetadataManager` implementation. If specified, the RemoteLogMetadataManager implementation and its dependent libraries will be loaded by a dedicated classloader which searches this class path before the Kafka broker class path. The syntax of this parameter is same as the standard Java class path string.
Type: string Default: null Valid Values: Importance: medium Update Mode: read-only -
remote.log.metadata.manager.impl.prefix
Prefix used for properties to be passed to RemoteLogMetadataManager implementation. For example this value can be `rlmm.config.`.
Type: string Default: rlmm.config. Valid Values: non-empty string Importance: medium Update Mode: read-only -
remote.log.metadata.manager.listener.name
Listener name of the local broker to which it should get connected if needed by RemoteLogMetadataManager implementation.
Type: string Default: null Valid Values: non-empty string Importance: medium Update Mode: read-only -
remote.log.reader.max.pending.tasks
Maximum remote log reader thread pool task queue size. If the task queue is full, fetch requests are served with an error.
Type: int Default: 100 Valid Values: [1,...] Importance: medium Update Mode: read-only -
remote.log.reader.threads
Size of the thread pool that is allocated for handling remote log reads.
Type: int Default: 10 Valid Values: [1,...] Importance: medium Update Mode: read-only -
remote.log.storage.manager.class.name
Fully qualified class name of `RemoteStorageManager` implementation.
Type: string Default: null Valid Values: non-empty string Importance: medium Update Mode: read-only -
remote.log.storage.manager.class.path
Class path of the `RemoteStorageManager` implementation. If specified, the RemoteStorageManager implementation and its dependent libraries will be loaded by a dedicated classloader which searches this class path before the Kafka broker class path. The syntax of this parameter is same as the standard Java class path string.
Type: string Default: null Valid Values: Importance: medium Update Mode: read-only -
remote.log.storage.manager.impl.prefix
Prefix used for properties to be passed to RemoteStorageManager implementation. For example this value can be `rsm.config.`.
Type: string Default: rsm.config. Valid Values: non-empty string Importance: medium Update Mode: read-only -
remote.log.storage.system.enable
Whether to enable tiered storage functionality in a broker or not. Valid values are `true` or `false` and the default value is false. When it is true broker starts all the services required for the tiered storage functionality.
Type: boolean Default: false Valid Values: Importance: medium Update Mode: read-only -
replica.fetch.backoff.ms
The amount of time to sleep when fetch partition error occurs.
Type: int Default: 1000 (1 second) Valid Values: [0,...] Importance: medium Update Mode: read-only -
replica.fetch.max.bytes
The number of bytes of messages to attempt to fetch for each partition. This is not an absolute maximum, if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. The maximum record batch size accepted by the broker is defined via
message.max.bytes
(broker config) ormax.message.bytes
(topic config).Type: int Default: 1048576 (1 mebibyte) Valid Values: [0,...] Importance: medium Update Mode: read-only -
replica.fetch.response.max.bytes
Maximum bytes expected for the entire fetch response. Records are fetched in batches, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined via
message.max.bytes
(broker config) ormax.message.bytes
(topic config).Type: int Default: 10485760 (10 mebibytes) Valid Values: [0,...] Importance: medium Update Mode: read-only -
replica.selector.class
The fully qualified class name that implements ReplicaSelector. This is used by the broker to find the preferred read replica. By default, we use an implementation that returns the leader.
Type: string Default: null Valid Values: Importance: medium Update Mode: read-only -
reserved.broker.max.id
Max number that can be used for a broker.id
Type: int Default: 1000 Valid Values: [0,...] Importance: medium Update Mode: read-only -
sasl.client.callback.handler.class
The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface.
Type: class Default: null Valid Values: Importance: medium Update Mode: read-only -
sasl.enabled.mechanisms
The list of SASL mechanisms enabled in the Kafka server. The list may contain any mechanism for which a security provider is available. Only GSSAPI is enabled by default.
Type: list Default: GSSAPI Valid Values: Importance: medium Update Mode: per-broker -
sasl.jaas.config
JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is:
loginModuleClass controlFlag (optionName=optionValue)*;
. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;Type: password Default: null Valid Values: Importance: medium Update Mode: per-broker -
sasl.kerberos.kinit.cmd
Kerberos kinit command path.
Type: string Default: /usr/bin/kinit Valid Values: Importance: medium Update Mode: per-broker -
sasl.kerberos.min.time.before.relogin
Login thread sleep time between refresh attempts.
Type: long Default: 60000 Valid Values: Importance: medium Update Mode: per-broker -
sasl.kerberos.principal.to.local.rules
A list of rules for mapping from principal names to short names (typically operating system usernames). The rules are evaluated in order and the first rule that matches a principal name is used to map it to a short name. Any later rules in the list are ignored. By default, principal names of the form
{username}/{hostname}@{REALM}
are mapped to{username}
. For more details on the format please see security authorization and acls. Note that this configuration is ignored if an extension ofKafkaPrincipalBuilder
is provided by theprincipal.builder.class
configuration.Type: list Default: DEFAULT Valid Values: Importance: medium Update Mode: per-broker -
sasl.kerberos.service.name
The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.
Type: string Default: null Valid Values: Importance: medium Update Mode: per-broker -
sasl.kerberos.ticket.renew.jitter
Percentage of random jitter added to the renewal time.
Type: double Default: 0.05 Valid Values: Importance: medium Update Mode: per-broker -
sasl.kerberos.ticket.renew.window.factor
Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.
Type: double Default: 0.8 Valid Values: Importance: medium Update Mode: per-broker -
sasl.login.callback.handler.class
The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler
Type: class Default: null Valid Values: Importance: medium Update Mode: read-only -
sasl.login.class
The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin
Type: class Default: null Valid Values: Importance: medium Update Mode: read-only -
sasl.login.refresh.buffer.seconds
The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.
Type: short Default: 300 Valid Values: Importance: medium Update Mode: per-broker -
sasl.login.refresh.min.period.seconds
The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.
Type: short Default: 60 Valid Values: Importance: medium Update Mode: per-broker -
sasl.login.refresh.window.factor
Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER.
Type: double Default: 0.8 Valid Values: Importance: medium Update Mode: per-broker -
sasl.login.refresh.window.jitter
The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.
Type: double Default: 0.05 Valid Values: Importance: medium Update Mode: per-broker -
sasl.mechanism.inter.broker.protocol
SASL mechanism used for inter-broker communication. Default is GSSAPI.
Type: string Default: GSSAPI Valid Values: Importance: medium Update Mode: per-broker -
sasl.oauthbearer.jwks.endpoint.url
The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. If the URL is HTTP(S)-based, the JWKS data will be retrieved from the OAuth/OIDC provider via the configured URL on broker startup. All then-current keys will be cached on the broker for incoming requests. If an authentication request is received for a JWT that includes a "kid" header claim value that isn't yet in the cache, the JWKS endpoint will be queried again on demand. However, the broker polls the URL every sasl.oauthbearer.jwks.endpoint.refresh.ms milliseconds to refresh the cache with any forthcoming keys before any JWT requests that include them are received. If the URL is file-based, the broker will load the JWKS file from a configured location on startup. In the event that the JWT includes a "kid" header value that isn't in the JWKS file, the broker will reject the JWT and authentication will fail.
Type: string Default: null Valid Values: Importance: medium Update Mode: read-only -
sasl.oauthbearer.token.endpoint.url
The URL for the OAuth/OIDC identity provider. If the URL is HTTP(S)-based, it is the issuer's token endpoint URL to which requests will be made to login based on the configuration in sasl.jaas.config. If the URL is file-based, it specifies a file containing an access token (in JWT serialized form) issued by the OAuth/OIDC identity provider to use for authorization.
Type: string Default: null Valid Values: Importance: medium Update Mode: read-only -
sasl.server.callback.handler.class
The fully qualified name of a SASL server callback handler class that implements the AuthenticateCallbackHandler interface. Server callback handlers must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.plain.sasl.server.callback.handler.class=com.example.CustomPlainCallbackHandler.
Type: class Default: null Valid Values: Importance: medium Update Mode: read-only -
sasl.server.max.receive.size
The maximum receive size allowed before and during initial SASL authentication. Default receive size is 512KB. GSSAPI limits requests to 64K, but we allow upto 512KB by default for custom SASL mechanisms. In practice, PLAIN, SCRAM and OAUTH mechanisms can use much smaller limits.
Type: int Default: 524288 Valid Values: Importance: medium Update Mode: read-only -
security.inter.broker.protocol
Security protocol used to communicate between brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL. It is an error to set this and inter.broker.listener.name properties at the same time.
Type: string Default: PLAINTEXT Valid Values: [PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL] Importance: medium Update Mode: read-only -
socket.connection.setup.timeout.max.ms
The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. To avoid connection storms, a randomization factor of 0.2 will be applied to the timeout resulting in a random range between 20% below and 20% above the computed value.
Type: long Default: 30000 (30 seconds) Valid Values: Importance: medium Update Mode: read-only -
socket.connection.setup.timeout.ms
The amount of time the client will wait for the socket connection to be established. If the connection is not built before the timeout elapses, clients will close the socket channel. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the
socket.connection.setup.timeout.max.ms
value.Type: long Default: 10000 (10 seconds) Valid Values: Importance: medium Update Mode: read-only -
socket.listen.backlog.size
The maximum number of pending connections on the socket. In Linux, you may also need to configure
somaxconn
andtcp_max_syn_backlog
kernel parameters accordingly to make the configuration takes effect.Type: int Default: 50 Valid Values: [1,...] Importance: medium Update Mode: read-only -
ssl.cipher.suites
A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.
Type: list Default: "" Valid Values: Importance: medium Update Mode: per-broker -
ssl.client.auth
Configures kafka broker to request client authentication. The following settings are common:
ssl.client.auth=required
If set to required client authentication is required.ssl.client.auth=requested
This means client authentication is optional. unlike required, if this option is set client can choose not to provide authentication information about itselfssl.client.auth=none
This means client authentication is not needed.
Type: string Default: none Valid Values: [required, requested, none] Importance: medium Update Mode: per-broker -
ssl.enabled.protocols
The list of protocols enabled for SSL connections. The default is 'TLSv1.2,TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). This default should be fine for most cases. Also see the config documentation for `ssl.protocol`.
Type: list Default: TLSv1.2 Valid Values: Importance: medium Update Mode: per-broker -
ssl.key.password
The password of the private key in the key store file or the PEM key specified in 'ssl.keystore.key'.
Type: password Default: null Valid Values: Importance: medium Update Mode: per-broker -
ssl.keymanager.algorithm
The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.
Type: string Default: SunX509 Valid Values: Importance: medium Update Mode: per-broker -
ssl.keystore.certificate.chain
Certificate chain in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with a list of X.509 certificates
Type: password Default: null Valid Values: Importance: medium Update Mode: per-broker -
ssl.keystore.key
Private key in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with PKCS#8 keys. If the key is encrypted, key password must be specified using 'ssl.key.password'
Type: password Default: null Valid Values: Importance: medium Update Mode: per-broker -
ssl.keystore.location
The location of the key store file. This is optional for client and can be used for two-way authentication for client.
Type: string Default: null Valid Values: Importance: medium Update Mode: per-broker -
ssl.keystore.password
The store password for the key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured. Key store password is not supported for PEM format.
Type: password Default: null Valid Values: Importance: medium Update Mode: per-broker -
ssl.keystore.type
The file format of the key store file. This is optional for client. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].
Type: string Default: JKS Valid Values: Importance: medium Update Mode: per-broker -
ssl.protocol
The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'.
Type: string Default: TLSv1.2 Valid Values: Importance: medium Update Mode: per-broker -
ssl.provider
The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.
Type: string Default: null Valid Values: Importance: medium Update Mode: per-broker -
ssl.trustmanager.algorithm
The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.
Type: string Default: PKIX Valid Values: Importance: medium Update Mode: per-broker -
ssl.truststore.certificates
Trusted certificates in the format specified by 'ssl.truststore.type'. Default SSL engine factory supports only PEM format with X.509 certificates.
Type: password Default: null Valid Values: Importance: medium Update Mode: per-broker -
ssl.truststore.location
The location of the trust store file.
Type: string Default: null Valid Values: Importance: medium Update Mode: per-broker -
ssl.truststore.password
The password for the trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled. Trust store password is not supported for PEM format.
Type: password Default: null Valid Values: Importance: medium Update Mode: per-broker -
ssl.truststore.type
The file format of the trust store file. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].
Type: string Default: JKS Valid Values: Importance: medium Update Mode: per-broker -
zookeeper.clientCnxnSocket
Typically set to
org.apache.zookeeper.ClientCnxnSocketNetty
when using TLS connectivity to ZooKeeper. Overrides any explicit value set via the same-namedzookeeper.clientCnxnSocket
system property.Type: string Default: null Valid Values: Importance: medium Update Mode: read-only -
zookeeper.ssl.client.enable
Set client to use TLS when connecting to ZooKeeper. An explicit value overrides any value set via the
zookeeper.client.secure
system property (note the different name). Defaults to false if neither is set; when true,zookeeper.clientCnxnSocket
must be set (typically toorg.apache.zookeeper.ClientCnxnSocketNetty
); other values to set may includezookeeper.ssl.cipher.suites
,zookeeper.ssl.crl.enable
,zookeeper.ssl.enabled.protocols
,zookeeper.ssl.endpoint.identification.algorithm
,zookeeper.ssl.keystore.location
,zookeeper.ssl.keystore.password
,zookeeper.ssl.keystore.type
,zookeeper.ssl.ocsp.enable
,zookeeper.ssl.protocol
,zookeeper.ssl.truststore.location
,zookeeper.ssl.truststore.password
,zookeeper.ssl.truststore.type
Type: boolean Default: false Valid Values: Importance: medium Update Mode: read-only -
zookeeper.ssl.keystore.location
Keystore location when using a client-side certificate with TLS connectivity to ZooKeeper. Overrides any explicit value set via the
zookeeper.ssl.keyStore.location
system property (note the camelCase).Type: string Default: null Valid Values: Importance: medium Update Mode: read-only -
zookeeper.ssl.keystore.password
Keystore password when using a client-side certificate with TLS connectivity to ZooKeeper. Overrides any explicit value set via the
zookeeper.ssl.keyStore.password
system property (note the camelCase). Note that ZooKeeper does not support a key password different from the keystore password, so be sure to set the key password in the keystore to be identical to the keystore password; otherwise the connection attempt to Zookeeper will fail.Type: password Default: null Valid Values: Importance: medium Update Mode: read-only -
zookeeper.ssl.keystore.type
Keystore type when using a client-side certificate with TLS connectivity to ZooKeeper. Overrides any explicit value set via the
zookeeper.ssl.keyStore.type
system property (note the camelCase). The default value ofnull
means the type will be auto-detected based on the filename extension of the keystore.Type: string Default: null Valid Values: Importance: medium Update Mode: read-only -
zookeeper.ssl.truststore.location
Truststore location when using TLS connectivity to ZooKeeper. Overrides any explicit value set via the
zookeeper.ssl.trustStore.location
system property (note the camelCase).Type: string Default: null Valid Values: Importance: medium Update Mode: read-only -
zookeeper.ssl.truststore.password
Truststore password when using TLS connectivity to ZooKeeper. Overrides any explicit value set via the
zookeeper.ssl.trustStore.password
system property (note the camelCase).Type: password Default: null Valid Values: Importance: medium Update Mode: read-only -
zookeeper.ssl.truststore.type
Truststore type when using TLS connectivity to ZooKeeper. Overrides any explicit value set via the
zookeeper.ssl.trustStore.type
system property (note the camelCase). The default value ofnull
means the type will be auto-detected based on the filename extension of the truststore.Type: string Default: null Valid Values: Importance: medium Update Mode: read-only -
alter.config.policy.class.name
The alter configs policy class that should be used for validation. The class should implement the
org.apache.kafka.server.policy.AlterConfigPolicy
interface.Type: class Default: null Valid Values: Importance: low Update Mode: read-only -
alter.log.dirs.replication.quota.window.num
The number of samples to retain in memory for alter log dirs replication quotas
Type: int Default: 11 Valid Values: [1,...] Importance: low Update Mode: read-only -
alter.log.dirs.replication.quota.window.size.seconds
The time span of each sample for alter log dirs replication quotas
Type: int Default: 1 Valid Values: [1,...] Importance: low Update Mode: read-only -
authorizer.class.name
The fully qualified name of a class that implements
org.apache.kafka.server.authorizer.Authorizer
interface, which is used by the broker for authorization.Type: string Default: "" Valid Values: non-null string Importance: low Update Mode: read-only -
auto.include.jmx.reporter
Deprecated. Whether to automatically include JmxReporter even if it's not listed in
metric.reporters
. This configuration will be removed in Kafka 4.0, users should instead includeorg.apache.kafka.common.metrics.JmxReporter
inmetric.reporters
in order to enable the JmxReporter.Type: boolean Default: true Valid Values: Importance: low Update Mode: read-only -
client.quota.callback.class
The fully qualified name of a class that implements the ClientQuotaCallback interface, which is used to determine quota limits applied to client requests. By default, the <user> and <client-id> quotas that are stored in ZooKeeper are applied. For any given request, the most specific quota that matches the user principal of the session and the client-id of the request is applied.
Type: class Default: null Valid Values: Importance: low Update Mode: read-only -
connection.failed.authentication.delay.ms
Connection close delay on failed authentication: this is the time (in milliseconds) by which connection close will be delayed on authentication failure. This must be configured to be less than connections.max.idle.ms to prevent connection timeout.
Type: int Default: 100 Valid Values: [0,...] Importance: low Update Mode: read-only -
controller.quorum.retry.backoff.ms
The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. This value is the initial backoff value and will increase exponentially for each failed request, up to the
retry.backoff.max.ms
value.Type: int Default: 20 Valid Values: Importance: low Update Mode: read-only -
controller.quota.window.num
The number of samples to retain in memory for controller mutation quotas
Type: int Default: 11 Valid Values: [1,...] Importance: low Update Mode: read-only -
controller.quota.window.size.seconds
The time span of each sample for controller mutations quotas
Type: int Default: 1 Valid Values: [1,...] Importance: low Update Mode: read-only -
create.topic.policy.class.name
The create topic policy class that should be used for validation. The class should implement the
org.apache.kafka.server.policy.CreateTopicPolicy
interface.Type: class Default: null Valid Values: Importance: low Update Mode: read-only -
delegation.token.expiry.check.interval.ms
Scan interval to remove expired delegation tokens.
Type: long Default: 3600000 (1 hour) Valid Values: [1,...] Importance: low Update Mode: read-only -
kafka.metrics.polling.interval.secs
The metrics polling interval (in seconds) which can be used in kafka.metrics.reporters implementations.
Type: int Default: 10 Valid Values: [1,...] Importance: low Update Mode: read-only -
kafka.metrics.reporters
A list of classes to use as Yammer metrics custom reporters. The reporters should implement
kafka.metrics.KafkaMetricsReporter
trait. If a client wants to expose JMX operations on a custom reporter, the custom reporter needs to additionally implement an MBean trait that extendskafka.metrics.KafkaMetricsReporterMBean
trait so that the registered MBean is compliant with the standard MBean convention.Type: list Default: "" Valid Values: Importance: low Update Mode: read-only -
listener.security.protocol.map
Map between listener names and security protocols. This must be defined for the same security protocol to be usable in more than one port or IP. For example, internal and external traffic can be separated even if SSL is required for both. Concretely, the user could define listeners with names INTERNAL and EXTERNAL and this property as:
INTERNAL:SSL,EXTERNAL:SSL
. As shown, key and value are separated by a colon and map entries are separated by commas. Each listener name should only appear once in the map. Different security (SSL and SASL) settings can be configured for each listener by adding a normalised prefix (the listener name is lowercased) to the config name. For example, to set a different keystore for the INTERNAL listener, a config with namelistener.name.internal.ssl.keystore.location
would be set. If the config for the listener name is not set, the config will fallback to the generic config (i.e.ssl.keystore.location
). Note that in KRaft a default mapping from the listener names defined bycontroller.listener.names
to PLAINTEXT is assumed if no explicit mapping is provided and no other security protocol is in use.Type: string Default: SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT Valid Values: Importance: low Update Mode: per-broker -
log.dir.failure.timeout.ms
If the broker is unable to successfully communicate to the controller that some log directory has failed for longer than this time, the broker will fail and shut down.
Type: long Default: 30000 (30 seconds) Valid Values: [1,...] Importance: low Update Mode: read-only -
log.message.downconversion.enable
This configuration controls whether down-conversion of message formats is enabled to satisfy consume requests. When set to
false
, broker will not perform down-conversion for consumers expecting an older message format. The broker responds withUNSUPPORTED_VERSION
error for consume requests from such older clients. This configurationdoes not apply to any message format conversion that might be required for replication to followers.Type: boolean Default: true Valid Values: Importance: low Update Mode: cluster-wide -
metadata.max.idle.interval.ms
This configuration controls how often the active controller should write no-op records to the metadata partition. If the value is 0, no-op records are not appended to the metadata partition. The default value is 500
Type: int Default: 500 Valid Values: [0,...] Importance: low Update Mode: read-only -
metric.reporters
A list of classes to use as metrics reporters. Implementing the
org.apache.kafka.common.metrics.MetricsReporter
interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.Type: list Default: "" Valid Values: Importance: low Update Mode: cluster-wide -
metrics.num.samples
The number of samples maintained to compute metrics.
Type: int Default: 2 Valid Values: [1,...] Importance: low Update Mode: read-only -
metrics.recording.level
The highest recording level for metrics.
Type: string Default: INFO Valid Values: Importance: low Update Mode: read-only -
metrics.sample.window.ms
The window of time a metrics sample is computed over.
Type: long Default: 30000 (30 seconds) Valid Values: [1,...] Importance: low Update Mode: read-only -
password.encoder.cipher.algorithm
The Cipher algorithm used for encoding dynamically configured passwords.
Type: string Default: AES/CBC/PKCS5Padding Valid Values: Importance: low Update Mode: read-only -
password.encoder.iterations
The iteration count used for encoding dynamically configured passwords.
Type: int Default: 4096 Valid Values: [1024,...] Importance: low Update Mode: read-only -
password.encoder.key.length
The key length used for encoding dynamically configured passwords.
Type: int Default: 128 Valid Values: [8,...] Importance: low Update Mode: read-only -
password.encoder.keyfactory.algorithm
The SecretKeyFactory algorithm used for encoding dynamically configured passwords. Default is PBKDF2WithHmacSHA512 if available and PBKDF2WithHmacSHA1 otherwise.
Type: string Default: null Valid Values: Importance: low Update Mode: read-only -
producer.id.expiration.ms
The time in ms that a topic partition leader will wait before expiring producer IDs. Producer IDs will not expire while a transaction associated to them is still ongoing. Note that producer IDs may expire sooner if the last write from the producer ID is deleted due to the topic's retention settings. Setting this value the same or higher than
delivery.timeout.ms
can help prevent expiration during retries and protect against message duplication, but the default should be reasonable for most use cases.Type: int Default: 86400000 (1 day) Valid Values: [1,...] Importance: low Update Mode: cluster-wide -
quota.window.num
The number of samples to retain in memory for client quotas
Type: int Default: 11 Valid Values: [1,...] Importance: low Update Mode: read-only -
quota.window.size.seconds
The time span of each sample for client quotas
Type: int Default: 1 Valid Values: [1,...] Importance: low Update Mode: read-only -
remote.log.index.file.cache.total.size.bytes
The total size of the space allocated to store index files fetched from remote storage in the local storage.
Type: long Default: 1073741824 (1 gibibyte) Valid Values: [1,...] Importance: low Update Mode: cluster-wide -
remote.log.manager.task.interval.ms
Interval at which remote log manager runs the scheduled tasks like copy segments, and clean up remote log segments.
Type: long Default: 30000 (30 seconds) Valid Values: [1,...] Importance: low Update Mode: read-only -
remote.log.metadata.custom.metadata.max.bytes
The maximum size of custom metadata in bytes that the broker should accept from a remote storage plugin. If custom metadata exceeds this limit, the updated segment metadata will not be stored, the copied data will be attempted to delete, and the remote copying task for this topic-partition will stop with an error.
Type: int Default: 128 Valid Values: [0,...] Importance: low Update Mode: read-only -
replication.quota.window.num
The number of samples to retain in memory for replication quotas
Type: int Default: 11 Valid Values: [1,...] Importance: low Update Mode: read-only -
replication.quota.window.size.seconds
The time span of each sample for replication quotas
Type: int Default: 1 Valid Values: [1,...] Importance: low Update Mode: read-only -
sasl.login.connect.timeout.ms
The (optional) value in milliseconds for the external authentication provider connection timeout. Currently applies only to OAUTHBEARER.
Type: int Default: null Valid Values: Importance: low Update Mode: read-only -
sasl.login.read.timeout.ms
The (optional) value in milliseconds for the external authentication provider read timeout. Currently applies only to OAUTHBEARER.
Type: int Default: null Valid Values: Importance: low Update Mode: read-only -
sasl.login.retry.backoff.max.ms
The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.
Type: long Default: 10000 (10 seconds) Valid Values: Importance: low Update Mode: read-only -
sasl.login.retry.backoff.ms
The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.
Type: long Default: 100 Valid Values: Importance: low Update Mode: read-only -
sasl.oauthbearer.clock.skew.seconds
The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker.
Type: int Default: 30 Valid Values: Importance: low Update Mode: read-only -
sasl.oauthbearer.expected.audience
The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWT's "aud" claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.
Type: list Default: null Valid Values: Importance: low Update Mode: read-only -
sasl.oauthbearer.expected.issuer
The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth "iss" claim and if this value is set, the broker will match it exactly against what is in the JWT's "iss" claim. If there is no match, the broker will reject the JWT and authentication will fail.
Type: string Default: null Valid Values: Importance: low Update Mode: read-only -
sasl.oauthbearer.jwks.endpoint.refresh.ms
The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.
Type: long Default: 3600000 (1 hour) Valid Values: Importance: low Update Mode: read-only -
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms
The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.
Type: long Default: 10000 (10 seconds) Valid Values: Importance: low Update Mode: read-only -
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms
The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.
Type: long Default: 100 Valid Values: Importance: low Update Mode: read-only -
sasl.oauthbearer.scope.claim.name
The OAuth claim for the scope is often named "scope", but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
Type: string Default: scope Valid Values: Importance: low Update Mode: read-only -
sasl.oauthbearer.sub.claim.name
The OAuth claim for the subject is often named "sub", but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
Type: string Default: sub Valid Values: Importance: low Update Mode: read-only -
security.providers
A list of configurable creator classes each returning a provider implementing security algorithms. These classes should implement the
org.apache.kafka.common.security.auth.SecurityProviderCreator
interface.Type: string Default: null Valid Values: Importance: low Update Mode: read-only -
ssl.allow.dn.changes
Indicates whether changes to the certificate distinguished name should be allowed during a dynamic reconfiguration of certificates or not.
Type: boolean Default: false Valid Values: Importance: low Update Mode: read-only -
ssl.allow.san.changes
Indicates whether changes to the certificate subject alternative names should be allowed during a dynamic reconfiguration of certificates or not.
Type: boolean Default: false Valid Values: Importance: low Update Mode: read-only -
ssl.endpoint.identification.algorithm
The endpoint identification algorithm to validate server hostname using server certificate.
Type: string Default: https Valid Values: Importance: low Update Mode: per-broker -
ssl.engine.factory.class
The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is org.apache.kafka.common.security.ssl.DefaultSslEngineFactory. Alternatively, setting this to org.apache.kafka.common.security.ssl.CommonNameLoggingSslEngineFactory will log the common name of expired SSL certificates used by clients to authenticate at any of the brokers with log level INFO. Note that this will cause a tiny delay during establishment of new connections from mTLS clients to brokers due to the extra code for examining the certificate chain provided by the client. Note further that the implementation uses a custom truststore based on the standard Java truststore and thus might be considered a security risk due to not being as mature as the standard one.
Type: class Default: null Valid Values: Importance: low Update Mode: per-broker -
ssl.principal.mapping.rules
A list of rules for mapping from distinguished name from the client certificate to short name. The rules are evaluated in order and the first rule that matches a principal name is used to map it to a short name. Any later rules in the list are ignored. By default, distinguished name of the X.500 certificate will be the principal. For more details on the format please see security authorization and acls. Note that this configuration is ignored if an extension of KafkaPrincipalBuilder is provided by the
principal.builder.class
configuration.Type: string Default: DEFAULT Valid Values: Importance: low Update Mode: read-only -
ssl.secure.random.implementation
The SecureRandom PRNG implementation to use for SSL cryptography operations.
Type: string Default: null Valid Values: Importance: low Update Mode: per-broker -
telemetry.max.bytes
The maximum size (after compression if compression is used) of telemetry metrics pushed from a client to the broker. The default value is 1048576 (1 MB).
Type: int Default: 1048576 (1 mebibyte) Valid Values: [1,...] Importance: low Update Mode: read-only -
transaction.abort.timed.out.transaction.cleanup.interval.ms
The interval at which to rollback transactions that have timed out
Type: int Default: 10000 (10 seconds) Valid Values: [1,...] Importance: low Update Mode: read-only -
transaction.partition.verification.enable
Enable verification that checks that the partition has been added to the transaction before writing transactional records to the partition
Type: boolean Default: true Valid Values: Importance: low Update Mode: cluster-wide -
transaction.remove.expired.transaction.cleanup.interval.ms
The interval at which to remove transactions that have expired due to
transactional.id.expiration.ms
passingType: int Default: 3600000 (1 hour) Valid Values: [1,...] Importance: low Update Mode: read-only -
zookeeper.ssl.cipher.suites
Specifies the enabled cipher suites to be used in ZooKeeper TLS negotiation (csv). Overrides any explicit value set via the
zookeeper.ssl.ciphersuites
system property (note the single word "ciphersuites"). The default value ofnull
means the list of enabled cipher suites is determined by the Java runtime being used.Type: list Default: null Valid Values: Importance: low Update Mode: read-only -
zookeeper.ssl.crl.enable
Specifies whether to enable Certificate Revocation List in the ZooKeeper TLS protocols. Overrides any explicit value set via the
zookeeper.ssl.crl
system property (note the shorter name).Type: boolean Default: false Valid Values: Importance: low Update Mode: read-only -
zookeeper.ssl.enabled.protocols
Specifies the enabled protocol(s) in ZooKeeper TLS negotiation (csv). Overrides any explicit value set via the
zookeeper.ssl.enabledProtocols
system property (note the camelCase). The default value ofnull
means the enabled protocol will be the value of thezookeeper.ssl.protocol
configuration property.Type: list Default: null Valid Values: Importance: low Update Mode: read-only -
zookeeper.ssl.endpoint.identification.algorithm
Specifies whether to enable hostname verification in the ZooKeeper TLS negotiation process, with (case-insensitively) "https" meaning ZooKeeper hostname verification is enabled and an explicit blank value meaning it is disabled (disabling it is only recommended for testing purposes). An explicit value overrides any "true" or "false" value set via the
zookeeper.ssl.hostnameVerification
system property (note the different name and values; true implies https and false implies blank).Type: string Default: HTTPS Valid Values: Importance: low Update Mode: read-only -
zookeeper.ssl.ocsp.enable
Specifies whether to enable Online Certificate Status Protocol in the ZooKeeper TLS protocols. Overrides any explicit value set via the
zookeeper.ssl.ocsp
system property (note the shorter name).Type: boolean Default: false Valid Values: Importance: low Update Mode: read-only -
zookeeper.ssl.protocol
Specifies the protocol to be used in ZooKeeper TLS negotiation. An explicit value overrides any value set via the same-named
zookeeper.ssl.protocol
system property.Type: string Default: TLSv1.2 Valid Values: Importance: low Update Mode: read-only
More details about broker configuration can be found in the scala class kafka.server.KafkaConfig
.
3.1.1 Updating Broker Configs
From Kafka version 1.1 onwards, some of the broker configs can be updated without restarting the broker. See theDynamic Update Mode
column in Broker Configs for the update mode of each broker config.
read-only
: Requires a broker restart for updateper-broker
: May be updated dynamically for each brokercluster-wide
: May be updated dynamically as a cluster-wide default. May also be updated as a per-broker value for testing.
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --add-config log.cleaner.threads=2
To describe the current dynamic broker configs for broker id 0:
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --describe
To delete a config override and revert to the statically configured or default value for broker id 0 (for example,
the number of log cleaner threads):
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --delete-config log.cleaner.threads
Some configs may be configured as a cluster-wide default to maintain consistent values across the whole cluster. All brokers
in the cluster will process the cluster default update. For example, to update log cleaner threads on all brokers:
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config log.cleaner.threads=2
To describe the currently configured dynamic cluster-wide default configs:
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --describe
All configs that are configurable at cluster level may also be configured at per-broker level (e.g. for testing).
If a config value is defined at different levels, the following order of precedence is used:
- Dynamic per-broker config stored in ZooKeeper
- Dynamic cluster-wide default config stored in ZooKeeper
- Static broker config from
server.properties
- Kafka default, see broker configs
Updating Password Configs Dynamically
Password config values that are dynamically updated are encrypted before storing in ZooKeeper. The broker config
password.encoder.secret
must be configured in server.properties
to enable dynamic update
of password configs. The secret may be different on different brokers.
The secret used for password encoding may be rotated with a rolling restart of brokers. The old secret used for encoding
passwords currently in ZooKeeper must be provided in the static broker config password.encoder.old.secret
and
the new secret must be provided in password.encoder.secret
. All dynamic password configs stored in ZooKeeper
will be re-encoded with the new secret when the broker starts up.
In Kafka 1.1.x, all dynamically updated password configs must be provided in every alter request when updating configs
using kafka-configs.sh
even if the password config is not being altered. This constraint will be removed in
a future release.
Updating Password Configs in ZooKeeper Before Starting Brokers
From Kafka 2.0.0 onwards,kafka-configs.sh
enables dynamic broker configs to be updated using ZooKeeper before
starting brokers for bootstrapping. This enables all password configs to be stored in encrypted form, avoiding the need for
clear passwords in server.properties
. The broker config password.encoder.secret
must also be specified
if any password configs are included in the alter command. Additional encryption parameters may also be specified. Password
encoder configs will not be persisted in ZooKeeper. For example, to store SSL key password for listener INTERNAL
on broker 0:
$ bin/kafka-configs.sh --zookeeper localhost:2182 --zk-tls-config-file zk_tls_config.properties --entity-type brokers --entity-name 0 --alter --add-config
'listener.name.internal.ssl.key.password=key-password,password.encoder.secret=secret,password.encoder.iterations=8192'
The configuration listener.name.internal.ssl.key.password
will be persisted in ZooKeeper in encrypted
form using the provided encoder configs. The encoder secret and iterations are not persisted in ZooKeeper.
Updating SSL Keystore of an Existing Listener
Brokers may be configured with SSL keystores with short validity periods to reduce the risk of compromised certificates. Keystores may be updated dynamically without restarting the broker. The config name must be prefixed with the listener prefixlistener.name.{listenerName}.
so that only the keystore config of a specific listener is updated.
The following configs may be updated in a single alter request at per-broker level:
ssl.keystore.type
ssl.keystore.location
ssl.keystore.password
ssl.key.password
Updating SSL Truststore of an Existing Listener
Broker truststores may be updated dynamically without restarting the broker to add or remove certificates. Updated truststore will be used to authenticate new client connections. The config name must be prefixed with the listener prefixlistener.name.{listenerName}.
so that only the truststore config of a specific listener
is updated. The following configs may be updated in a single alter request at per-broker level:
ssl.truststore.type
ssl.truststore.location
ssl.truststore.password
Updating Default Topic Configuration
Default topic configuration options used by brokers may be updated without broker restart. The configs are applied to topics without a topic config override for the equivalent per-topic config. One or more of these configs may be overridden at cluster-default level used by all brokers.log.segment.bytes
log.roll.ms
log.roll.hours
log.roll.jitter.ms
log.roll.jitter.hours
log.index.size.max.bytes
log.flush.interval.messages
log.flush.interval.ms
log.retention.bytes
log.retention.ms
log.retention.minutes
log.retention.hours
log.index.interval.bytes
log.cleaner.delete.retention.ms
log.cleaner.min.compaction.lag.ms
log.cleaner.max.compaction.lag.ms
log.cleaner.min.cleanable.ratio
log.cleanup.policy
log.segment.delete.delay.ms
unclean.leader.election.enable
min.insync.replicas
max.message.bytes
compression.type
log.preallocate
log.message.timestamp.type
log.message.timestamp.difference.max.ms
unclean.leader.election.enable
is dynamically updated.
In Kafka version 1.1.x, changes to unclean.leader.election.enable
take effect only when a new controller is elected.
Controller re-election may be forced by running:
$ bin/zookeeper-shell.sh localhost
rmr /controller
Updating Log Cleaner Configs
Log cleaner configs may be updated dynamically at cluster-default level used by all brokers. The changes take effect on the next iteration of log cleaning. One or more of these configs may be updated:log.cleaner.threads
log.cleaner.io.max.bytes.per.second
log.cleaner.dedupe.buffer.size
log.cleaner.io.buffer.size
log.cleaner.io.buffer.load.factor
log.cleaner.backoff.ms
Updating Thread Configs
The size of various thread pools used by the broker may be updated dynamically at cluster-default level used by all brokers. Updates are restricted to the rangecurrentSize / 2
to currentSize * 2
to ensure that config updates are
handled gracefully.
num.network.threads
num.io.threads
num.replica.fetchers
num.recovery.threads.per.data.dir
log.cleaner.threads
background.threads
Updating ConnectionQuota Configs
The maximum number of connections allowed for a given IP/host by the broker may be updated dynamically at cluster-default level used by all brokers. The changes will apply for new connection creations and the existing connections count will be taken into account by the new limits.max.connections.per.ip
max.connections.per.ip.overrides
Adding and Removing Listeners
Listeners may be added or removed dynamically. When a new listener is added, security configs of the listener must be provided
as listener configs with the listener prefix listener.name.{listenerName}.
. If the new listener uses SASL,
the JAAS configuration of the listener must be provided using the JAAS configuration property sasl.jaas.config
with the listener and mechanism prefix. See JAAS configuration for Kafka brokers for details.
In Kafka version 1.1.x, the listener used by the inter-broker listener may not be updated dynamically. To update the inter-broker
listener to a new listener, the new listener may be added on all brokers without restarting the broker. A rolling restart is then
required to update inter.broker.listener.name
.
listeners
advertised.listeners
listener.security.protocol.map
inter.broker.listener.name
or security.inter.broker.protocol
.
3.2 Topic-Level Configs
Configurations pertinent to topics have both a server default as well an optional per-topic override. If no per-topic configuration is given the server default is used. The override can be set at topic creation time by giving one or more--config
options. This example creates a topic named my-topic with a custom max message size and flush rate:
$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 1 \
--replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1
Overrides can also be changed or set later using the alter configs command. This example updates the max message size for my-topic:
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic
--alter --add-config max.message.bytes=128000
To check overrides set on the topic you can do
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic --describe
To remove an override you can do
$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic
--alter --delete-config max.message.bytes
The following are the topic-level configurations. The server's default configuration for this property is given under the Server Default Property heading. A given server default config value only applies to a topic if it does not have an explicit topic config override.
-
cleanup.policy
This config designates the retention policy to use on log segments. The "delete" policy (which is the default) will discard old segments when their retention time or size limit has been reached. The "compact" policy will enable log compaction, which retains the latest value for each key. It is also possible to specify both policies in a comma-separated list (e.g. "delete,compact"). In this case, old segments will be discarded per the retention time and size configuration, while retained segments will be compacted.
Type: list Default: delete Valid Values: [compact, delete] Server Default Property: log.cleanup.policy Importance: medium -
compression.gzip.level
The compression level to use if compression.type is set to
gzip
.Type: int Default: -1 Valid Values: [1,...,9] or -1 Server Default Property: compression.gzip.level Importance: medium -
compression.lz4.level
The compression level to use if compression.type is set to
lz4
.Type: int Default: 9 Valid Values: [1,...,17] Server Default Property: compression.lz4.level Importance: medium -
compression.type
Specify the final compression type for a given topic. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'uncompressed' which is equivalent to no compression; and 'producer' which means retain the original compression codec set by the producer.
Type: string Default: producer Valid Values: [uncompressed, zstd, lz4, snappy, gzip, producer] Server Default Property: compression.type Importance: medium -
compression.zstd.level
The compression level to use if compression.type is set to
zstd
.Type: int Default: 3 Valid Values: [-131072,...,22] Server Default Property: compression.zstd.level Importance: medium -
delete.retention.ms
The amount of time to retain delete tombstone markers for log compacted topics. This setting also gives a bound on the time in which a consumer must complete a read if they begin from offset 0 to ensure that they get a valid snapshot of the final stage (otherwise delete tombstones may be collected before they complete their scan).
Type: long Default: 86400000 (1 day) Valid Values: [0,...] Server Default Property: log.cleaner.delete.retention.ms Importance: medium -
file.delete.delay.ms
The time to wait before deleting a file from the filesystem
Type: long Default: 60000 (1 minute) Valid Values: [0,...] Server Default Property: log.segment.delete.delay.ms Importance: medium -
flush.messages
This setting allows specifying an interval at which we will force an fsync of data written to the log. For example if this was set to 1 we would fsync after every message; if it were 5 we would fsync after every five messages. In general we recommend you not set this and use replication for durability and allow the operating system's background flush capabilities as it is more efficient. This setting can be overridden on a per-topic basis (see the per-topic configuration section).
Type: long Default: 9223372036854775807 Valid Values: [1,...] Server Default Property: log.flush.interval.messages Importance: medium -
flush.ms
This setting allows specifying a time interval at which we will force an fsync of data written to the log. For example if this was set to 1000 we would fsync after 1000 ms had passed. In general we recommend you not set this and use replication for durability and allow the operating system's background flush capabilities as it is more efficient.
Type: long Default: 9223372036854775807 Valid Values: [0,...] Server Default Property: log.flush.interval.ms Importance: medium -
follower.replication.throttled.replicas
A list of replicas for which log replication should be throttled on the follower side. The list should describe a set of replicas in the form [PartitionId]:[BrokerId],[PartitionId]:[BrokerId]:... or alternatively the wildcard '*' can be used to throttle all replicas for this topic.
Type: list Default: "" Valid Values: [partitionId]:[brokerId],[partitionId]:[brokerId],... Server Default Property: null Importance: medium -
index.interval.bytes
This setting controls how frequently Kafka adds an index entry to its offset index. The default setting ensures that we index a message roughly every 4096 bytes. More indexing allows reads to jump closer to the exact position in the log but makes the index larger. You probably don't need to change this.
Type: int Default: 4096 (4 kibibytes) Valid Values: [0,...] Server Default Property: log.index.interval.bytes Importance: medium -
leader.replication.throttled.replicas
A list of replicas for which log replication should be throttled on the leader side. The list should describe a set of replicas in the form [PartitionId]:[BrokerId],[PartitionId]:[BrokerId]:... or alternatively the wildcard '*' can be used to throttle all replicas for this topic.
Type: list Default: "" Valid Values: [partitionId]:[brokerId],[partitionId]:[brokerId],... Server Default Property: null Importance: medium -
local.retention.bytes
The maximum size of local log segments that can grow for a partition before it deletes the old segments. Default value is -2, it represents `retention.bytes` value to be used. The effective value should always be less than or equal to `retention.bytes` value.
Type: long Default: -2 Valid Values: [-2,...] Server Default Property: log.local.retention.bytes Importance: medium -
local.retention.ms
The number of milliseconds to keep the local log segment before it gets deleted. Default value is -2, it represents `retention.ms` value is to be used. The effective value should always be less than or equal to `retention.ms` value.
Type: long Default: -2 Valid Values: [-2,...] Server Default Property: log.local.retention.ms Importance: medium -
max.compaction.lag.ms
The maximum time a message will remain ineligible for compaction in the log. Only applicable for logs that are being compacted.
Type: long Default: 9223372036854775807 Valid Values: [1,...] Server Default Property: log.cleaner.max.compaction.lag.ms Importance: medium -
max.message.bytes
The largest record batch size allowed by Kafka (after compression if compression is enabled). If this is increased and there are consumers older than 0.10.2, the consumers' fetch size must also be increased so that they can fetch record batches this large. In the latest message format version, records are always grouped into batches for efficiency. In previous message format versions, uncompressed records are not grouped into batches and this limit only applies to a single record in that case.
Type: int Default: 1048588 Valid Values: [0,...] Server Default Property: message.max.bytes Importance: medium -
message.format.version
[DEPRECATED] Specify the message format version the broker will use to append messages to the logs. The value of this config is always assumed to be `3.0` if `inter.broker.protocol.version` is 3.0 or higher (the actual config value is ignored). Otherwise, the value should be a valid ApiVersion. Some examples are: 0.10.0, 1.1, 2.8, 3.0. By setting a particular message format version, the user is certifying that all the existing messages on disk are smaller or equal than the specified version. Setting this value incorrectly will cause consumers with older versions to break as they will receive messages with a format that they don't understand.
Type: string Default: 3.0-IV1 Valid Values: [0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.10.0-IV0, 0.10.0-IV1, 0.10.1-IV0, 0.10.1-IV1, 0.10.1-IV2, 0.10.2-IV0, 0.11.0-IV0, 0.11.0-IV1, 0.11.0-IV2, 1.0-IV0, 1.1-IV0, 2.0-IV0, 2.0-IV1, 2.1-IV0, 2.1-IV1, 2.1-IV2, 2.2-IV0, 2.2-IV1, 2.3-IV0, 2.3-IV1, 2.4-IV0, 2.4-IV1, 2.5-IV0, 2.6-IV0, 2.7-IV0, 2.7-IV1, 2.7-IV2, 2.8-IV0, 2.8-IV1, 3.0-IV0, 3.0-IV1, 3.1-IV0, 3.2-IV0, 3.3-IV0, 3.3-IV1, 3.3-IV2, 3.3-IV3, 3.4-IV0, 3.5-IV0, 3.5-IV1, 3.5-IV2, 3.6-IV0, 3.6-IV1, 3.6-IV2, 3.7-IV0, 3.7-IV1, 3.7-IV2, 3.7-IV3, 3.7-IV4, 3.8-IV0, 3.9-IV0, 4.0-IV0, 4.0-IV1] Server Default Property: log.message.format.version Importance: medium -
message.timestamp.after.max.ms
This configuration sets the allowable timestamp difference between the message timestamp and the broker's timestamp. The message timestamp can be later than or equal to the broker's timestamp, with the maximum allowable difference determined by the value set in this configuration. If message.timestamp.type=CreateTime, the message will be rejected if the difference in timestamps exceeds this specified threshold. This configuration is ignored if message.timestamp.type=LogAppendTime.
Type: long Default: 9223372036854775807 Valid Values: [0,...] Server Default Property: log.message.timestamp.after.max.ms Importance: medium -
message.timestamp.before.max.ms
This configuration sets the allowable timestamp difference between the broker's timestamp and the message timestamp. The message timestamp can be earlier than or equal to the broker's timestamp, with the maximum allowable difference determined by the value set in this configuration. If message.timestamp.type=CreateTime, the message will be rejected if the difference in timestamps exceeds this specified threshold. This configuration is ignored if message.timestamp.type=LogAppendTime.
Type: long Default: 9223372036854775807 Valid Values: [0,...] Server Default Property: log.message.timestamp.before.max.ms Importance: medium -
message.timestamp.difference.max.ms
[DEPRECATED] The maximum difference allowed between the timestamp when a broker receives a message and the timestamp specified in the message. If message.timestamp.type=CreateTime, a message will be rejected if the difference in timestamp exceeds this threshold. This configuration is ignored if message.timestamp.type=LogAppendTime.
Type: long Default: 9223372036854775807 Valid Values: [0,...] Server Default Property: log.message.timestamp.difference.max.ms Importance: medium -
message.timestamp.type
Define whether the timestamp in the message is message create time or log append time. The value should be either `CreateTime` or `LogAppendTime`
Type: string Default: CreateTime Valid Values: [CreateTime, LogAppendTime] Server Default Property: log.message.timestamp.type Importance: medium -
min.cleanable.dirty.ratio
This configuration controls how frequently the log compactor will attempt to clean the log (assuming log compaction is enabled). By default we will avoid cleaning a log where more than 50% of the log has been compacted. This ratio bounds the maximum space wasted in the log by duplicates (at 50% at most 50% of the log could be duplicates). A higher ratio will mean fewer, more efficient cleanings but will mean more wasted space in the log. If the max.compaction.lag.ms or the min.compaction.lag.ms configurations are also specified, then the log compactor considers the log to be eligible for compaction as soon as either: (i) the dirty ratio threshold has been met and the log has had dirty (uncompacted) records for at least the min.compaction.lag.ms duration, or (ii) if the log has had dirty (uncompacted) records for at most the max.compaction.lag.ms period.
Type: double Default: 0.5 Valid Values: [0,...,1] Server Default Property: log.cleaner.min.cleanable.ratio Importance: medium -
min.compaction.lag.ms
The minimum time a message will remain uncompacted in the log. Only applicable for logs that are being compacted.
Type: long Default: 0 Valid Values: [0,...] Server Default Property: log.cleaner.min.compaction.lag.ms Importance: medium -
min.insync.replicas
When a producer sets acks to "all" (or "-1"), this configuration specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this minimum cannot be met, then the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend).
When used together,min.insync.replicas
andacks
allow you to enforce greater durability guarantees. A typical scenario would be to create a topic with a replication factor of 3, setmin.insync.replicas
to 2, and produce withacks
of "all". This will ensure that the producer raises an exception if a majority of replicas do not receive a write.Type: int Default: 1 Valid Values: [1,...] Server Default Property: min.insync.replicas Importance: medium -
preallocate
True if we should preallocate the file on disk when creating a new log segment.
Type: boolean Default: false Valid Values: Server Default Property: log.preallocate Importance: medium -
remote.log.copy.disable
Determines whether tiered data for a topic should become read only, and no more data uploading on a topic. Once this config is set to true, the local retention configuration (i.e. local.retention.ms/bytes) becomes irrelevant, and all data expiration follows the topic-wide retention configuration(i.e. retention.ms/bytes).
Type: boolean Default: false Valid Values: Server Default Property: null Importance: medium -
remote.log.delete.on.disable
Determines whether tiered data for a topic should be deleted after tiered storage is disabled on a topic. This configuration should be enabled when trying to set `remote.storage.enable` from true to false
Type: boolean Default: false Valid Values: Server Default Property: null Importance: medium -
remote.storage.enable
To enable tiered storage for a topic, set this configuration as true. You can not disable this config once it is enabled. It will be provided in future versions.
Type: boolean Default: false Valid Values: Server Default Property: null Importance: medium -
retention.bytes
This configuration controls the maximum size a partition (which consists of log segments) can grow to before we will discard old log segments to free up space if we are using the "delete" retention policy. By default there is no size limit only a time limit. Since this limit is enforced at the partition level, multiply it by the number of partitions to compute the topic retention in bytes. Additionally, retention.bytes configuration operates independently of "segment.ms" and "segment.bytes" configurations. Moreover, it triggers the rolling of new segment if the retention.bytes is configured to zero.
Type: long Default: -1 Valid Values: Server Default Property: log.retention.bytes Importance: medium -
retention.ms
This configuration controls the maximum time we will retain a log before we will discard old log segments to free up space if we are using the "delete" retention policy. This represents an SLA on how soon consumers must read their data. If set to -1, no time limit is applied. Additionally, retention.ms configuration operates independently of "segment.ms" and "segment.bytes" configurations. Moreover, it triggers the rolling of new segment if the retention.ms condition is satisfied.
Type: long Default: 604800000 (7 days) Valid Values: [-1,...] Server Default Property: log.retention.ms Importance: medium -
segment.bytes
This configuration controls the segment file size for the log. Retention and cleaning is always done a file at a time so a larger segment size means fewer files but less granular control over retention.
Type: int Default: 1073741824 (1 gibibyte) Valid Values: [14,...] Server Default Property: log.segment.bytes Importance: medium -
segment.index.bytes
This configuration controls the size of the index that maps offsets to file positions. We preallocate this index file and shrink it only after log rolls. You generally should not need to change this setting.
Type: int Default: 10485760 (10 mebibytes) Valid Values: [4,...] Server Default Property: log.index.size.max.bytes Importance: medium -
segment.jitter.ms
The maximum random jitter subtracted from the scheduled segment roll time to avoid thundering herds of segment rolling
Type: long Default: 0 Valid Values: [0,...] Server Default Property: log.roll.jitter.ms Importance: medium -
segment.ms
This configuration controls the period of time after which Kafka will force the log to roll even if the segment file isn't full to ensure that retention can delete or compact old data.
Type: long Default: 604800000 (7 days) Valid Values: [1,...] Server Default Property: log.roll.ms Importance: medium -
unclean.leader.election.enable
Indicates whether to enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss.
Note: In KRaft mode, when enabling this config dynamically, it needs to wait for the unclean leader electionthread to trigger election periodically (default is 5 minutes). Please run `kafka-leader-election.sh` with `unclean` option to trigger the unclean leader election immediately if needed.
Type: boolean Default: false Valid Values: Server Default Property: unclean.leader.election.enable Importance: medium -
message.downconversion.enable
This configuration controls whether down-conversion of message formats is enabled to satisfy consume requests. When set to
false
, broker will not perform down-conversion for consumers expecting an older message format. The broker responds withUNSUPPORTED_VERSION
error for consume requests from such older clients. This configurationdoes not apply to any message format conversion that might be required for replication to followers.Type: boolean Default: true Valid Values: Server Default Property: log.message.downconversion.enable Importance: low
3.3 Producer Configs
Below is the configuration of the producer:-
key.serializer
Serializer class for key that implements the
org.apache.kafka.common.serialization.Serializer
interface.Type: class Default: Valid Values: Importance: high -
value.serializer
Serializer class for value that implements the
org.apache.kafka.common.serialization.Serializer
interface.Type: class Default: Valid Values: Importance: high -
bootstrap.servers
A list of host/port pairs used to establish the initial connection to the Kafka cluster. Clients use this list to bootstrap and discover the full set of Kafka brokers. While the order of servers in the list does not matter, we recommend including more than one server to ensure resilience if any servers are down. This list does not need to contain the entire set of brokers, as Kafka clients automatically manage and update connections to the cluster efficiently. This list must be in the form
host1:port1,host2:port2,...
.Type: list Default: "" Valid Values: non-null string Importance: high -
buffer.memory
The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block for
max.block.ms
after which it will throw an exception.This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests.
Type: long Default: 33554432 Valid Values: [0,...] Importance: high -
compression.type
The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are
none
,gzip
,snappy
,lz4
, orzstd
. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression).Type: string Default: none Valid Values: [none, gzip, snappy, lz4, zstd] Importance: high -
retries
Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Produce requests will be failed before the number of retries has been exhausted if the timeout configured by
delivery.timeout.ms
expires first before successful acknowledgement. Users should generally prefer to leave this config unset and instead usedelivery.timeout.ms
to control retry behavior.Enabling idempotence requires this config value to be greater than 0. If conflicting configurations are set and idempotence is not explicitly enabled, idempotence is disabled.
Allowing retries while setting
enable.idempotence
tofalse
andmax.in.flight.requests.per.connection
to greater than 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first.Type: int Default: 2147483647 Valid Values: [0,...,2147483647] Importance: high -
ssl.key.password
The password of the private key in the key store file or the PEM key specified in 'ssl.keystore.key'.
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.certificate.chain
Certificate chain in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with a list of X.509 certificates
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.key
Private key in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with PKCS#8 keys. If the key is encrypted, key password must be specified using 'ssl.key.password'
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.location
The location of the key store file. This is optional for client and can be used for two-way authentication for client.
Type: string Default: null Valid Values: Importance: high -
ssl.keystore.password
The store password for the key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured. Key store password is not supported for PEM format.
Type: password Default: null Valid Values: Importance: high -
ssl.truststore.certificates
Trusted certificates in the format specified by 'ssl.truststore.type'. Default SSL engine factory supports only PEM format with X.509 certificates.
Type: password Default: null Valid Values: Importance: high -
ssl.truststore.location
The location of the trust store file.
Type: string Default: null Valid Values: Importance: high -
ssl.truststore.password
The password for the trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled. Trust store password is not supported for PEM format.
Type: password Default: null Valid Values: Importance: high -
batch.size
The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes.
No attempt will be made to batch records larger than this size.
Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent.
A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records.
Note: This setting gives the upper bound of the batch size to be sent. If we have fewer than this many bytes accumulated for this partition, we will 'linger' for the
linger.ms
time waiting for more records to show up. Thislinger.ms
setting defaults to 0, which means we'll immediately send out a record even the accumulated batch size is under thisbatch.size
setting.Type: int Default: 16384 Valid Values: [0,...] Importance: medium -
client.dns.lookup
Controls how the client uses DNS lookups. If set to
use_all_dns_ips
, connect to each returned IP address in sequence until a successful connection is established. After a disconnection, the next IP is used. Once all IPs have been used once, the client resolves the IP(s) from the hostname again (both the JVM and the OS cache DNS name lookups, however). If set toresolve_canonical_bootstrap_servers_only
, resolve each bootstrap address into a list of canonical names. After the bootstrap phase, this behaves the same asuse_all_dns_ips
.Type: string Default: use_all_dns_ips Valid Values: [use_all_dns_ips, resolve_canonical_bootstrap_servers_only] Importance: medium -
client.id
An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.
Type: string Default: "" Valid Values: Importance: medium -
compression.gzip.level
The compression level to use if compression.type is set to
gzip
.Type: int Default: -1 Valid Values: [1,...,9] or -1 Importance: medium -
compression.lz4.level
The compression level to use if compression.type is set to
lz4
.Type: int Default: 9 Valid Values: [1,...,17] Importance: medium -
compression.zstd.level
The compression level to use if compression.type is set to
zstd
.Type: int Default: 3 Valid Values: [-131072,...,22] Importance: medium -
connections.max.idle.ms
Close idle connections after the number of milliseconds specified by this config.
Type: long Default: 540000 (9 minutes) Valid Values: Importance: medium -
delivery.timeout.ms
An upper bound on the time to report success or failure after a call to
send()
returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgement from the broker (if expected), and the time allowed for retriable send failures. The producer may report failure to send a record earlier than this config if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch which reached an earlier delivery expiration deadline. The value of this config should be greater than or equal to the sum ofrequest.timeout.ms
andlinger.ms
.Type: int Default: 120000 (2 minutes) Valid Values: [0,...] Importance: medium -
linger.ms
The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay—that is, rather than immediately sending out a record, the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle's algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get
batch.size
worth of records for a partition it will be sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Settinglinger.ms=5
, for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.Type: long Default: 0 Valid Values: [0,...] Importance: medium -
max.block.ms
The configuration controls how long the
KafkaProducer
'ssend()
,partitionsFor()
,initTransactions()
,sendOffsetsToTransaction()
,commitTransaction()
andabortTransaction()
methods will block. Forsend()
this timeout bounds the total time waiting for both metadata fetch and buffer allocation (blocking in the user-supplied serializers or partitioner is not counted against this timeout). ForpartitionsFor()
this timeout bounds the time spent waiting for metadata if it is unavailable. The transaction-related methods always block, but may timeout if the transaction coordinator could not be discovered or did not respond within the timeout.Type: long Default: 60000 (1 minute) Valid Values: [0,...] Importance: medium -
max.request.size
The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum uncompressed record batch size. Note that the server has its own cap on the record batch size (after compression if compression is enabled) which may be different from this.
Type: int Default: 1048576 Valid Values: [0,...] Importance: medium -
partitioner.class
Determines which partition to send a record to when records are produced. Available options are:
- If not set, the default partitioning logic is used. This strategy send records to a partition until at least batch.size bytes is produced to the partition. It works with the strategy:
- If no partition is specified but a key is present, choose a partition based on a hash of the key.
- If no partition or key is present, choose the sticky partition that changes when at least batch.size bytes are produced to the partition.
org.apache.kafka.clients.producer.RoundRobinPartitioner
: A partitioning strategy where each record in a series of consecutive records is sent to a different partition, regardless of whether the 'key' is provided or not, until partitions run out and the process starts over again. Note: There's a known issue that will cause uneven distribution when a new batch is created. See KAFKA-9965 for more detail.
Implementing the
org.apache.kafka.clients.producer.Partitioner
interface allows you to plug in a custom partitioner.Type: class Default: null Valid Values: Importance: medium - If not set, the default partitioning logic is used. This strategy send records to a partition until at least batch.size bytes is produced to the partition. It works with the strategy:
-
partitioner.ignore.keys
When set to 'true' the producer won't use record keys to choose a partition. If 'false', producer would choose a partition based on a hash of the key when a key is present. Note: this setting has no effect if a custom partitioner is used.
Type: boolean Default: false Valid Values: Importance: medium -
receive.buffer.bytes
The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.
Type: int Default: 32768 (32 kibibytes) Valid Values: [-1,...] Importance: medium -
request.timeout.ms
The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This should be larger than
replica.lag.time.max.ms
(a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.Type: int Default: 30000 (30 seconds) Valid Values: [0,...] Importance: medium -
sasl.client.callback.handler.class
The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface.
Type: class Default: null Valid Values: Importance: medium -
sasl.jaas.config
JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is:
loginModuleClass controlFlag (optionName=optionValue)*;
. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;Type: password Default: null Valid Values: Importance: medium -
sasl.kerberos.service.name
The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.
Type: string Default: null Valid Values: Importance: medium -
sasl.login.callback.handler.class
The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler
Type: class Default: null Valid Values: Importance: medium -
sasl.login.class
The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin
Type: class Default: null Valid Values: Importance: medium -
sasl.mechanism
SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.
Type: string Default: GSSAPI Valid Values: Importance: medium -
sasl.oauthbearer.jwks.endpoint.url
The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. If the URL is HTTP(S)-based, the JWKS data will be retrieved from the OAuth/OIDC provider via the configured URL on broker startup. All then-current keys will be cached on the broker for incoming requests. If an authentication request is received for a JWT that includes a "kid" header claim value that isn't yet in the cache, the JWKS endpoint will be queried again on demand. However, the broker polls the URL every sasl.oauthbearer.jwks.endpoint.refresh.ms milliseconds to refresh the cache with any forthcoming keys before any JWT requests that include them are received. If the URL is file-based, the broker will load the JWKS file from a configured location on startup. In the event that the JWT includes a "kid" header value that isn't in the JWKS file, the broker will reject the JWT and authentication will fail.
Type: string Default: null Valid Values: Importance: medium -
sasl.oauthbearer.token.endpoint.url
The URL for the OAuth/OIDC identity provider. If the URL is HTTP(S)-based, it is the issuer's token endpoint URL to which requests will be made to login based on the configuration in sasl.jaas.config. If the URL is file-based, it specifies a file containing an access token (in JWT serialized form) issued by the OAuth/OIDC identity provider to use for authorization.
Type: string Default: null Valid Values: Importance: medium -
security.protocol
Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.
Type: string Default: PLAINTEXT Valid Values: (case insensitive) [SASL_SSL, PLAINTEXT, SSL, SASL_PLAINTEXT] Importance: medium -
send.buffer.bytes
The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.
Type: int Default: 131072 (128 kibibytes) Valid Values: [-1,...] Importance: medium -
socket.connection.setup.timeout.max.ms
The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. To avoid connection storms, a randomization factor of 0.2 will be applied to the timeout resulting in a random range between 20% below and 20% above the computed value.
Type: long Default: 30000 (30 seconds) Valid Values: Importance: medium -
socket.connection.setup.timeout.ms
The amount of time the client will wait for the socket connection to be established. If the connection is not built before the timeout elapses, clients will close the socket channel. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the
socket.connection.setup.timeout.max.ms
value.Type: long Default: 10000 (10 seconds) Valid Values: Importance: medium -
ssl.enabled.protocols
The list of protocols enabled for SSL connections. The default is 'TLSv1.2,TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). This default should be fine for most cases. Also see the config documentation for `ssl.protocol`.
Type: list Default: TLSv1.2 Valid Values: Importance: medium -
ssl.keystore.type
The file format of the key store file. This is optional for client. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].
Type: string Default: JKS Valid Values: Importance: medium -
ssl.protocol
The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'.
Type: string Default: TLSv1.2 Valid Values: Importance: medium -
ssl.provider
The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.
Type: string Default: null Valid Values: Importance: medium -
ssl.truststore.type
The file format of the trust store file. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].
Type: string Default: JKS Valid Values: Importance: medium -
acks
The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. The following settings are allowed:
acks=0
If set to zero then the producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and theretries
configuration will not take effect (as the client won't generally know of any failures). The offset given back for each record will always be set to-1
.acks=1
This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers. In this case should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost.acks=all
This means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee. This is equivalent to the acks=-1 setting.
Note that enabling idempotence requires this config value to be 'all'. If conflicting configurations are set and idempotence is not explicitly enabled, idempotence is disabled.
Type: string Default: all Valid Values: [all, -1, 0, 1] Importance: low -
auto.include.jmx.reporter
Deprecated. Whether to automatically include JmxReporter even if it's not listed in
metric.reporters
. This configuration will be removed in Kafka 4.0, users should instead includeorg.apache.kafka.common.metrics.JmxReporter
inmetric.reporters
in order to enable the JmxReporter.Type: boolean Default: true Valid Values: Importance: low -
enable.idempotence
When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc., may write duplicates of the retried message in the stream. Note that enabling idempotence requires
max.in.flight.requests.per.connection
to be less than or equal to 5 (with message ordering preserved for any allowable value),retries
to be greater than 0, andacks
must be 'all'.Idempotence is enabled by default if no conflicting configurations are set. If conflicting configurations are set and idempotence is not explicitly enabled, idempotence is disabled. If idempotence is explicitly enabled and conflicting configurations are set, a
ConfigException
is thrown.Type: boolean Default: true Valid Values: Importance: low -
enable.metrics.push
Whether to enable pushing of client metrics to the cluster, if the cluster has a client metrics subscription which matches this client.
Type: boolean Default: true Valid Values: Importance: low -
interceptor.classes
A list of classes to use as interceptors. Implementing the
org.apache.kafka.clients.producer.ProducerInterceptor
interface allows you to intercept (and possibly mutate) the records received by the producer before they are published to the Kafka cluster. By default, there are no interceptors.Type: list Default: "" Valid Values: non-null string Importance: low -
max.in.flight.requests.per.connection
The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this configuration is set to be greater than 1 and
enable.idempotence
is set to false, there is a risk of message reordering after a failed send due to retries (i.e., if retries are enabled); if retries are disabled or ifenable.idempotence
is set to true, ordering will be preserved. Additionally, enabling idempotence requires the value of this configuration to be less than or equal to 5. If conflicting configurations are set and idempotence is not explicitly enabled, idempotence is disabled.Type: int Default: 5 Valid Values: [1,...] Importance: low -
metadata.max.age.ms
The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.
Type: long Default: 300000 (5 minutes) Valid Values: [0,...] Importance: low -
metadata.max.idle.ms
Controls how long the producer will cache metadata for a topic that's idle. If the elapsed time since a topic was last produced to exceeds the metadata idle duration, then the topic's metadata is forgotten and the next access to it will force a metadata fetch request.
Type: long Default: 300000 (5 minutes) Valid Values: [5000,...] Importance: low -
metadata.recovery.strategy
Controls how the client recovers when none of the brokers known to it is available. If set to
none
, the client fails. If set torebootstrap
, the client repeats the bootstrap process usingbootstrap.servers
. Rebootstrapping is useful when a client communicates with brokers so infrequently that the set of brokers may change entirely before the client refreshes metadata. Metadata recovery is triggered when all last-known brokers appear unavailable simultaneously. Brokers appear unavailable when disconnected and no current retry attempt is in-progress. Consider increasingreconnect.backoff.ms
andreconnect.backoff.max.ms
and decreasingsocket.connection.setup.timeout.ms
andsocket.connection.setup.timeout.max.ms
for the client.Type: string Default: none Valid Values: (case insensitive) [REBOOTSTRAP, NONE] Importance: low -
metric.reporters
A list of classes to use as metrics reporters. Implementing the
org.apache.kafka.common.metrics.MetricsReporter
interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.Type: list Default: "" Valid Values: non-null string Importance: low -
metrics.num.samples
The number of samples maintained to compute metrics.
Type: int Default: 2 Valid Values: [1,...] Importance: low -
metrics.recording.level
The highest recording level for metrics.
Type: string Default: INFO Valid Values: [INFO, DEBUG, TRACE] Importance: low -
metrics.sample.window.ms
The window of time a metrics sample is computed over.
Type: long Default: 30000 (30 seconds) Valid Values: [0,...] Importance: low -
partitioner.adaptive.partitioning.enable
When set to 'true', the producer will try to adapt to broker performance and produce more messages to partitions hosted on faster brokers. If 'false', producer will try to distribute messages uniformly. Note: this setting has no effect if a custom partitioner is used
Type: boolean Default: true Valid Values: Importance: low -
partitioner.availability.timeout.ms
If a broker cannot process produce requests from a partition for
partitioner.availability.timeout.ms
time, the partitioner treats that partition as not available. If the value is 0, this logic is disabled. Note: this setting has no effect if a custom partitioner is used orpartitioner.adaptive.partitioning.enable
is set to 'false'Type: long Default: 0 Valid Values: [0,...] Importance: low -
reconnect.backoff.max.ms
The maximum amount of time in milliseconds to wait when reconnecting to a broker that has repeatedly failed to connect. If provided, the backoff per host will increase exponentially for each consecutive connection failure, up to this maximum. After calculating the backoff increase, 20% random jitter is added to avoid connection storms.
Type: long Default: 1000 (1 second) Valid Values: [0,...] Importance: low -
reconnect.backoff.ms
The base amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all connection attempts by the client to a broker. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the
reconnect.backoff.max.ms
value.Type: long Default: 50 Valid Values: [0,...] Importance: low -
retry.backoff.max.ms
The maximum amount of time in milliseconds to wait when retrying a request to the broker that has repeatedly failed. If provided, the backoff per client will increase exponentially for each failed request, up to this maximum. To prevent all clients from being synchronized upon retry, a randomized jitter with a factor of 0.2 will be applied to the backoff, resulting in the backoff falling within a range between 20% below and 20% above the computed value. If
retry.backoff.ms
is set to be higher thanretry.backoff.max.ms
, thenretry.backoff.max.ms
will be used as a constant backoff from the beginning without any exponential increaseType: long Default: 1000 (1 second) Valid Values: [0,...] Importance: low -
retry.backoff.ms
The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. This value is the initial backoff value and will increase exponentially for each failed request, up to the
retry.backoff.max.ms
value.Type: long Default: 100 Valid Values: [0,...] Importance: low -
sasl.kerberos.kinit.cmd
Kerberos kinit command path.
Type: string Default: /usr/bin/kinit Valid Values: Importance: low -
sasl.kerberos.min.time.before.relogin
Login thread sleep time between refresh attempts.
Type: long Default: 60000 Valid Values: Importance: low -
sasl.kerberos.ticket.renew.jitter
Percentage of random jitter added to the renewal time.
Type: double Default: 0.05 Valid Values: Importance: low -
sasl.kerberos.ticket.renew.window.factor
Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.
Type: double Default: 0.8 Valid Values: Importance: low -
sasl.login.connect.timeout.ms
The (optional) value in milliseconds for the external authentication provider connection timeout. Currently applies only to OAUTHBEARER.
Type: int Default: null Valid Values: Importance: low -
sasl.login.read.timeout.ms
The (optional) value in milliseconds for the external authentication provider read timeout. Currently applies only to OAUTHBEARER.
Type: int Default: null Valid Values: Importance: low -
sasl.login.refresh.buffer.seconds
The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.
Type: short Default: 300 Valid Values: [0,...,3600] Importance: low -
sasl.login.refresh.min.period.seconds
The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.
Type: short Default: 60 Valid Values: [0,...,900] Importance: low -
sasl.login.refresh.window.factor
Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER.
Type: double Default: 0.8 Valid Values: [0.5,...,1.0] Importance: low -
sasl.login.refresh.window.jitter
The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.
Type: double Default: 0.05 Valid Values: [0.0,...,0.25] Importance: low -
sasl.login.retry.backoff.max.ms
The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.
Type: long Default: 10000 (10 seconds) Valid Values: Importance: low -
sasl.login.retry.backoff.ms
The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.
Type: long Default: 100 Valid Values: Importance: low -
sasl.oauthbearer.clock.skew.seconds
The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker.
Type: int Default: 30 Valid Values: Importance: low -
sasl.oauthbearer.expected.audience
The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWT's "aud" claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.
Type: list Default: null Valid Values: Importance: low -
sasl.oauthbearer.expected.issuer
The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth "iss" claim and if this value is set, the broker will match it exactly against what is in the JWT's "iss" claim. If there is no match, the broker will reject the JWT and authentication will fail.
Type: string Default: null Valid Values: Importance: low -
sasl.oauthbearer.header.urlencode
The (optional) setting to enable the OAuth client to URL-encode the client_id and client_secret in the authorization header in accordance with RFC6749, see here for more details. The default value is set to 'false' for backward compatibility
Type: boolean Default: false Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.refresh.ms
The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.
Type: long Default: 3600000 (1 hour) Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms
The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.
Type: long Default: 10000 (10 seconds) Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms
The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.
Type: long Default: 100 Valid Values: Importance: low -
sasl.oauthbearer.scope.claim.name
The OAuth claim for the scope is often named "scope", but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
Type: string Default: scope Valid Values: Importance: low -
sasl.oauthbearer.sub.claim.name
The OAuth claim for the subject is often named "sub", but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
Type: string Default: sub Valid Values: Importance: low -
security.providers
A list of configurable creator classes each returning a provider implementing security algorithms. These classes should implement the
org.apache.kafka.common.security.auth.SecurityProviderCreator
interface.Type: string Default: null Valid Values: Importance: low -
ssl.cipher.suites
A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.
Type: list Default: null Valid Values: Importance: low -
ssl.endpoint.identification.algorithm
The endpoint identification algorithm to validate server hostname using server certificate.
Type: string Default: https Valid Values: Importance: low -
ssl.engine.factory.class
The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is org.apache.kafka.common.security.ssl.DefaultSslEngineFactory. Alternatively, setting this to org.apache.kafka.common.security.ssl.CommonNameLoggingSslEngineFactory will log the common name of expired SSL certificates used by clients to authenticate at any of the brokers with log level INFO. Note that this will cause a tiny delay during establishment of new connections from mTLS clients to brokers due to the extra code for examining the certificate chain provided by the client. Note further that the implementation uses a custom truststore based on the standard Java truststore and thus might be considered a security risk due to not being as mature as the standard one.
Type: class Default: null Valid Values: Importance: low -
ssl.keymanager.algorithm
The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.
Type: string Default: SunX509 Valid Values: Importance: low -
ssl.secure.random.implementation
The SecureRandom PRNG implementation to use for SSL cryptography operations.
Type: string Default: null Valid Values: Importance: low -
ssl.trustmanager.algorithm
The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.
Type: string Default: PKIX Valid Values: Importance: low -
transaction.timeout.ms
The maximum amount of time in milliseconds that a transaction will remain open before the coordinator proactively aborts it. The start of the transaction is set at the time that the first partition is added to it. If this value is larger than the
transaction.max.timeout.ms
setting in the broker, the request will fail with aInvalidTxnTimeoutException
error.Type: int Default: 60000 (1 minute) Valid Values: Importance: low -
transactional.id
The TransactionalId to use for transactional delivery. This enables reliability semantics which span multiple producer sessions since it allows the client to guarantee that transactions using the same TransactionalId have been completed prior to starting any new transactions. If no TransactionalId is provided, then the producer is limited to idempotent delivery. If a TransactionalId is configured,
enable.idempotence
is implied. By default the TransactionId is not configured, which means transactions cannot be used. Note that, by default, transactions require a cluster of at least three brokers which is the recommended setting for production; for development you can change this, by adjusting broker settingtransaction.state.log.replication.factor
.Type: string Default: null Valid Values: non-empty string Importance: low
3.4 Consumer Configs
Below is the configuration for the consumer:-
key.deserializer
Deserializer class for key that implements the
org.apache.kafka.common.serialization.Deserializer
interface.Type: class Default: Valid Values: Importance: high -
value.deserializer
Deserializer class for value that implements the
org.apache.kafka.common.serialization.Deserializer
interface.Type: class Default: Valid Values: Importance: high -
bootstrap.servers
A list of host/port pairs used to establish the initial connection to the Kafka cluster. Clients use this list to bootstrap and discover the full set of Kafka brokers. While the order of servers in the list does not matter, we recommend including more than one server to ensure resilience if any servers are down. This list does not need to contain the entire set of brokers, as Kafka clients automatically manage and update connections to the cluster efficiently. This list must be in the form
host1:port1,host2:port2,...
.Type: list Default: "" Valid Values: non-null string Importance: high -
fetch.min.bytes
The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as that many byte(s) of data is available or the fetch request times out waiting for data to arrive. Setting this to a larger value will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency.
Type: int Default: 1 Valid Values: [0,...] Importance: high -
group.id
A unique string that identifies the consumer group this consumer belongs to. This property is required if the consumer uses either the group management functionality by using
subscribe(topic)
or the Kafka-based offset management strategy.Type: string Default: null Valid Values: Importance: high -
group.protocol
The group protocol consumer should use. We currently support "classic" or "consumer". If "consumer" is specified, then the consumer group protocol will be used. Otherwise, the classic group protocol will be used.
Type: string Default: classic Valid Values: (case insensitive) [CONSUMER, CLASSIC] Importance: high -
heartbeat.interval.ms
The expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than
session.timeout.ms
, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.Type: int Default: 3000 (3 seconds) Valid Values: Importance: high -
max.partition.fetch.bytes
The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined via
message.max.bytes
(broker config) ormax.message.bytes
(topic config). See fetch.max.bytes for limiting the consumer request size.Type: int Default: 1048576 (1 mebibyte) Valid Values: [0,...] Importance: high -
session.timeout.ms
The timeout used to detect client failures when using Kafka's group management facility. The client sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this client from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by
group.min.session.timeout.ms
andgroup.max.session.timeout.ms
.Type: int Default: 45000 (45 seconds) Valid Values: Importance: high -
ssl.key.password
The password of the private key in the key store file or the PEM key specified in 'ssl.keystore.key'.
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.certificate.chain
Certificate chain in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with a list of X.509 certificates
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.key
Private key in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with PKCS#8 keys. If the key is encrypted, key password must be specified using 'ssl.key.password'
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.location
The location of the key store file. This is optional for client and can be used for two-way authentication for client.
Type: string Default: null Valid Values: Importance: high -
ssl.keystore.password
The store password for the key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured. Key store password is not supported for PEM format.
Type: password Default: null Valid Values: Importance: high -
ssl.truststore.certificates
Trusted certificates in the format specified by 'ssl.truststore.type'. Default SSL engine factory supports only PEM format with X.509 certificates.
Type: password Default: null Valid Values: Importance: high -
ssl.truststore.location
The location of the trust store file.
Type: string Default: null Valid Values: Importance: high -
ssl.truststore.password
The password for the trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled. Trust store password is not supported for PEM format.
Type: password Default: null Valid Values: Importance: high -
allow.auto.create.topics
Allow automatic topic creation on the broker when subscribing to or assigning a topic. A topic being subscribed to will be automatically created only if the broker allows for it using `auto.create.topics.enable` broker configuration. This configuration must be set to `false` when using brokers older than 0.11.0
Type: boolean Default: true Valid Values: Importance: medium -
auto.offset.reset
What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):
- earliest: automatically reset the offset to the earliest offset
- latest: automatically reset the offset to the latest offset
- none: throw exception to the consumer if no previous offset is found for the consumer's group
- anything else: throw exception to the consumer.
Note that altering partition numbers while setting this config to latest may cause message delivery loss since producers could start to send messages to newly added partitions (i.e. no initial offsets exist yet) before consumers reset their offsets.
Type: string Default: latest Valid Values: [latest, earliest, none] Importance: medium -
client.dns.lookup
Controls how the client uses DNS lookups. If set to
use_all_dns_ips
, connect to each returned IP address in sequence until a successful connection is established. After a disconnection, the next IP is used. Once all IPs have been used once, the client resolves the IP(s) from the hostname again (both the JVM and the OS cache DNS name lookups, however). If set toresolve_canonical_bootstrap_servers_only
, resolve each bootstrap address into a list of canonical names. After the bootstrap phase, this behaves the same asuse_all_dns_ips
.Type: string Default: use_all_dns_ips Valid Values: [use_all_dns_ips, resolve_canonical_bootstrap_servers_only] Importance: medium -
connections.max.idle.ms
Close idle connections after the number of milliseconds specified by this config.
Type: long Default: 540000 (9 minutes) Valid Values: Importance: medium -
default.api.timeout.ms
Specifies the timeout (in milliseconds) for client APIs. This configuration is used as the default timeout for all client operations that do not specify a
timeout
parameter.Type: int Default: 60000 (1 minute) Valid Values: [0,...] Importance: medium -
enable.auto.commit
If true the consumer's offset will be periodically committed in the background.
Type: boolean Default: true Valid Values: Importance: medium -
exclude.internal.topics
Whether internal topics matching a subscribed pattern should be excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.
Type: boolean Default: true Valid Values: Importance: medium -
fetch.max.bytes
The maximum amount of data the server should return for a fetch request. Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not a absolute maximum. The maximum record batch size accepted by the broker is defined via
message.max.bytes
(broker config) ormax.message.bytes
(topic config). Note that the consumer performs multiple fetches in parallel.Type: int Default: 52428800 (50 mebibytes) Valid Values: [0,...] Importance: medium -
group.instance.id
A unique identifier of the consumer instance provided by the end user. Only non-empty strings are permitted. If set, the consumer is treated as a static member, which means that only one instance with this ID is allowed in the consumer group at any time. This can be used in combination with a larger session timeout to avoid group rebalances caused by transient unavailability (e.g. process restarts). If not set, the consumer will join the group as a dynamic member, which is the traditional behavior.
Type: string Default: null Valid Values: non-empty string Importance: medium -
group.remote.assignor
The server-side assignor to use. If no assignor is specified, the group coordinator will pick one. This configuration is applied only if
group.protocol
is set to "consumer".Type: string Default: null Valid Values: Importance: medium -
isolation.level
Controls how to read messages written transactionally. If set to
read_committed
, consumer.poll() will only return transactional messages which have been committed. If set toread_uncommitted
(the default), consumer.poll() will return all messages, even transactional messages which have been aborted. Non-transactional messages will be returned unconditionally in either mode.Messages will always be returned in offset order. Hence, in
read_committed
mode, consumer.poll() will only return messages up to the last stable offset (LSO), which is the one less than the offset of the first open transaction. In particular any messages appearing after messages belonging to ongoing transactions will be withheld until the relevant transaction has been completed. As a result,read_committed
consumers will not be able to read up to the high watermark when there are in flight transactions.Further, when in
read_committed
the seekToEnd method will return the LSOType: string Default: read_uncommitted Valid Values: [read_committed, read_uncommitted] Importance: medium -
max.poll.interval.ms
The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member. For consumers using a non-null
group.instance.id
which reach this timeout, partitions will not be immediately reassigned. Instead, the consumer will stop sending heartbeats and partitions will be reassigned after expiration ofsession.timeout.ms
. This mirrors the behavior of a static consumer which has shutdown.Type: int Default: 300000 (5 minutes) Valid Values: [1,...] Importance: medium -
max.poll.records
The maximum number of records returned in a single call to poll(). Note, that
max.poll.records
does not impact the underlying fetching behavior. The consumer will cache the records from each fetch request and returns them incrementally from each poll.Type: int Default: 500 Valid Values: [1,...] Importance: medium -
partition.assignment.strategy
A list of class names or class types, ordered by preference, of supported partition assignment strategies that the client will use to distribute partition ownership amongst consumer instances when group management is used. Available options are:
org.apache.kafka.clients.consumer.RangeAssignor
: Assigns partitions on a per-topic basis.org.apache.kafka.clients.consumer.RoundRobinAssignor
: Assigns partitions to consumers in a round-robin fashion.org.apache.kafka.clients.consumer.StickyAssignor
: Guarantees an assignment that is maximally balanced while preserving as many existing partition assignments as possible.org.apache.kafka.clients.consumer.CooperativeStickyAssignor
: Follows the same StickyAssignor logic, but allows for cooperative rebalancing.
The default assignor is [RangeAssignor, CooperativeStickyAssignor], which will use the RangeAssignor by default, but allows upgrading to the CooperativeStickyAssignor with just a single rolling bounce that removes the RangeAssignor from the list.
Implementing the
org.apache.kafka.clients.consumer.ConsumerPartitionAssignor
interface allows you to plug in a custom assignment strategy.Type: list Default: class org.apache.kafka.clients.consumer.RangeAssignor,class org.apache.kafka.clients.consumer.CooperativeStickyAssignor Valid Values: non-null string Importance: medium -
receive.buffer.bytes
The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.
Type: int Default: 65536 (64 kibibytes) Valid Values: [-1,...] Importance: medium -
request.timeout.ms
The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.
Type: int Default: 30000 (30 seconds) Valid Values: [0,...] Importance: medium -
sasl.client.callback.handler.class
The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface.
Type: class Default: null Valid Values: Importance: medium -
sasl.jaas.config
JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is:
loginModuleClass controlFlag (optionName=optionValue)*;
. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;Type: password Default: null Valid Values: Importance: medium -
sasl.kerberos.service.name
The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.
Type: string Default: null Valid Values: Importance: medium -
sasl.login.callback.handler.class
The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler
Type: class Default: null Valid Values: Importance: medium -
sasl.login.class
The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin
Type: class Default: null Valid Values: Importance: medium -
sasl.mechanism
SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.
Type: string Default: GSSAPI Valid Values: Importance: medium -
sasl.oauthbearer.jwks.endpoint.url
The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. If the URL is HTTP(S)-based, the JWKS data will be retrieved from the OAuth/OIDC provider via the configured URL on broker startup. All then-current keys will be cached on the broker for incoming requests. If an authentication request is received for a JWT that includes a "kid" header claim value that isn't yet in the cache, the JWKS endpoint will be queried again on demand. However, the broker polls the URL every sasl.oauthbearer.jwks.endpoint.refresh.ms milliseconds to refresh the cache with any forthcoming keys before any JWT requests that include them are received. If the URL is file-based, the broker will load the JWKS file from a configured location on startup. In the event that the JWT includes a "kid" header value that isn't in the JWKS file, the broker will reject the JWT and authentication will fail.
Type: string Default: null Valid Values: Importance: medium -
sasl.oauthbearer.token.endpoint.url
The URL for the OAuth/OIDC identity provider. If the URL is HTTP(S)-based, it is the issuer's token endpoint URL to which requests will be made to login based on the configuration in sasl.jaas.config. If the URL is file-based, it specifies a file containing an access token (in JWT serialized form) issued by the OAuth/OIDC identity provider to use for authorization.
Type: string Default: null Valid Values: Importance: medium -
security.protocol
Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.
Type: string Default: PLAINTEXT Valid Values: (case insensitive) [SASL_SSL, PLAINTEXT, SSL, SASL_PLAINTEXT] Importance: medium -
send.buffer.bytes
The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.
Type: int Default: 131072 (128 kibibytes) Valid Values: [-1,...] Importance: medium -
socket.connection.setup.timeout.max.ms
The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. To avoid connection storms, a randomization factor of 0.2 will be applied to the timeout resulting in a random range between 20% below and 20% above the computed value.
Type: long Default: 30000 (30 seconds) Valid Values: Importance: medium -
socket.connection.setup.timeout.ms
The amount of time the client will wait for the socket connection to be established. If the connection is not built before the timeout elapses, clients will close the socket channel. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the
socket.connection.setup.timeout.max.ms
value.Type: long Default: 10000 (10 seconds) Valid Values: Importance: medium -
ssl.enabled.protocols
The list of protocols enabled for SSL connections. The default is 'TLSv1.2,TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). This default should be fine for most cases. Also see the config documentation for `ssl.protocol`.
Type: list Default: TLSv1.2 Valid Values: Importance: medium -
ssl.keystore.type
The file format of the key store file. This is optional for client. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].
Type: string Default: JKS Valid Values: Importance: medium -
ssl.protocol
The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'.
Type: string Default: TLSv1.2 Valid Values: Importance: medium -
ssl.provider
The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.
Type: string Default: null Valid Values: Importance: medium -
ssl.truststore.type
The file format of the trust store file. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].
Type: string Default: JKS Valid Values: Importance: medium -
auto.commit.interval.ms
The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if
enable.auto.commit
is set totrue
.Type: int Default: 5000 (5 seconds) Valid Values: [0,...] Importance: low -
auto.include.jmx.reporter
Deprecated. Whether to automatically include JmxReporter even if it's not listed in
metric.reporters
. This configuration will be removed in Kafka 4.0, users should instead includeorg.apache.kafka.common.metrics.JmxReporter
inmetric.reporters
in order to enable the JmxReporter.Type: boolean Default: true Valid Values: Importance: low -
check.crcs
Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance.
Type: boolean Default: true Valid Values: Importance: low -
client.id
An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.
Type: string Default: "" Valid Values: Importance: low -
client.rack
A rack identifier for this client. This can be any string value which indicates where this client is physically located. It corresponds with the broker config 'broker.rack'
Type: string Default: "" Valid Values: Importance: low -
enable.metrics.push
Whether to enable pushing of client metrics to the cluster, if the cluster has a client metrics subscription which matches this client.
Type: boolean Default: true Valid Values: Importance: low -
fetch.max.wait.ms
The maximum amount of time the server will block before answering the fetch request there isn't sufficient data to immediately satisfy the requirement given by fetch.min.bytes. This config is used only for local log fetch. To tune the remote fetch maximum wait time, please refer to 'remote.fetch.max.wait.ms' broker config
Type: int Default: 500 Valid Values: [0,...] Importance: low -
interceptor.classes
A list of classes to use as interceptors. Implementing the
org.apache.kafka.clients.consumer.ConsumerInterceptor
interface allows you to intercept (and possibly mutate) records received by the consumer. By default, there are no interceptors.Type: list Default: "" Valid Values: non-null string Importance: low -
metadata.max.age.ms
The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.
Type: long Default: 300000 (5 minutes) Valid Values: [0,...] Importance: low -
metadata.recovery.strategy
Controls how the client recovers when none of the brokers known to it is available. If set to
none
, the client fails. If set torebootstrap
, the client repeats the bootstrap process usingbootstrap.servers
. Rebootstrapping is useful when a client communicates with brokers so infrequently that the set of brokers may change entirely before the client refreshes metadata. Metadata recovery is triggered when all last-known brokers appear unavailable simultaneously. Brokers appear unavailable when disconnected and no current retry attempt is in-progress. Consider increasingreconnect.backoff.ms
andreconnect.backoff.max.ms
and decreasingsocket.connection.setup.timeout.ms
andsocket.connection.setup.timeout.max.ms
for the client.Type: string Default: none Valid Values: (case insensitive) [REBOOTSTRAP, NONE] Importance: low -
metric.reporters
A list of classes to use as metrics reporters. Implementing the
org.apache.kafka.common.metrics.MetricsReporter
interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.Type: list Default: "" Valid Values: non-null string Importance: low -
metrics.num.samples
The number of samples maintained to compute metrics.
Type: int Default: 2 Valid Values: [1,...] Importance: low -
metrics.recording.level
The highest recording level for metrics.
Type: string Default: INFO Valid Values: [INFO, DEBUG, TRACE] Importance: low -
metrics.sample.window.ms
The window of time a metrics sample is computed over.
Type: long Default: 30000 (30 seconds) Valid Values: [0,...] Importance: low -
reconnect.backoff.max.ms
The maximum amount of time in milliseconds to wait when reconnecting to a broker that has repeatedly failed to connect. If provided, the backoff per host will increase exponentially for each consecutive connection failure, up to this maximum. After calculating the backoff increase, 20% random jitter is added to avoid connection storms.
Type: long Default: 1000 (1 second) Valid Values: [0,...] Importance: low -
reconnect.backoff.ms
The base amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all connection attempts by the client to a broker. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the
reconnect.backoff.max.ms
value.Type: long Default: 50 Valid Values: [0,...] Importance: low -
retry.backoff.max.ms
The maximum amount of time in milliseconds to wait when retrying a request to the broker that has repeatedly failed. If provided, the backoff per client will increase exponentially for each failed request, up to this maximum. To prevent all clients from being synchronized upon retry, a randomized jitter with a factor of 0.2 will be applied to the backoff, resulting in the backoff falling within a range between 20% below and 20% above the computed value. If
retry.backoff.ms
is set to be higher thanretry.backoff.max.ms
, thenretry.backoff.max.ms
will be used as a constant backoff from the beginning without any exponential increaseType: long Default: 1000 (1 second) Valid Values: [0,...] Importance: low -
retry.backoff.ms
The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. This value is the initial backoff value and will increase exponentially for each failed request, up to the
retry.backoff.max.ms
value.Type: long Default: 100 Valid Values: [0,...] Importance: low -
sasl.kerberos.kinit.cmd
Kerberos kinit command path.
Type: string Default: /usr/bin/kinit Valid Values: Importance: low -
sasl.kerberos.min.time.before.relogin
Login thread sleep time between refresh attempts.
Type: long Default: 60000 Valid Values: Importance: low -
sasl.kerberos.ticket.renew.jitter
Percentage of random jitter added to the renewal time.
Type: double Default: 0.05 Valid Values: Importance: low -
sasl.kerberos.ticket.renew.window.factor
Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.
Type: double Default: 0.8 Valid Values: Importance: low -
sasl.login.connect.timeout.ms
The (optional) value in milliseconds for the external authentication provider connection timeout. Currently applies only to OAUTHBEARER.
Type: int Default: null Valid Values: Importance: low -
sasl.login.read.timeout.ms
The (optional) value in milliseconds for the external authentication provider read timeout. Currently applies only to OAUTHBEARER.
Type: int Default: null Valid Values: Importance: low -
sasl.login.refresh.buffer.seconds
The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.
Type: short Default: 300 Valid Values: [0,...,3600] Importance: low -
sasl.login.refresh.min.period.seconds
The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.
Type: short Default: 60 Valid Values: [0,...,900] Importance: low -
sasl.login.refresh.window.factor
Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER.
Type: double Default: 0.8 Valid Values: [0.5,...,1.0] Importance: low -
sasl.login.refresh.window.jitter
The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.
Type: double Default: 0.05 Valid Values: [0.0,...,0.25] Importance: low -
sasl.login.retry.backoff.max.ms
The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.
Type: long Default: 10000 (10 seconds) Valid Values: Importance: low -
sasl.login.retry.backoff.ms
The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.
Type: long Default: 100 Valid Values: Importance: low -
sasl.oauthbearer.clock.skew.seconds
The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker.
Type: int Default: 30 Valid Values: Importance: low -
sasl.oauthbearer.expected.audience
The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWT's "aud" claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.
Type: list Default: null Valid Values: Importance: low -
sasl.oauthbearer.expected.issuer
The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth "iss" claim and if this value is set, the broker will match it exactly against what is in the JWT's "iss" claim. If there is no match, the broker will reject the JWT and authentication will fail.
Type: string Default: null Valid Values: Importance: low -
sasl.oauthbearer.header.urlencode
The (optional) setting to enable the OAuth client to URL-encode the client_id and client_secret in the authorization header in accordance with RFC6749, see here for more details. The default value is set to 'false' for backward compatibility
Type: boolean Default: false Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.refresh.ms
The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.
Type: long Default: 3600000 (1 hour) Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms
The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.
Type: long Default: 10000 (10 seconds) Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms
The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.
Type: long Default: 100 Valid Values: Importance: low -
sasl.oauthbearer.scope.claim.name
The OAuth claim for the scope is often named "scope", but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
Type: string Default: scope Valid Values: Importance: low -
sasl.oauthbearer.sub.claim.name
The OAuth claim for the subject is often named "sub", but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
Type: string Default: sub Valid Values: Importance: low -
security.providers
A list of configurable creator classes each returning a provider implementing security algorithms. These classes should implement the
org.apache.kafka.common.security.auth.SecurityProviderCreator
interface.Type: string Default: null Valid Values: Importance: low -
ssl.cipher.suites
A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.
Type: list Default: null Valid Values: Importance: low -
ssl.endpoint.identification.algorithm
The endpoint identification algorithm to validate server hostname using server certificate.
Type: string Default: https Valid Values: Importance: low -
ssl.engine.factory.class
The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is org.apache.kafka.common.security.ssl.DefaultSslEngineFactory. Alternatively, setting this to org.apache.kafka.common.security.ssl.CommonNameLoggingSslEngineFactory will log the common name of expired SSL certificates used by clients to authenticate at any of the brokers with log level INFO. Note that this will cause a tiny delay during establishment of new connections from mTLS clients to brokers due to the extra code for examining the certificate chain provided by the client. Note further that the implementation uses a custom truststore based on the standard Java truststore and thus might be considered a security risk due to not being as mature as the standard one.
Type: class Default: null Valid Values: Importance: low -
ssl.keymanager.algorithm
The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.
Type: string Default: SunX509 Valid Values: Importance: low -
ssl.secure.random.implementation
The SecureRandom PRNG implementation to use for SSL cryptography operations.
Type: string Default: null Valid Values: Importance: low -
ssl.trustmanager.algorithm
The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.
Type: string Default: PKIX Valid Values: Importance: low
3.5 Kafka Connect Configs
Below is the configuration of the Kafka Connect framework.-
config.storage.topic
The name of the Kafka topic where connector configurations are stored
Type: string Default: Valid Values: Importance: high -
group.id
A unique string that identifies the Connect cluster group this worker belongs to.
Type: string Default: Valid Values: Importance: high -
key.converter
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.
Type: class Default: Valid Values: Importance: high -
offset.storage.topic
The name of the Kafka topic where source connector offsets are stored
Type: string Default: Valid Values: Importance: high -
status.storage.topic
The name of the Kafka topic where connector and task status are stored
Type: string Default: Valid Values: Importance: high -
value.converter
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.
Type: class Default: Valid Values: Importance: high -
bootstrap.servers
A list of host/port pairs used to establish the initial connection to the Kafka cluster. Clients use this list to bootstrap and discover the full set of Kafka brokers. While the order of servers in the list does not matter, we recommend including more than one server to ensure resilience if any servers are down. This list does not need to contain the entire set of brokers, as Kafka clients automatically manage and update connections to the cluster efficiently. This list must be in the form
host1:port1,host2:port2,...
.Type: list Default: localhost:9092 Valid Values: Importance: high -
exactly.once.source.support
Whether to enable exactly-once support for source connectors in the cluster by using transactions to write source records and their source offsets, and by proactively fencing out old task generations before bringing up new ones.
To enable exactly-once source support on a new cluster, set this property to 'enabled'. To enable support on an existing cluster, first set to 'preparing' on every worker in the cluster, then set to 'enabled'. A rolling upgrade may be used for both changes. For more information on this feature, see the exactly-once source support documentation.Type: string Default: disabled Valid Values: (case insensitive) [DISABLED, ENABLED, PREPARING] Importance: high -
heartbeat.interval.ms
The expected time between heartbeats to the group coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the worker's session stays active and to facilitate rebalancing when new members join or leave the group. The value must be set lower than
session.timeout.ms
, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.Type: int Default: 3000 (3 seconds) Valid Values: Importance: high -
rebalance.timeout.ms
The maximum allowed time for each worker to join the group once a rebalance has begun. This is basically a limit on the amount of time needed for all tasks to flush any pending data and commit offsets. If the timeout is exceeded, then the worker will be removed from the group, which will cause offset commit failures.
Type: int Default: 60000 (1 minute) Valid Values: Importance: high -
session.timeout.ms
The timeout used to detect worker failures. The worker sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove the worker from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by
group.min.session.timeout.ms
andgroup.max.session.timeout.ms
.Type: int Default: 10000 (10 seconds) Valid Values: Importance: high -
ssl.key.password
The password of the private key in the key store file or the PEM key specified in 'ssl.keystore.key'.
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.certificate.chain
Certificate chain in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with a list of X.509 certificates
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.key
Private key in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with PKCS#8 keys. If the key is encrypted, key password must be specified using 'ssl.key.password'
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.location
The location of the key store file. This is optional for client and can be used for two-way authentication for client.
Type: string Default: null Valid Values: Importance: high -
ssl.keystore.password
The store password for the key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured. Key store password is not supported for PEM format.
Type: password Default: null Valid Values: Importance: high -
ssl.truststore.certificates
Trusted certificates in the format specified by 'ssl.truststore.type'. Default SSL engine factory supports only PEM format with X.509 certificates.
Type: password Default: null Valid Values: Importance: high -
ssl.truststore.location
The location of the trust store file.
Type: string Default: null Valid Values: Importance: high -
ssl.truststore.password
The password for the trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled. Trust store password is not supported for PEM format.
Type: password Default: null Valid Values: Importance: high -
client.dns.lookup
Controls how the client uses DNS lookups. If set to
use_all_dns_ips
, connect to each returned IP address in sequence until a successful connection is established. After a disconnection, the next IP is used. Once all IPs have been used once, the client resolves the IP(s) from the hostname again (both the JVM and the OS cache DNS name lookups, however). If set toresolve_canonical_bootstrap_servers_only
, resolve each bootstrap address into a list of canonical names. After the bootstrap phase, this behaves the same asuse_all_dns_ips
.Type: string Default: use_all_dns_ips Valid Values: [use_all_dns_ips, resolve_canonical_bootstrap_servers_only] Importance: medium -
connections.max.idle.ms
Close idle connections after the number of milliseconds specified by this config.
Type: long Default: 540000 (9 minutes) Valid Values: Importance: medium -
connector.client.config.override.policy
Class name or alias of implementation of
ConnectorClientConfigOverridePolicy
. Defines what client configurations can be overridden by the connector. The default implementation is `All`, meaning connector configurations can override all client properties. The other possible policies in the framework include `None` to disallow connectors from overriding client properties, and `Principal` to allow connectors to override only client principals.Type: string Default: All Valid Values: Importance: medium -
receive.buffer.bytes
The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.
Type: int Default: 32768 (32 kibibytes) Valid Values: [-1,...] Importance: medium -
request.timeout.ms
The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.
Type: int Default: 40000 (40 seconds) Valid Values: [0,...] Importance: medium -
sasl.client.callback.handler.class
The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface.
Type: class Default: null Valid Values: Importance: medium -
sasl.jaas.config
JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is:
loginModuleClass controlFlag (optionName=optionValue)*;
. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;Type: password Default: null Valid Values: Importance: medium -
sasl.kerberos.service.name
The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.
Type: string Default: null Valid Values: Importance: medium -
sasl.login.callback.handler.class
The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler
Type: class Default: null Valid Values: Importance: medium -
sasl.login.class
The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin
Type: class Default: null Valid Values: Importance: medium -
sasl.mechanism
SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.
Type: string Default: GSSAPI Valid Values: Importance: medium -
sasl.oauthbearer.jwks.endpoint.url
The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. If the URL is HTTP(S)-based, the JWKS data will be retrieved from the OAuth/OIDC provider via the configured URL on broker startup. All then-current keys will be cached on the broker for incoming requests. If an authentication request is received for a JWT that includes a "kid" header claim value that isn't yet in the cache, the JWKS endpoint will be queried again on demand. However, the broker polls the URL every sasl.oauthbearer.jwks.endpoint.refresh.ms milliseconds to refresh the cache with any forthcoming keys before any JWT requests that include them are received. If the URL is file-based, the broker will load the JWKS file from a configured location on startup. In the event that the JWT includes a "kid" header value that isn't in the JWKS file, the broker will reject the JWT and authentication will fail.
Type: string Default: null Valid Values: Importance: medium -
sasl.oauthbearer.token.endpoint.url
The URL for the OAuth/OIDC identity provider. If the URL is HTTP(S)-based, it is the issuer's token endpoint URL to which requests will be made to login based on the configuration in sasl.jaas.config. If the URL is file-based, it specifies a file containing an access token (in JWT serialized form) issued by the OAuth/OIDC identity provider to use for authorization.
Type: string Default: null Valid Values: Importance: medium -
security.protocol
Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.
Type: string Default: PLAINTEXT Valid Values: (case insensitive) [SASL_SSL, PLAINTEXT, SSL, SASL_PLAINTEXT] Importance: medium -
send.buffer.bytes
The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.
Type: int Default: 131072 (128 kibibytes) Valid Values: [-1,...] Importance: medium -
ssl.enabled.protocols
The list of protocols enabled for SSL connections. The default is 'TLSv1.2,TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). This default should be fine for most cases. Also see the config documentation for `ssl.protocol`.
Type: list Default: TLSv1.2 Valid Values: Importance: medium -
ssl.keystore.type
The file format of the key store file. This is optional for client. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].
Type: string Default: JKS Valid Values: Importance: medium -
ssl.protocol
The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'.
Type: string Default: TLSv1.2 Valid Values: Importance: medium -
ssl.provider
The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.
Type: string Default: null Valid Values: Importance: medium -
ssl.truststore.type
The file format of the trust store file. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].
Type: string Default: JKS Valid Values: Importance: medium -
worker.sync.timeout.ms
When the worker is out of sync with other workers and needs to resynchronize configurations, wait up to this amount of time before giving up, leaving the group, and waiting a backoff period before rejoining.
Type: int Default: 3000 (3 seconds) Valid Values: Importance: medium -
worker.unsync.backoff.ms
When the worker is out of sync with other workers and fails to catch up within worker.sync.timeout.ms, leave the Connect cluster for this long before rejoining.
Type: int Default: 300000 (5 minutes) Valid Values: Importance: medium -
access.control.allow.methods
Sets the methods supported for cross origin requests by setting the Access-Control-Allow-Methods header. The default value of the Access-Control-Allow-Methods header allows cross origin requests for GET, POST and HEAD.
Type: string Default: "" Valid Values: Importance: low -
access.control.allow.origin
Value to set the Access-Control-Allow-Origin header to for REST API requests.To enable cross origin access, set this to the domain of the application that should be permitted to access the API, or '*' to allow access from any domain. The default value only allows access from the domain of the REST API.
Type: string Default: "" Valid Values: Importance: low -
admin.listeners
List of comma-separated URIs the Admin REST API will listen on. The supported protocols are HTTP and HTTPS. An empty or blank string will disable this feature. The default behavior is to use the regular listener (specified by the 'listeners' property).
Type: list Default: null Valid Values: List of comma-separated URLs, ex: http://localhost:8080,https://localhost:8443. Importance: low -
auto.include.jmx.reporter
Deprecated. Whether to automatically include JmxReporter even if it's not listed in
metric.reporters
. This configuration will be removed in Kafka 4.0, users should instead includeorg.apache.kafka.common.metrics.JmxReporter
inmetric.reporters
in order to enable the JmxReporter.Type: boolean Default: true Valid Values: Importance: low -
client.id
An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.
Type: string Default: "" Valid Values: Importance: low -
config.providers
Comma-separated names of
ConfigProvider
classes, loaded and used in the order specified. Implementing the interfaceConfigProvider
allows you to replace variable references in connector configurations, such as for externalized secrets.Type: list Default: "" Valid Values: Importance: low -
config.storage.replication.factor
Replication factor used when creating the configuration storage topic
Type: short Default: 3 Valid Values: Positive number not larger than the number of brokers in the Kafka cluster, or -1 to use the broker's default Importance: low -
connect.protocol
Compatibility mode for Kafka Connect Protocol
Type: string Default: sessioned Valid Values: [eager, compatible, sessioned] Importance: low -
header.converter
HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas.
Type: class Default: org.apache.kafka.connect.storage.SimpleHeaderConverter Valid Values: Importance: low -
inter.worker.key.generation.algorithm
The algorithm to use for generating internal request keys. The algorithm 'HmacSHA256' will be used as a default on JVMs that support it; on other JVMs, no default is used and a value for this property must be manually specified in the worker config.
Type: string Default: HmacSHA256 Valid Values: Any KeyGenerator algorithm supported by the worker JVM Importance: low -
inter.worker.key.size
The size of the key to use for signing internal requests, in bits. If null, the default key size for the key generation algorithm will be used.
Type: int Default: null Valid Values: Importance: low -
inter.worker.key.ttl.ms
The TTL of generated session keys used for internal request validation (in milliseconds)
Type: int Default: 3600000 (1 hour) Valid Values: [0,...,2147483647] Importance: low -
inter.worker.signature.algorithm
The algorithm used to sign internal requestsThe algorithm 'inter.worker.signature.algorithm' will be used as a default on JVMs that support it; on other JVMs, no default is used and a value for this property must be manually specified in the worker config.
Type: string Default: HmacSHA256 Valid Values: Any MAC algorithm supported by the worker JVM Importance: low -
inter.worker.verification.algorithms
A list of permitted algorithms for verifying internal requests, which must include the algorithm used for the inter.worker.signature.algorithm property. The algorithm(s) '[HmacSHA256]' will be used as a default on JVMs that provide them; on other JVMs, no default is used and a value for this property must be manually specified in the worker config.
Type: list Default: HmacSHA256 Valid Values: A list of one or more MAC algorithms, each supported by the worker JVM Importance: low -
listeners
List of comma-separated URIs the REST API will listen on. The supported protocols are HTTP and HTTPS.
Specify hostname as 0.0.0.0 to bind to all interfaces.
Leave hostname empty to bind to default interface.
Examples of legal listener lists: HTTP://myhost:8083,HTTPS://myhost:8084Type: list Default: http://:8083 Valid Values: List of comma-separated URLs, ex: http://localhost:8080,https://localhost:8443. Importance: low -
metadata.max.age.ms
The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.
Type: long Default: 300000 (5 minutes) Valid Values: [0,...] Importance: low -
metadata.recovery.strategy
Controls how the client recovers when none of the brokers known to it is available. If set to
none
, the client fails. If set torebootstrap
, the client repeats the bootstrap process usingbootstrap.servers
. Rebootstrapping is useful when a client communicates with brokers so infrequently that the set of brokers may change entirely before the client refreshes metadata. Metadata recovery is triggered when all last-known brokers appear unavailable simultaneously. Brokers appear unavailable when disconnected and no current retry attempt is in-progress. Consider increasingreconnect.backoff.ms
andreconnect.backoff.max.ms
and decreasingsocket.connection.setup.timeout.ms
andsocket.connection.setup.timeout.max.ms
for the client.Type: string Default: none Valid Values: (case insensitive) [REBOOTSTRAP, NONE] Importance: low -
metric.reporters
A list of classes to use as metrics reporters. Implementing the
org.apache.kafka.common.metrics.MetricsReporter
interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.Type: list Default: "" Valid Values: Importance: low -
metrics.num.samples
The number of samples maintained to compute metrics.
Type: int Default: 2 Valid Values: [1,...] Importance: low -
metrics.recording.level
The highest recording level for metrics.
Type: string Default: INFO Valid Values: [INFO, DEBUG] Importance: low -
metrics.sample.window.ms
The window of time a metrics sample is computed over.
Type: long Default: 30000 (30 seconds) Valid Values: [0,...] Importance: low -
offset.flush.interval.ms
Interval at which to try committing offsets for tasks.
Type: long Default: 60000 (1 minute) Valid Values: Importance: low -
offset.flush.timeout.ms
Maximum number of milliseconds to wait for records to flush and partition offset data to be committed to offset storage before cancelling the process and restoring the offset data to be committed in a future attempt. This property has no effect for source connectors running with exactly-once support.
Type: long Default: 5000 (5 seconds) Valid Values: Importance: low -
offset.storage.partitions
The number of partitions used when creating the offset storage topic
Type: int Default: 25 Valid Values: Positive number, or -1 to use the broker's default Importance: low -
offset.storage.replication.factor
Replication factor used when creating the offset storage topic
Type: short Default: 3 Valid Values: Positive number not larger than the number of brokers in the Kafka cluster, or -1 to use the broker's default Importance: low -
plugin.discovery
Method to use to discover plugins present in the classpath and plugin.path configuration. This can be one of multiple values with the following meanings:
* only_scan: Discover plugins only by reflection. Plugins which are not discoverable by ServiceLoader will not impact worker startup.
* hybrid_warn: Discover plugins reflectively and by ServiceLoader. Plugins which are not discoverable by ServiceLoader will print warnings during worker startup.
* hybrid_fail: Discover plugins reflectively and by ServiceLoader. Plugins which are not discoverable by ServiceLoader will cause worker startup to fail.
* service_load: Discover plugins only by ServiceLoader. Faster startup than other modes. Plugins which are not discoverable by ServiceLoader may not be usable.Type: string Default: hybrid_warn Valid Values: (case insensitive) [ONLY_SCAN, SERVICE_LOAD, HYBRID_WARN, HYBRID_FAIL] Importance: low -
plugin.path
List of paths separated by commas (,) that contain plugins (connectors, converters, transformations). The list should consist of top level directories that include any combination of:
a) directories immediately containing jars with plugins and their dependencies
b) uber-jars with plugins and their dependencies
c) directories immediately containing the package directory structure of classes of plugins and their dependencies
Note: symlinks will be followed to discover dependencies or plugins.
Examples: plugin.path=/usr/local/share/java,/usr/local/share/kafka/plugins,/opt/connectors
Do not use config provider variables in this property, since the raw path is used by the worker's scanner before config providers are initialized and used to replace variables.Type: list Default: null Valid Values: Importance: low -
reconnect.backoff.max.ms
The maximum amount of time in milliseconds to wait when reconnecting to a broker that has repeatedly failed to connect. If provided, the backoff per host will increase exponentially for each consecutive connection failure, up to this maximum. After calculating the backoff increase, 20% random jitter is added to avoid connection storms.
Type: long Default: 1000 (1 second) Valid Values: [0,...] Importance: low -
reconnect.backoff.ms
The base amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all connection attempts by the client to a broker. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the
reconnect.backoff.max.ms
value.Type: long Default: 50 Valid Values: [0,...] Importance: low -
response.http.headers.config
Rules for REST API HTTP response headers
Type: string Default: "" Valid Values: Comma-separated header rules, where each header rule is of the form '[action] [header name]:[header value]' and optionally surrounded by double quotes if any part of a header rule contains a comma Importance: low -
rest.advertised.host.name
If this is set, this is the hostname that will be given out to other workers to connect to.
Type: string Default: null Valid Values: Importance: low -
rest.advertised.listener
Sets the advertised listener (HTTP or HTTPS) which will be given to other workers to use.
Type: string Default: null Valid Values: Importance: low -
rest.advertised.port
If this is set, this is the port that will be given out to other workers to connect to.
Type: int Default: null Valid Values: Importance: low -
rest.extension.classes
Comma-separated names of
ConnectRestExtension
classes, loaded and called in the order specified. Implementing the interfaceConnectRestExtension
allows you to inject into Connect's REST API user defined resources like filters. Typically used to add custom capability like logging, security, etc.Type: list Default: "" Valid Values: Importance: low -
retry.backoff.max.ms
The maximum amount of time in milliseconds to wait when retrying a request to the broker that has repeatedly failed. If provided, the backoff per client will increase exponentially for each failed request, up to this maximum. To prevent all clients from being synchronized upon retry, a randomized jitter with a factor of 0.2 will be applied to the backoff, resulting in the backoff falling within a range between 20% below and 20% above the computed value. If
retry.backoff.ms
is set to be higher thanretry.backoff.max.ms
, thenretry.backoff.max.ms
will be used as a constant backoff from the beginning without any exponential increaseType: long Default: 1000 (1 second) Valid Values: [0,...] Importance: low -
retry.backoff.ms
The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. This value is the initial backoff value and will increase exponentially for each failed request, up to the
retry.backoff.max.ms
value.Type: long Default: 100 Valid Values: [0,...] Importance: low -
sasl.kerberos.kinit.cmd
Kerberos kinit command path.
Type: string Default: /usr/bin/kinit Valid Values: Importance: low -
sasl.kerberos.min.time.before.relogin
Login thread sleep time between refresh attempts.
Type: long Default: 60000 Valid Values: Importance: low -
sasl.kerberos.ticket.renew.jitter
Percentage of random jitter added to the renewal time.
Type: double Default: 0.05 Valid Values: Importance: low -
sasl.kerberos.ticket.renew.window.factor
Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.
Type: double Default: 0.8 Valid Values: Importance: low -
sasl.login.connect.timeout.ms
The (optional) value in milliseconds for the external authentication provider connection timeout. Currently applies only to OAUTHBEARER.
Type: int Default: null Valid Values: Importance: low -
sasl.login.read.timeout.ms
The (optional) value in milliseconds for the external authentication provider read timeout. Currently applies only to OAUTHBEARER.
Type: int Default: null Valid Values: Importance: low -
sasl.login.refresh.buffer.seconds
The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.
Type: short Default: 300 Valid Values: [0,...,3600] Importance: low -
sasl.login.refresh.min.period.seconds
The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.
Type: short Default: 60 Valid Values: [0,...,900] Importance: low -
sasl.login.refresh.window.factor
Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER.
Type: double Default: 0.8 Valid Values: [0.5,...,1.0] Importance: low -
sasl.login.refresh.window.jitter
The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.
Type: double Default: 0.05 Valid Values: [0.0,...,0.25] Importance: low -
sasl.login.retry.backoff.max.ms
The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.
Type: long Default: 10000 (10 seconds) Valid Values: Importance: low -
sasl.login.retry.backoff.ms
The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.
Type: long Default: 100 Valid Values: Importance: low -
sasl.oauthbearer.clock.skew.seconds
The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker.
Type: int Default: 30 Valid Values: Importance: low -
sasl.oauthbearer.expected.audience
The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWT's "aud" claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.
Type: list Default: null Valid Values: Importance: low -
sasl.oauthbearer.expected.issuer
The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth "iss" claim and if this value is set, the broker will match it exactly against what is in the JWT's "iss" claim. If there is no match, the broker will reject the JWT and authentication will fail.
Type: string Default: null Valid Values: Importance: low -
sasl.oauthbearer.header.urlencode
The (optional) setting to enable the OAuth client to URL-encode the client_id and client_secret in the authorization header in accordance with RFC6749, see here for more details. The default value is set to 'false' for backward compatibility
Type: boolean Default: false Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.refresh.ms
The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.
Type: long Default: 3600000 (1 hour) Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms
The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.
Type: long Default: 10000 (10 seconds) Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms
The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.
Type: long Default: 100 Valid Values: Importance: low -
sasl.oauthbearer.scope.claim.name
The OAuth claim for the scope is often named "scope", but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
Type: string Default: scope Valid Values: Importance: low -
sasl.oauthbearer.sub.claim.name
The OAuth claim for the subject is often named "sub", but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
Type: string Default: sub Valid Values: Importance: low -
scheduled.rebalance.max.delay.ms
The maximum delay that is scheduled in order to wait for the return of one or more departed workers before rebalancing and reassigning their connectors and tasks to the group. During this period the connectors and tasks of the departed workers remain unassigned
Type: int Default: 300000 (5 minutes) Valid Values: [0,...,2147483647] Importance: low -
socket.connection.setup.timeout.max.ms
The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. To avoid connection storms, a randomization factor of 0.2 will be applied to the timeout resulting in a random range between 20% below and 20% above the computed value.
Type: long Default: 30000 (30 seconds) Valid Values: [0,...] Importance: low -
socket.connection.setup.timeout.ms
The amount of time the client will wait for the socket connection to be established. If the connection is not built before the timeout elapses, clients will close the socket channel. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the
socket.connection.setup.timeout.max.ms
value.Type: long Default: 10000 (10 seconds) Valid Values: [0,...] Importance: low -
ssl.cipher.suites
A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.
Type: list Default: null Valid Values: Importance: low -
ssl.client.auth
Configures kafka broker to request client authentication. The following settings are common:
ssl.client.auth=required
If set to required client authentication is required.ssl.client.auth=requested
This means client authentication is optional. unlike required, if this option is set client can choose not to provide authentication information about itselfssl.client.auth=none
This means client authentication is not needed.
Type: string Default: none Valid Values: [required, requested, none] Importance: low -
ssl.endpoint.identification.algorithm
The endpoint identification algorithm to validate server hostname using server certificate.
Type: string Default: https Valid Values: Importance: low -
ssl.engine.factory.class
The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is org.apache.kafka.common.security.ssl.DefaultSslEngineFactory. Alternatively, setting this to org.apache.kafka.common.security.ssl.CommonNameLoggingSslEngineFactory will log the common name of expired SSL certificates used by clients to authenticate at any of the brokers with log level INFO. Note that this will cause a tiny delay during establishment of new connections from mTLS clients to brokers due to the extra code for examining the certificate chain provided by the client. Note further that the implementation uses a custom truststore based on the standard Java truststore and thus might be considered a security risk due to not being as mature as the standard one.
Type: class Default: null Valid Values: Importance: low -
ssl.keymanager.algorithm
The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.
Type: string Default: SunX509 Valid Values: Importance: low -
ssl.secure.random.implementation
The SecureRandom PRNG implementation to use for SSL cryptography operations.
Type: string Default: null Valid Values: Importance: low -
ssl.trustmanager.algorithm
The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.
Type: string Default: PKIX Valid Values: Importance: low -
status.storage.partitions
The number of partitions used when creating the status storage topic
Type: int Default: 5 Valid Values: Positive number, or -1 to use the broker's default Importance: low -
status.storage.replication.factor
Replication factor used when creating the status storage topic
Type: short Default: 3 Valid Values: Positive number not larger than the number of brokers in the Kafka cluster, or -1 to use the broker's default Importance: low -
task.shutdown.graceful.timeout.ms
Amount of time to wait for tasks to shutdown gracefully. This is the total amount of time, not per task. All task have shutdown triggered, then they are waited on sequentially.
Type: long Default: 5000 (5 seconds) Valid Values: Importance: low -
topic.creation.enable
Whether to allow automatic creation of topics used by source connectors, when source connectors are configured with `topic.creation.` properties. Each task will use an admin client to create its topics and will not depend on the Kafka brokers to create topics automatically.
Type: boolean Default: true Valid Values: Importance: low -
topic.tracking.allow.reset
If set to true, it allows user requests to reset the set of active topics per connector.
Type: boolean Default: true Valid Values: Importance: low -
topic.tracking.enable
Enable tracking the set of active topics per connector during runtime.
Type: boolean Default: true Valid Values: Importance: low
3.5.1 Source Connector Configs
Below is the configuration of a source connector.-
name
Globally unique name to use for this connector.
Type: string Default: Valid Values: non-empty string without ISO control characters Importance: high -
connector.class
Name or alias of the class for this connector. Must be a subclass of org.apache.kafka.connect.connector.Connector. If the connector is org.apache.kafka.connect.file.FileStreamSinkConnector, you can either specify this full name, or use "FileStreamSink" or "FileStreamSinkConnector" to make the configuration a bit shorter
Type: string Default: Valid Values: Importance: high -
tasks.max
Maximum number of tasks to use for this connector.
Type: int Default: 1 Valid Values: [1,...] Importance: high -
tasks.max.enforce
(Deprecated) Whether to enforce that the tasks.max property is respected by the connector. By default, connectors that generate too many tasks will fail, and existing sets of tasks that exceed the tasks.max property will also be failed. If this property is set to false, then connectors will be allowed to generate more than the maximum number of tasks, and existing sets of tasks that exceed the tasks.max property will be allowed to run. This property is deprecated and will be removed in an upcoming major release.
Type: boolean Default: true Valid Values: Importance: low -
key.converter
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.
Type: class Default: null Valid Values: A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor Importance: low -
value.converter
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.
Type: class Default: null Valid Values: A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor Importance: low -
header.converter
HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas.
Type: class Default: null Valid Values: A concrete subclass of org.apache.kafka.connect.storage.HeaderConverter, A class with a public, no-argument constructor Importance: low -
config.action.reload
The action that Connect should take on the connector when changes in external configuration providers result in a change in the connector's configuration properties. A value of 'none' indicates that Connect will do nothing. A value of 'restart' indicates that Connect should restart/reload the connector with the updated configuration properties.The restart may actually be scheduled in the future if the external configuration provider indicates that a configuration value will expire in the future.
Type: string Default: restart Valid Values: [none, restart] Importance: low -
transforms
Aliases for the transformations to be applied to records.
Type: list Default: "" Valid Values: non-null string, unique transformation aliases Importance: low -
predicates
Aliases for the predicates used by transformations.
Type: list Default: "" Valid Values: non-null string, unique predicate aliases Importance: low -
errors.retry.timeout
The maximum duration in milliseconds that a failed operation will be reattempted. The default is 0, which means no retries will be attempted. Use -1 for infinite retries.
Type: long Default: 0 Valid Values: Importance: medium -
errors.retry.delay.max.ms
The maximum duration in milliseconds between consecutive retry attempts. Jitter will be added to the delay once this limit is reached to prevent thundering herd issues.
Type: long Default: 60000 (1 minute) Valid Values: Importance: medium -
errors.tolerance
Behavior for tolerating errors during connector operation. 'none' is the default value and signals that any error will result in an immediate connector task failure; 'all' changes the behavior to skip over problematic records.
Type: string Default: none Valid Values: [none, all] Importance: medium -
errors.log.enable
If true, write each error and the details of the failed operation and problematic record to the Connect application log. This is 'false' by default, so that only errors that are not tolerated are reported.
Type: boolean Default: false Valid Values: Importance: medium -
errors.log.include.messages
Whether to include in the log the Connect record that resulted in a failure. For sink records, the topic, partition, offset, and timestamp will be logged. For source records, the key and value (and their schemas), all headers, and the timestamp, Kafka topic, Kafka partition, source partition, and source offset will be logged. This is 'false' by default, which will prevent record keys, values, and headers from being written to log files.
Type: boolean Default: false Valid Values: Importance: medium -
topic.creation.groups
Groups of configurations for topics created by source connectors
Type: list Default: "" Valid Values: non-null string, unique topic creation groups Importance: low -
exactly.once.support
Permitted values are requested, required. If set to "required", forces a preflight check for the connector to ensure that it can provide exactly-once semantics with the given configuration. Some connectors may be capable of providing exactly-once semantics but not signal to Connect that they support this; in that case, documentation for the connector should be consulted carefully before creating it, and the value for this property should be set to "requested". Additionally, if the value is set to "required" but the worker that performs preflight validation does not have exactly-once support enabled for source connectors, requests to create or validate the connector will fail.
Type: string Default: requested Valid Values: (case insensitive) [REQUIRED, REQUESTED] Importance: medium -
transaction.boundary
Permitted values are: poll, interval, connector. If set to 'poll', a new producer transaction will be started and committed for every batch of records that each task from this connector provides to Connect. If set to 'connector', relies on connector-defined transaction boundaries; note that not all connectors are capable of defining their own transaction boundaries, and in that case, attempts to instantiate a connector with this value will fail. Finally, if set to 'interval', commits transactions only after a user-defined time interval has passed.
Type: string Default: poll Valid Values: (case insensitive) [INTERVAL, POLL, CONNECTOR] Importance: medium -
transaction.boundary.interval.ms
If 'transaction.boundary' is set to 'interval', determines the interval for producer transaction commits by connector tasks. If unset, defaults to the value of the worker-level 'offset.flush.interval.ms' property. It has no effect if a different transaction.boundary is specified.
Type: long Default: null Valid Values: [0,...] Importance: low -
offsets.storage.topic
The name of a separate offsets topic to use for this connector. If empty or not specified, the worker’s global offsets topic name will be used. If specified, the offsets topic will be created if it does not already exist on the Kafka cluster targeted by this connector (which may be different from the one used for the worker's global offsets topic if the bootstrap.servers property of the connector's producer has been overridden from the worker's). Only applicable in distributed mode; in standalone mode, setting this property will have no effect.
Type: string Default: null Valid Values: non-empty string Importance: low
3.5.2 Sink Connector Configs
Below is the configuration of a sink connector.-
name
Globally unique name to use for this connector.
Type: string Default: Valid Values: non-empty string without ISO control characters Importance: high -
connector.class
Name or alias of the class for this connector. Must be a subclass of org.apache.kafka.connect.connector.Connector. If the connector is org.apache.kafka.connect.file.FileStreamSinkConnector, you can either specify this full name, or use "FileStreamSink" or "FileStreamSinkConnector" to make the configuration a bit shorter
Type: string Default: Valid Values: Importance: high -
tasks.max
Maximum number of tasks to use for this connector.
Type: int Default: 1 Valid Values: [1,...] Importance: high -
topics
List of topics to consume, separated by commas
Type: list Default: "" Valid Values: Importance: high -
topics.regex
Regular expression giving topics to consume. Under the hood, the regex is compiled to a
java.util.regex.Pattern
. Only one of topics or topics.regex should be specified.Type: string Default: "" Valid Values: valid regex Importance: high -
tasks.max.enforce
(Deprecated) Whether to enforce that the tasks.max property is respected by the connector. By default, connectors that generate too many tasks will fail, and existing sets of tasks that exceed the tasks.max property will also be failed. If this property is set to false, then connectors will be allowed to generate more than the maximum number of tasks, and existing sets of tasks that exceed the tasks.max property will be allowed to run. This property is deprecated and will be removed in an upcoming major release.
Type: boolean Default: true Valid Values: Importance: low -
key.converter
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.
Type: class Default: null Valid Values: A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor Importance: low -
value.converter
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.
Type: class Default: null Valid Values: A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor Importance: low -
header.converter
HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas.
Type: class Default: null Valid Values: A concrete subclass of org.apache.kafka.connect.storage.HeaderConverter, A class with a public, no-argument constructor Importance: low -
config.action.reload
The action that Connect should take on the connector when changes in external configuration providers result in a change in the connector's configuration properties. A value of 'none' indicates that Connect will do nothing. A value of 'restart' indicates that Connect should restart/reload the connector with the updated configuration properties.The restart may actually be scheduled in the future if the external configuration provider indicates that a configuration value will expire in the future.
Type: string Default: restart Valid Values: [none, restart] Importance: low -
transforms
Aliases for the transformations to be applied to records.
Type: list Default: "" Valid Values: non-null string, unique transformation aliases Importance: low -
predicates
Aliases for the predicates used by transformations.
Type: list Default: "" Valid Values: non-null string, unique predicate aliases Importance: low -
errors.retry.timeout
The maximum duration in milliseconds that a failed operation will be reattempted. The default is 0, which means no retries will be attempted. Use -1 for infinite retries.
Type: long Default: 0 Valid Values: Importance: medium -
errors.retry.delay.max.ms
The maximum duration in milliseconds between consecutive retry attempts. Jitter will be added to the delay once this limit is reached to prevent thundering herd issues.
Type: long Default: 60000 (1 minute) Valid Values: Importance: medium -
errors.tolerance
Behavior for tolerating errors during connector operation. 'none' is the default value and signals that any error will result in an immediate connector task failure; 'all' changes the behavior to skip over problematic records.
Type: string Default: none Valid Values: [none, all] Importance: medium -
errors.log.enable
If true, write each error and the details of the failed operation and problematic record to the Connect application log. This is 'false' by default, so that only errors that are not tolerated are reported.
Type: boolean Default: false Valid Values: Importance: medium -
errors.log.include.messages
Whether to include in the log the Connect record that resulted in a failure. For sink records, the topic, partition, offset, and timestamp will be logged. For source records, the key and value (and their schemas), all headers, and the timestamp, Kafka topic, Kafka partition, source partition, and source offset will be logged. This is 'false' by default, which will prevent record keys, values, and headers from being written to log files.
Type: boolean Default: false Valid Values: Importance: medium -
errors.deadletterqueue.topic.name
The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, or its transformations or converters. The topic name is blank by default, which means that no messages are to be recorded in the DLQ.
Type: string Default: "" Valid Values: Importance: medium -
errors.deadletterqueue.topic.replication.factor
Replication factor used to create the dead letter queue topic when it doesn't already exist.
Type: short Default: 3 Valid Values: Importance: medium -
errors.deadletterqueue.context.headers.enable
If true, add headers containing error context to the messages written to the dead letter queue. To avoid clashing with headers from the original record, all error context header keys, all error context header keys will start with
__connect.errors.
Type: boolean Default: false Valid Values: Importance: medium
3.6 Kafka Streams Configs
Below is the configuration of the Kafka Streams client library.-
application.id
An identifier for the stream processing application. Must be unique within the Kafka cluster. It is used as 1) the default client-id prefix, 2) the group-id for membership management, 3) the changelog topic prefix.
Type: string Default: Valid Values: Importance: high -
bootstrap.servers
A list of host/port pairs used to establish the initial connection to the Kafka cluster. Clients use this list to bootstrap and discover the full set of Kafka brokers. While the order of servers in the list does not matter, we recommend including more than one server to ensure resilience if any servers are down. This list does not need to contain the entire set of brokers, as Kafka clients automatically manage and update connections to the cluster efficiently. This list must be in the form
host1:port1,host2:port2,...
.Type: list Default: Valid Values: Importance: high -
num.standby.replicas
The number of standby replicas for each task.
Type: int Default: 0 Valid Values: Importance: high -
state.dir
Directory location for state store. This path must be unique for each streams instance sharing the same underlying filesystem. Note that if not configured, then the default location will be different in each environment as it is computed using System.getProperty("java.io.tmpdir")
Type: string Default: ${java.io.tmpdir} Valid Values: Importance: high -
acceptable.recovery.lag
The maximum acceptable lag (number of offsets to catch up) for a client to be considered caught-up enough to receive an active task assignment. Upon assignment, it will still restore the rest of the changelog before processing. To avoid a pause in processing during rebalances, this config should correspond to a recovery time of well under a minute for a given workload. Must be at least 0.
Type: long Default: 10000 Valid Values: [0,...] Importance: medium -
cache.max.bytes.buffering
Maximum number of memory bytes to be used for buffering across all threads
Type: long Default: 10485760 Valid Values: [0,...] Importance: medium -
client.id
An ID prefix string used for the client IDs of internal (main, restore, and global) consumers , producers, and admin clients with pattern
<client.id>-[Global]StreamThread[-<threadSequenceNumber>]-<consumer|producer|restore-consumer|global-consumer>
.Type: string Default: <application.id>-<random-UUID>
Valid Values: Importance: medium -
default.deserialization.exception.handler
Exception handling class that implements the
org.apache.kafka.streams.errors.DeserializationExceptionHandler
interface.Type: class Default: org.apache.kafka.streams.errors.LogAndFailExceptionHandler Valid Values: Importance: medium -
default.key.serde
Default serializer / deserializer class for key that implements the
org.apache.kafka.common.serialization.Serde
interface. Note when windowed serde class is used, one needs to set the inner serde class that implements theorg.apache.kafka.common.serialization.Serde
interface via 'default.windowed.key.serde.inner' or 'default.windowed.value.serde.inner' as wellType: class Default: null Valid Values: Importance: medium -
default.list.key.serde.inner
Default inner class of list serde for key that implements the
org.apache.kafka.common.serialization.Serde
interface. This configuration will be read if and only ifdefault.key.serde
configuration is set toorg.apache.kafka.common.serialization.Serdes.ListSerde
Type: class Default: null Valid Values: Importance: medium -
default.list.key.serde.type
Default class for key that implements the
java.util.List
interface. This configuration will be read if and only ifdefault.key.serde
configuration is set toorg.apache.kafka.common.serialization.Serdes.ListSerde
Note when list serde class is used, one needs to set the inner serde class that implements theorg.apache.kafka.common.serialization.Serde
interface via 'default.list.key.serde.inner'Type: class Default: null Valid Values: Importance: medium -
default.list.value.serde.inner
Default inner class of list serde for value that implements the
org.apache.kafka.common.serialization.Serde
interface. This configuration will be read if and only ifdefault.value.serde
configuration is set toorg.apache.kafka.common.serialization.Serdes.ListSerde
Type: class Default: null Valid Values: Importance: medium -
default.list.value.serde.type
Default class for value that implements the
java.util.List
interface. This configuration will be read if and only ifdefault.value.serde
configuration is set toorg.apache.kafka.common.serialization.Serdes.ListSerde
Note when list serde class is used, one needs to set the inner serde class that implements theorg.apache.kafka.common.serialization.Serde
interface via 'default.list.value.serde.inner'Type: class Default: null Valid Values: Importance: medium -
default.production.exception.handler
Exception handling class that implements the
org.apache.kafka.streams.errors.ProductionExceptionHandler
interface.Type: class Default: org.apache.kafka.streams.errors.DefaultProductionExceptionHandler Valid Values: Importance: medium -
default.timestamp.extractor
Default timestamp extractor class that implements the
org.apache.kafka.streams.processor.TimestampExtractor
interface.Type: class Default: org.apache.kafka.streams.processor.FailOnInvalidTimestamp Valid Values: Importance: medium -
default.value.serde
Default serializer / deserializer class for value that implements the
org.apache.kafka.common.serialization.Serde
interface. Note when windowed serde class is used, one needs to set the inner serde class that implements theorg.apache.kafka.common.serialization.Serde
interface via 'default.windowed.key.serde.inner' or 'default.windowed.value.serde.inner' as wellType: class Default: null Valid Values: Importance: medium -
max.task.idle.ms
This config controls whether joins and merges may produce out-of-order results. The config value is the maximum amount of time in milliseconds a stream task will stay idle when it is fully caught up on some (but not all) input partitions to wait for producers to send additional records and avoid potential out-of-order record processing across multiple input streams. The default (zero) does not wait for producers to send more records, but it does wait to fetch data that is already present on the brokers. This default means that for records that are already present on the brokers, Streams will process them in timestamp order. Set to -1 to disable idling entirely and process any locally available data, even though doing so may produce out-of-order processing.
Type: long Default: 0 Valid Values: Importance: medium -
max.warmup.replicas
The maximum number of warmup replicas (extra standbys beyond the configured num.standbys) that can be assigned at once for the purpose of keeping the task available on one instance while it is warming up on another instance it has been reassigned to. Used to throttle how much extra broker traffic and cluster state can be used for high availability. Must be at least 1.Note that one warmup replica corresponds to one Stream Task. Furthermore, note that each warmup replica can only be promoted to an active task during a rebalance (normally during a so-called probing rebalance, which occur at a frequency specified by the `probing.rebalance.interval.ms` config). This means that the maximum rate at which active tasks can be migrated from one Kafka Streams Instance to another instance can be determined by (`max.warmup.replicas` / `probing.rebalance.interval.ms`).
Type: int Default: 2 Valid Values: [1,...] Importance: medium -
num.stream.threads
The number of threads to execute stream processing.
Type: int Default: 1 Valid Values: Importance: medium -
processing.exception.handler
Exception handling class that implements the
org.apache.kafka.streams.errors.ProcessingExceptionHandler
interface.Type: class Default: org.apache.kafka.streams.errors.LogAndFailProcessingExceptionHandler Valid Values: Importance: medium -
processing.guarantee
The processing guarantee that should be used. Possible values are
at_least_once
(default) andexactly_once_v2
(requires brokers version 2.5 or higher). Deprecated options areexactly_once
(requires brokers version 0.11.0 or higher) andexactly_once_beta
(requires brokers version 2.5 or higher). Note that exactly-once processing requires a cluster of at least three brokers by default what is the recommended setting for production; for development you can change this, by adjusting broker settingtransaction.state.log.replication.factor
andtransaction.state.log.min.isr
.Type: string Default: at_least_once Valid Values: [at_least_once, exactly_once, exactly_once_beta, exactly_once_v2] Importance: medium -
rack.aware.assignment.non_overlap_cost
Cost associated with moving tasks from existing assignment. This config and
rack.aware.assignment.traffic_cost
controls whether the optimization algorithm favors minimizing cross rack traffic or minimize the movement of tasks in existing assignment. If set a larger valueorg.apache.kafka.streams.processor.internals.assignment.RackAwareTaskAssignor
will optimize to maintain the existing assignment. The default value is null which means it will use default non_overlap cost values in different assignors.Type: int Default: null Valid Values: Importance: medium -
rack.aware.assignment.strategy
The strategy we use for rack aware assignment. Rack aware assignment will take
client.rack
andracks
ofTopicPartition
into account when assigning tasks to minimize cross rack traffic. Valid settings are :none
(default), which will disable rack aware assignment;min_traffic
, which will compute minimum cross rack traffic assignment;balance_subtopology
, which will compute minimum cross rack traffic and try to balance the tasks of same subtopolgies across different clientsType: string Default: none Valid Values: [none, min_traffic, balance_subtopology] Importance: medium -
rack.aware.assignment.tags
List of client tag keys used to distribute standby replicas across Kafka Streams instances. When configured, Kafka Streams will make a best-effort to distribute the standby tasks over each client tag dimension.
Type: list Default: "" Valid Values: List containing maximum of 5 elements Importance: medium -
rack.aware.assignment.traffic_cost
Cost associated with cross rack traffic. This config and
rack.aware.assignment.non_overlap_cost
controls whether the optimization algorithm favors minimizing cross rack traffic or minimize the movement of tasks in existing assignment. If set a larger valueorg.apache.kafka.streams.processor.internals.assignment.RackAwareTaskAssignor
will optimize for minimizing cross rack traffic. The default value is null which means it will use default traffic cost values in different assignors.Type: int Default: null Valid Values: Importance: medium -
replication.factor
The replication factor for change log topics and repartition topics created by the stream processing application. The default of
-1
(meaning: use broker default replication factor) requires broker version 2.4 or newerType: int Default: -1 Valid Values: Importance: medium -
security.protocol
Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.
Type: string Default: PLAINTEXT Valid Values: (case insensitive) [SASL_SSL, PLAINTEXT, SSL, SASL_PLAINTEXT] Importance: medium -
statestore.cache.max.bytes
Maximum number of memory bytes to be used for statestore cache across all threads
Type: long Default: 10485760 (10 mebibytes) Valid Values: [0,...] Importance: medium -
task.assignor.class
A task assignor class or class name implementing the
org.apache.kafka.streams.processor.assignment.TaskAssignor
interface. Defaults to theHighAvailabilityTaskAssignor
class.Type: string Default: null Valid Values: Importance: medium -
task.timeout.ms
The maximum amount of time in milliseconds a task might stall due to internal errors and retries until an error is raised. For a timeout of 0ms, a task would raise an error for the first internal error. For any timeout larger than 0ms, a task will retry at least once before an error is raised.
Type: long Default: 300000 (5 minutes) Valid Values: [0,...] Importance: medium -
topology.optimization
A configuration telling Kafka Streams if it should optimize the topology and what optimizations to apply. Acceptable values are: "+NO_OPTIMIZATION+", "+OPTIMIZE+", or a comma separated list of specific optimizations: ("+REUSE_KTABLE_SOURCE_TOPICS+", "+MERGE_REPARTITION_TOPICS+" + "SINGLE_STORE_SELF_JOIN+")."NO_OPTIMIZATION" by default.
Type: string Default: none Valid Values: [all, none, reuse.ktable.source.topics, merge.repartition.topics, single.store.self.join] Importance: medium -
application.server
A host:port pair pointing to a user-defined endpoint that can be used for state store discovery and interactive queries on this KafkaStreams instance.
Type: string Default: "" Valid Values: Importance: low -
auto.include.jmx.reporter
Deprecated. Whether to automatically include JmxReporter even if it's not listed in
metric.reporters
. This configuration will be removed in Kafka 4.0, users should instead includeorg.apache.kafka.common.metrics.JmxReporter
inmetric.reporters
in order to enable the JmxReporter.Type: boolean Default: true Valid Values: Importance: low -
buffered.records.per.partition
Maximum number of records to buffer per partition.
Type: int Default: 1000 Valid Values: Importance: low -
built.in.metrics.version
Version of the built-in metrics to use.
Type: string Default: latest Valid Values: [latest] Importance: low -
commit.interval.ms
The frequency in milliseconds with which to commit processing progress. For at-least-once processing, committing means to save the position (ie, offsets) of the processor. For exactly-once processing, it means to commit the transaction which includes to save the position and to make the committed data in the output topic visible to consumers with isolation level read_committed. (Note, if
processing.guarantee
is set toexactly_once_v2
,exactly_once
,the default value is100
, otherwise the default value is30000
.Type: long Default: 30000 (30 seconds) Valid Values: [0,...] Importance: low -
connections.max.idle.ms
Close idle connections after the number of milliseconds specified by this config.
Type: long Default: 540000 (9 minutes) Valid Values: Importance: low -
default.client.supplier
Client supplier class that implements the
org.apache.kafka.streams.KafkaClientSupplier
interface.Type: class Default: org.apache.kafka.streams.processor.internals.DefaultKafkaClientSupplier Valid Values: Importance: low -
default.dsl.store
The default state store type used by DSL operators.
Type: string Default: rocksDB Valid Values: [rocksDB, in_memory] Importance: low -
dsl.store.suppliers.class
Defines which store implementations to plug in to DSL operators. Must implement the
org.apache.kafka.streams.state.DslStoreSuppliers
interface.Type: class Default: org.apache.kafka.streams.state.BuiltInDslStoreSuppliers$RocksDBDslStoreSuppliers Valid Values: Importance: low -
enable.metrics.push
Whether to enable pushing of internal client metrics for (main, restore, and global) consumers, producers, and admin clients. The cluster must have a client metrics subscription which corresponds to a client.
Type: boolean Default: true Valid Values: Importance: low -
log.summary.interval.ms
This configuration controls the output interval for summary information.
If greater or equal to 0, the summary log will be output according to the set time interval;
If less than 0, summary output is disabled.Type: long Default: 120000 (2 minutes) Valid Values: Importance: low -
metadata.max.age.ms
The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.
Type: long Default: 300000 (5 minutes) Valid Values: [0,...] Importance: low -
metric.reporters
A list of classes to use as metrics reporters. Implementing the
org.apache.kafka.common.metrics.MetricsReporter
interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.Type: list Default: "" Valid Values: Importance: low -
metrics.num.samples
The number of samples maintained to compute metrics.
Type: int Default: 2 Valid Values: [1,...] Importance: low -
metrics.recording.level
The highest recording level for metrics.
Type: string Default: INFO Valid Values: [INFO, DEBUG, TRACE] Importance: low -
metrics.sample.window.ms
The window of time a metrics sample is computed over.
Type: long Default: 30000 (30 seconds) Valid Values: [0,...] Importance: low -
poll.ms
The amount of time in milliseconds to block waiting for input.
Type: long Default: 100 Valid Values: Importance: low -
probing.rebalance.interval.ms
The maximum time in milliseconds to wait before triggering a rebalance to probe for warmup replicas that have finished warming up and are ready to become active. Probing rebalances will continue to be triggered until the assignment is balanced. Must be at least 1 minute.
Type: long Default: 600000 (10 minutes) Valid Values: [60000,...] Importance: low -
receive.buffer.bytes
The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.
Type: int Default: 32768 (32 kibibytes) Valid Values: [-1,...] Importance: low -
reconnect.backoff.max.ms
The maximum amount of time in milliseconds to wait when reconnecting to a broker that has repeatedly failed to connect. If provided, the backoff per host will increase exponentially for each consecutive connection failure, up to this maximum. After calculating the backoff increase, 20% random jitter is added to avoid connection storms.
Type: long Default: 1000 (1 second) Valid Values: [0,...] Importance: low -
reconnect.backoff.ms
The base amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all connection attempts by the client to a broker. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the
reconnect.backoff.max.ms
value.Type: long Default: 50 Valid Values: [0,...] Importance: low -
repartition.purge.interval.ms
The frequency in milliseconds with which to delete fully consumed records from repartition topics. Purging will occur after at least this value since the last purge, but may be delayed until later. (Note, unlike
commit.interval.ms
, the default for this value remains unchanged whenprocessing.guarantee
is set toexactly_once_v2
).Type: long Default: 30000 (30 seconds) Valid Values: [0,...] Importance: low -
request.timeout.ms
The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.
Type: int Default: 40000 (40 seconds) Valid Values: [0,...] Importance: low -
retries
Setting a value greater than zero will cause the client to resend any request that fails with a potentially transient error. It is recommended to set the value to either zero or `MAX_VALUE` and use corresponding timeout parameters to control how long a client should retry a request.
Type: int Default: 0 Valid Values: [0,...,2147483647] Importance: low -
retry.backoff.ms
The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. This value is the initial backoff value and will increase exponentially for each failed request, up to the
retry.backoff.max.ms
value.Type: long Default: 100 Valid Values: [0,...] Importance: low -
rocksdb.config.setter
A Rocks DB config setter class or class name that implements the
org.apache.kafka.streams.state.RocksDBConfigSetter
interfaceType: class Default: null Valid Values: Importance: low -
send.buffer.bytes
The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.
Type: int Default: 131072 (128 kibibytes) Valid Values: [-1,...] Importance: low -
state.cleanup.delay.ms
The amount of time in milliseconds to wait before deleting state when a partition has migrated. Only state directories that have not been modified for at least
state.cleanup.delay.ms
will be removedType: long Default: 600000 (10 minutes) Valid Values: Importance: low -
upgrade.from
Allows upgrading in a backward compatible way. This is needed when upgrading from [0.10.0, 1.1] to 2.0+, or when upgrading from [2.0, 2.3] to 2.4+. When upgrading from 3.3 to a newer version it is not required to specify this config. Default is `null`. Accepted values are "0.10.0", "0.10.1", "0.10.2", "0.11.0", "1.0", "1.1", "2.0", "2.1", "2.2", "2.3", "2.4", "2.5", "2.6", "2.7", "2.8", "3.0", "3.1", "3.2", "3.3", "3.4", "3.5", "3.6", "3.7", "3.8(for upgrading from the corresponding old version).
Type: string Default: null Valid Values: [null, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0, 1.1, 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8] Importance: low -
window.size.ms
Sets window size for the deserializer in order to calculate window end times.
Type: long Default: null Valid Values: Importance: low -
windowed.inner.class.serde
Default serializer / deserializer for the inner class of a windowed record. Must implement the
org.apache.kafka.common.serialization.Serde
interface. Note that setting this config in KafkaStreams application would result in an error as it is meant to be used only from Plain consumer client.Type: string Default: null Valid Values: Importance: low -
windowstore.changelog.additional.retention.ms
Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day
Type: long Default: 86400000 (1 day) Valid Values: Importance: low
3.7 Admin Configs
Below is the configuration of the Kafka Admin client library.-
bootstrap.controllers
A list of host/port pairs to use for establishing the initial connection to the KRaft controller quorum. This list should be in the form
host1:port1,host2:port2,...
.Type: list Default: "" Valid Values: Importance: high -
bootstrap.servers
A list of host/port pairs used to establish the initial connection to the Kafka cluster. Clients use this list to bootstrap and discover the full set of Kafka brokers. While the order of servers in the list does not matter, we recommend including more than one server to ensure resilience if any servers are down. This list does not need to contain the entire set of brokers, as Kafka clients automatically manage and update connections to the cluster efficiently. This list must be in the form
host1:port1,host2:port2,...
.Type: list Default: "" Valid Values: Importance: high -
ssl.key.password
The password of the private key in the key store file or the PEM key specified in 'ssl.keystore.key'.
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.certificate.chain
Certificate chain in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with a list of X.509 certificates
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.key
Private key in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with PKCS#8 keys. If the key is encrypted, key password must be specified using 'ssl.key.password'
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.location
The location of the key store file. This is optional for client and can be used for two-way authentication for client.
Type: string Default: null Valid Values: Importance: high -
ssl.keystore.password
The store password for the key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured. Key store password is not supported for PEM format.
Type: password Default: null Valid Values: Importance: high -
ssl.truststore.certificates
Trusted certificates in the format specified by 'ssl.truststore.type'. Default SSL engine factory supports only PEM format with X.509 certificates.
Type: password Default: null Valid Values: Importance: high -
ssl.truststore.location
The location of the trust store file.
Type: string Default: null Valid Values: Importance: high -
ssl.truststore.password
The password for the trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled. Trust store password is not supported for PEM format.
Type: password Default: null Valid Values: Importance: high -
client.dns.lookup
Controls how the client uses DNS lookups. If set to
use_all_dns_ips
, connect to each returned IP address in sequence until a successful connection is established. After a disconnection, the next IP is used. Once all IPs have been used once, the client resolves the IP(s) from the hostname again (both the JVM and the OS cache DNS name lookups, however). If set toresolve_canonical_bootstrap_servers_only
, resolve each bootstrap address into a list of canonical names. After the bootstrap phase, this behaves the same asuse_all_dns_ips
.Type: string Default: use_all_dns_ips Valid Values: [use_all_dns_ips, resolve_canonical_bootstrap_servers_only] Importance: medium -
client.id
An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.
Type: string Default: "" Valid Values: Importance: medium -
connections.max.idle.ms
Close idle connections after the number of milliseconds specified by this config.
Type: long Default: 300000 (5 minutes) Valid Values: Importance: medium -
default.api.timeout.ms
Specifies the timeout (in milliseconds) for client APIs. This configuration is used as the default timeout for all client operations that do not specify a
timeout
parameter.Type: int Default: 60000 (1 minute) Valid Values: [0,...] Importance: medium -
receive.buffer.bytes
The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.
Type: int Default: 65536 (64 kibibytes) Valid Values: [-1,...] Importance: medium -
request.timeout.ms
The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.
Type: int Default: 30000 (30 seconds) Valid Values: [0,...] Importance: medium -
sasl.client.callback.handler.class
The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface.
Type: class Default: null Valid Values: Importance: medium -
sasl.jaas.config
JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is:
loginModuleClass controlFlag (optionName=optionValue)*;
. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;Type: password Default: null Valid Values: Importance: medium -
sasl.kerberos.service.name
The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.
Type: string Default: null Valid Values: Importance: medium -
sasl.login.callback.handler.class
The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler
Type: class Default: null Valid Values: Importance: medium -
sasl.login.class
The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin
Type: class Default: null Valid Values: Importance: medium -
sasl.mechanism
SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.
Type: string Default: GSSAPI Valid Values: Importance: medium -
sasl.oauthbearer.jwks.endpoint.url
The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. If the URL is HTTP(S)-based, the JWKS data will be retrieved from the OAuth/OIDC provider via the configured URL on broker startup. All then-current keys will be cached on the broker for incoming requests. If an authentication request is received for a JWT that includes a "kid" header claim value that isn't yet in the cache, the JWKS endpoint will be queried again on demand. However, the broker polls the URL every sasl.oauthbearer.jwks.endpoint.refresh.ms milliseconds to refresh the cache with any forthcoming keys before any JWT requests that include them are received. If the URL is file-based, the broker will load the JWKS file from a configured location on startup. In the event that the JWT includes a "kid" header value that isn't in the JWKS file, the broker will reject the JWT and authentication will fail.
Type: string Default: null Valid Values: Importance: medium -
sasl.oauthbearer.token.endpoint.url
The URL for the OAuth/OIDC identity provider. If the URL is HTTP(S)-based, it is the issuer's token endpoint URL to which requests will be made to login based on the configuration in sasl.jaas.config. If the URL is file-based, it specifies a file containing an access token (in JWT serialized form) issued by the OAuth/OIDC identity provider to use for authorization.
Type: string Default: null Valid Values: Importance: medium -
security.protocol
Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.
Type: string Default: PLAINTEXT Valid Values: (case insensitive) [SASL_SSL, PLAINTEXT, SSL, SASL_PLAINTEXT] Importance: medium -
send.buffer.bytes
The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.
Type: int Default: 131072 (128 kibibytes) Valid Values: [-1,...] Importance: medium -
socket.connection.setup.timeout.max.ms
The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. To avoid connection storms, a randomization factor of 0.2 will be applied to the timeout resulting in a random range between 20% below and 20% above the computed value.
Type: long Default: 30000 (30 seconds) Valid Values: Importance: medium -
socket.connection.setup.timeout.ms
The amount of time the client will wait for the socket connection to be established. If the connection is not built before the timeout elapses, clients will close the socket channel. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the
socket.connection.setup.timeout.max.ms
value.Type: long Default: 10000 (10 seconds) Valid Values: Importance: medium -
ssl.enabled.protocols
The list of protocols enabled for SSL connections. The default is 'TLSv1.2,TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). This default should be fine for most cases. Also see the config documentation for `ssl.protocol`.
Type: list Default: TLSv1.2 Valid Values: Importance: medium -
ssl.keystore.type
The file format of the key store file. This is optional for client. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].
Type: string Default: JKS Valid Values: Importance: medium -
ssl.protocol
The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'.
Type: string Default: TLSv1.2 Valid Values: Importance: medium -
ssl.provider
The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.
Type: string Default: null Valid Values: Importance: medium -
ssl.truststore.type
The file format of the trust store file. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].
Type: string Default: JKS Valid Values: Importance: medium -
auto.include.jmx.reporter
Deprecated. Whether to automatically include JmxReporter even if it's not listed in
metric.reporters
. This configuration will be removed in Kafka 4.0, users should instead includeorg.apache.kafka.common.metrics.JmxReporter
inmetric.reporters
in order to enable the JmxReporter.Type: boolean Default: true Valid Values: Importance: low -
enable.metrics.push
Whether to enable pushing of client metrics to the cluster, if the cluster has a client metrics subscription which matches this client.
Type: boolean Default: true Valid Values: Importance: low -
metadata.max.age.ms
The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.
Type: long Default: 300000 (5 minutes) Valid Values: [0,...] Importance: low -
metadata.recovery.strategy
Controls how the client recovers when none of the brokers known to it is available. If set to
none
, the client fails. If set torebootstrap
, the client repeats the bootstrap process usingbootstrap.servers
. Rebootstrapping is useful when a client communicates with brokers so infrequently that the set of brokers may change entirely before the client refreshes metadata. Metadata recovery is triggered when all last-known brokers appear unavailable simultaneously. Brokers appear unavailable when disconnected and no current retry attempt is in-progress. Consider increasingreconnect.backoff.ms
andreconnect.backoff.max.ms
and decreasingsocket.connection.setup.timeout.ms
andsocket.connection.setup.timeout.max.ms
for the client.Type: string Default: none Valid Values: (case insensitive) [REBOOTSTRAP, NONE] Importance: low -
metric.reporters
A list of classes to use as metrics reporters. Implementing the
org.apache.kafka.common.metrics.MetricsReporter
interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.Type: list Default: "" Valid Values: Importance: low -
metrics.num.samples
The number of samples maintained to compute metrics.
Type: int Default: 2 Valid Values: [1,...] Importance: low -
metrics.recording.level
The highest recording level for metrics.
Type: string Default: INFO Valid Values: [INFO, DEBUG, TRACE] Importance: low -
metrics.sample.window.ms
The window of time a metrics sample is computed over.
Type: long Default: 30000 (30 seconds) Valid Values: [0,...] Importance: low -
reconnect.backoff.max.ms
The maximum amount of time in milliseconds to wait when reconnecting to a broker that has repeatedly failed to connect. If provided, the backoff per host will increase exponentially for each consecutive connection failure, up to this maximum. After calculating the backoff increase, 20% random jitter is added to avoid connection storms.
Type: long Default: 1000 (1 second) Valid Values: [0,...] Importance: low -
reconnect.backoff.ms
The base amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all connection attempts by the client to a broker. This value is the initial backoff value and will increase exponentially for each consecutive connection failure, up to the
reconnect.backoff.max.ms
value.Type: long Default: 50 Valid Values: [0,...] Importance: low -
retries
Setting a value greater than zero will cause the client to resend any request that fails with a potentially transient error. It is recommended to set the value to either zero or `MAX_VALUE` and use corresponding timeout parameters to control how long a client should retry a request.
Type: int Default: 2147483647 Valid Values: [0,...,2147483647] Importance: low -
retry.backoff.max.ms
The maximum amount of time in milliseconds to wait when retrying a request to the broker that has repeatedly failed. If provided, the backoff per client will increase exponentially for each failed request, up to this maximum. To prevent all clients from being synchronized upon retry, a randomized jitter with a factor of 0.2 will be applied to the backoff, resulting in the backoff falling within a range between 20% below and 20% above the computed value. If
retry.backoff.ms
is set to be higher thanretry.backoff.max.ms
, thenretry.backoff.max.ms
will be used as a constant backoff from the beginning without any exponential increaseType: long Default: 1000 (1 second) Valid Values: [0,...] Importance: low -
retry.backoff.ms
The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios. This value is the initial backoff value and will increase exponentially for each failed request, up to the
retry.backoff.max.ms
value.Type: long Default: 100 Valid Values: [0,...] Importance: low -
sasl.kerberos.kinit.cmd
Kerberos kinit command path.
Type: string Default: /usr/bin/kinit Valid Values: Importance: low -
sasl.kerberos.min.time.before.relogin
Login thread sleep time between refresh attempts.
Type: long Default: 60000 Valid Values: Importance: low -
sasl.kerberos.ticket.renew.jitter
Percentage of random jitter added to the renewal time.
Type: double Default: 0.05 Valid Values: Importance: low -
sasl.kerberos.ticket.renew.window.factor
Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.
Type: double Default: 0.8 Valid Values: Importance: low -
sasl.login.connect.timeout.ms
The (optional) value in milliseconds for the external authentication provider connection timeout. Currently applies only to OAUTHBEARER.
Type: int Default: null Valid Values: Importance: low -
sasl.login.read.timeout.ms
The (optional) value in milliseconds for the external authentication provider read timeout. Currently applies only to OAUTHBEARER.
Type: int Default: null Valid Values: Importance: low -
sasl.login.refresh.buffer.seconds
The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.
Type: short Default: 300 Valid Values: [0,...,3600] Importance: low -
sasl.login.refresh.min.period.seconds
The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.
Type: short Default: 60 Valid Values: [0,...,900] Importance: low -
sasl.login.refresh.window.factor
Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER.
Type: double Default: 0.8 Valid Values: [0.5,...,1.0] Importance: low -
sasl.login.refresh.window.jitter
The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.
Type: double Default: 0.05 Valid Values: [0.0,...,0.25] Importance: low -
sasl.login.retry.backoff.max.ms
The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.
Type: long Default: 10000 (10 seconds) Valid Values: Importance: low -
sasl.login.retry.backoff.ms
The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.
Type: long Default: 100 Valid Values: Importance: low -
sasl.oauthbearer.clock.skew.seconds
The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker.
Type: int Default: 30 Valid Values: Importance: low -
sasl.oauthbearer.expected.audience
The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWT's "aud" claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.
Type: list Default: null Valid Values: Importance: low -
sasl.oauthbearer.expected.issuer
The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth "iss" claim and if this value is set, the broker will match it exactly against what is in the JWT's "iss" claim. If there is no match, the broker will reject the JWT and authentication will fail.
Type: string Default: null Valid Values: Importance: low -
sasl.oauthbearer.header.urlencode
The (optional) setting to enable the OAuth client to URL-encode the client_id and client_secret in the authorization header in accordance with RFC6749, see here for more details. The default value is set to 'false' for backward compatibility
Type: boolean Default: false Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.refresh.ms
The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.
Type: long Default: 3600000 (1 hour) Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms
The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.
Type: long Default: 10000 (10 seconds) Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms
The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.
Type: long Default: 100 Valid Values: Importance: low -
sasl.oauthbearer.scope.claim.name
The OAuth claim for the scope is often named "scope", but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
Type: string Default: scope Valid Values: Importance: low -
sasl.oauthbearer.sub.claim.name
The OAuth claim for the subject is often named "sub", but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
Type: string Default: sub Valid Values: Importance: low -
security.providers
A list of configurable creator classes each returning a provider implementing security algorithms. These classes should implement the
org.apache.kafka.common.security.auth.SecurityProviderCreator
interface.Type: string Default: null Valid Values: Importance: low -
ssl.cipher.suites
A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.
Type: list Default: null Valid Values: Importance: low -
ssl.endpoint.identification.algorithm
The endpoint identification algorithm to validate server hostname using server certificate.
Type: string Default: https Valid Values: Importance: low -
ssl.engine.factory.class
The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is org.apache.kafka.common.security.ssl.DefaultSslEngineFactory. Alternatively, setting this to org.apache.kafka.common.security.ssl.CommonNameLoggingSslEngineFactory will log the common name of expired SSL certificates used by clients to authenticate at any of the brokers with log level INFO. Note that this will cause a tiny delay during establishment of new connections from mTLS clients to brokers due to the extra code for examining the certificate chain provided by the client. Note further that the implementation uses a custom truststore based on the standard Java truststore and thus might be considered a security risk due to not being as mature as the standard one.
Type: class Default: null Valid Values: Importance: low -
ssl.keymanager.algorithm
The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.
Type: string Default: SunX509 Valid Values: Importance: low -
ssl.secure.random.implementation
The SecureRandom PRNG implementation to use for SSL cryptography operations.
Type: string Default: null Valid Values: Importance: low -
ssl.trustmanager.algorithm
The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.
Type: string Default: PKIX Valid Values: Importance: low
3.8 MirrorMaker Configs
Below is the configuration of the connectors that make up MirrorMaker 2.3.8.1 MirrorMaker Common Configs
Below are the common configuration properties that apply to all three connectors.-
source.cluster.alias
Alias of source cluster
Type: string Default: Valid Values: Importance: high -
ssl.key.password
The password of the private key in the key store file or the PEM key specified in 'ssl.keystore.key'.
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.certificate.chain
Certificate chain in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with a list of X.509 certificates
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.key
Private key in the format specified by 'ssl.keystore.type'. Default SSL engine factory supports only PEM format with PKCS#8 keys. If the key is encrypted, key password must be specified using 'ssl.key.password'
Type: password Default: null Valid Values: Importance: high -
ssl.keystore.location
The location of the key store file. This is optional for client and can be used for two-way authentication for client.
Type: string Default: null Valid Values: Importance: high -
ssl.keystore.password
The store password for the key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured. Key store password is not supported for PEM format.
Type: password Default: null Valid Values: Importance: high -
ssl.truststore.certificates
Trusted certificates in the format specified by 'ssl.truststore.type'. Default SSL engine factory supports only PEM format with X.509 certificates.
Type: password Default: null Valid Values: Importance: high -
ssl.truststore.location
The location of the trust store file.
Type: string Default: null Valid Values: Importance: high -
ssl.truststore.password
The password for the trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled. Trust store password is not supported for PEM format.
Type: password Default: null Valid Values: Importance: high -
target.cluster.alias
Alias of target cluster. Used in metrics reporting.
Type: string Default: target Valid Values: Importance: high -
sasl.client.callback.handler.class
The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface.
Type: class Default: null Valid Values: Importance: medium -
sasl.jaas.config
JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is:
loginModuleClass controlFlag (optionName=optionValue)*;
. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;Type: password Default: null Valid Values: Importance: medium -
sasl.kerberos.service.name
The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config.
Type: string Default: null Valid Values: Importance: medium -
sasl.login.callback.handler.class
The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler
Type: class Default: null Valid Values: Importance: medium -
sasl.login.class
The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin
Type: class Default: null Valid Values: Importance: medium -
sasl.mechanism
SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.
Type: string Default: GSSAPI Valid Values: Importance: medium -
sasl.oauthbearer.jwks.endpoint.url
The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. If the URL is HTTP(S)-based, the JWKS data will be retrieved from the OAuth/OIDC provider via the configured URL on broker startup. All then-current keys will be cached on the broker for incoming requests. If an authentication request is received for a JWT that includes a "kid" header claim value that isn't yet in the cache, the JWKS endpoint will be queried again on demand. However, the broker polls the URL every sasl.oauthbearer.jwks.endpoint.refresh.ms milliseconds to refresh the cache with any forthcoming keys before any JWT requests that include them are received. If the URL is file-based, the broker will load the JWKS file from a configured location on startup. In the event that the JWT includes a "kid" header value that isn't in the JWKS file, the broker will reject the JWT and authentication will fail.
Type: string Default: null Valid Values: Importance: medium -
sasl.oauthbearer.token.endpoint.url
The URL for the OAuth/OIDC identity provider. If the URL is HTTP(S)-based, it is the issuer's token endpoint URL to which requests will be made to login based on the configuration in sasl.jaas.config. If the URL is file-based, it specifies a file containing an access token (in JWT serialized form) issued by the OAuth/OIDC identity provider to use for authorization.
Type: string Default: null Valid Values: Importance: medium -
security.protocol
Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.
Type: string Default: PLAINTEXT Valid Values: (case insensitive) [SASL_SSL, PLAINTEXT, SSL, SASL_PLAINTEXT] Importance: medium -
ssl.enabled.protocols
The list of protocols enabled for SSL connections. The default is 'TLSv1.2,TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). This default should be fine for most cases. Also see the config documentation for `ssl.protocol`.
Type: list Default: TLSv1.2 Valid Values: Importance: medium -
ssl.keystore.type
The file format of the key store file. This is optional for client. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].
Type: string Default: JKS Valid Values: Importance: medium -
ssl.protocol
The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'.
Type: string Default: TLSv1.2 Valid Values: Importance: medium -
ssl.provider
The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.
Type: string Default: null Valid Values: Importance: medium -
ssl.truststore.type
The file format of the trust store file. The values currently supported by the default `ssl.engine.factory.class` are [JKS, PKCS12, PEM].
Type: string Default: JKS Valid Values: Importance: medium -
admin.timeout.ms
Timeout for administrative tasks, e.g. detecting new topics.
Type: long Default: 60000 (1 minute) Valid Values: Importance: low -
auto.include.jmx.reporter
Deprecated. Whether to automatically include JmxReporter even if it's not listed in
metric.reporters
. This configuration will be removed in Kafka 4.0, users should instead includeorg.apache.kafka.common.metrics.JmxReporter
inmetric.reporters
in order to enable the JmxReporter.Type: boolean Default: true Valid Values: Importance: low -
enabled
Whether to replicate source->target.
Type: boolean Default: true Valid Values: Importance: low -
forwarding.admin.class
Class which extends ForwardingAdmin to define custom cluster resource management (topics, configs, etc). The class must have a constructor with signature
(Map
that is used to configure a KafkaAdminClient and may also be used to configure clients for external systems if necessary.config) Type: class Default: org.apache.kafka.clients.admin.ForwardingAdmin Valid Values: Importance: low -
metric.reporters
A list of classes to use as metrics reporters. Implementing the
org.apache.kafka.common.metrics.MetricsReporter
interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.Type: list Default: null Valid Values: Importance: low -
replication.policy.class
Class which defines the remote topic naming convention.
Type: class Default: org.apache.kafka.connect.mirror.DefaultReplicationPolicy Valid Values: Importance: low -
replication.policy.internal.topic.separator.enabled
Whether to use replication.policy.separator to control the names of topics used for checkpoints and offset syncs. By default, custom separators are used in these topic names; however, if upgrading MirrorMaker 2 from older versions that did not allow for these topic names to be customized, it may be necessary to set this property to 'false' in order to continue using the same names for those topics.
Type: boolean Default: true Valid Values: Importance: low -
replication.policy.separator
Separator used in remote topic naming convention.
Type: string Default: . Valid Values: Importance: low -
sasl.kerberos.kinit.cmd
Kerberos kinit command path.
Type: string Default: /usr/bin/kinit Valid Values: Importance: low -
sasl.kerberos.min.time.before.relogin
Login thread sleep time between refresh attempts.
Type: long Default: 60000 Valid Values: Importance: low -
sasl.kerberos.ticket.renew.jitter
Percentage of random jitter added to the renewal time.
Type: double Default: 0.05 Valid Values: Importance: low -
sasl.kerberos.ticket.renew.window.factor
Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket.
Type: double Default: 0.8 Valid Values: Importance: low -
sasl.login.connect.timeout.ms
The (optional) value in milliseconds for the external authentication provider connection timeout. Currently applies only to OAUTHBEARER.
Type: int Default: null Valid Values: Importance: low -
sasl.login.read.timeout.ms
The (optional) value in milliseconds for the external authentication provider read timeout. Currently applies only to OAUTHBEARER.
Type: int Default: null Valid Values: Importance: low -
sasl.login.refresh.buffer.seconds
The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.
Type: short Default: 300 Valid Values: [0,...,3600] Importance: low -
sasl.login.refresh.min.period.seconds
The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER.
Type: short Default: 60 Valid Values: [0,...,900] Importance: low -
sasl.login.refresh.window.factor
Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER.
Type: double Default: 0.8 Valid Values: [0.5,...,1.0] Importance: low -
sasl.login.refresh.window.jitter
The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.
Type: double Default: 0.05 Valid Values: [0.0,...,0.25] Importance: low -
sasl.login.retry.backoff.max.ms
The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.
Type: long Default: 10000 (10 seconds) Valid Values: Importance: low -
sasl.login.retry.backoff.ms
The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Currently applies only to OAUTHBEARER.
Type: long Default: 100 Valid Values: Importance: low -
sasl.oauthbearer.clock.skew.seconds
The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker.
Type: int Default: 30 Valid Values: Importance: low -
sasl.oauthbearer.expected.audience
The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWT's "aud" claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.
Type: list Default: null Valid Values: Importance: low -
sasl.oauthbearer.expected.issuer
The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth "iss" claim and if this value is set, the broker will match it exactly against what is in the JWT's "iss" claim. If there is no match, the broker will reject the JWT and authentication will fail.
Type: string Default: null Valid Values: Importance: low -
sasl.oauthbearer.header.urlencode
The (optional) setting to enable the OAuth client to URL-encode the client_id and client_secret in the authorization header in accordance with RFC6749, see here for more details. The default value is set to 'false' for backward compatibility
Type: boolean Default: false Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.refresh.ms
The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.
Type: long Default: 3600000 (1 hour) Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms
The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.
Type: long Default: 10000 (10 seconds) Valid Values: Importance: low -
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms
The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting.
Type: long Default: 100 Valid Values: Importance: low -
sasl.oauthbearer.scope.claim.name
The OAuth claim for the scope is often named "scope", but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
Type: string Default: scope Valid Values: Importance: low -
sasl.oauthbearer.sub.claim.name
The OAuth claim for the subject is often named "sub", but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
Type: string Default: sub Valid Values: Importance: low -
ssl.cipher.suites
A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.
Type: list Default: null Valid Values: Importance: low -
ssl.endpoint.identification.algorithm
The endpoint identification algorithm to validate server hostname using server certificate.
Type: string Default: https Valid Values: Importance: low -
ssl.engine.factory.class
The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is org.apache.kafka.common.security.ssl.DefaultSslEngineFactory. Alternatively, setting this to org.apache.kafka.common.security.ssl.CommonNameLoggingSslEngineFactory will log the common name of expired SSL certificates used by clients to authenticate at any of the brokers with log level INFO. Note that this will cause a tiny delay during establishment of new connections from mTLS clients to brokers due to the extra code for examining the certificate chain provided by the client. Note further that the implementation uses a custom truststore based on the standard Java truststore and thus might be considered a security risk due to not being as mature as the standard one.
Type: class Default: null Valid Values: Importance: low -
ssl.keymanager.algorithm
The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.
Type: string Default: SunX509 Valid Values: Importance: low -
ssl.secure.random.implementation
The SecureRandom PRNG implementation to use for SSL cryptography operations.
Type: string Default: null Valid Values: Importance: low -
ssl.trustmanager.algorithm
The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.
Type: string Default: PKIX Valid Values: Importance: low -
name
Globally unique name to use for this connector.
Type: string Default: Valid Values: non-empty string without ISO control characters Importance: high -
connector.class
Name or alias of the class for this connector. Must be a subclass of org.apache.kafka.connect.connector.Connector. If the connector is org.apache.kafka.connect.file.FileStreamSinkConnector, you can either specify this full name, or use "FileStreamSink" or "FileStreamSinkConnector" to make the configuration a bit shorter
Type: string Default: Valid Values: Importance: high -
tasks.max
Maximum number of tasks to use for this connector.
Type: int Default: 1 Valid Values: [1,...] Importance: high -
tasks.max.enforce
(Deprecated) Whether to enforce that the tasks.max property is respected by the connector. By default, connectors that generate too many tasks will fail, and existing sets of tasks that exceed the tasks.max property will also be failed. If this property is set to false, then connectors will be allowed to generate more than the maximum number of tasks, and existing sets of tasks that exceed the tasks.max property will be allowed to run. This property is deprecated and will be removed in an upcoming major release.
Type: boolean Default: true Valid Values: Importance: low -
key.converter
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.
Type: class Default: null Valid Values: A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor Importance: low -
value.converter
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.
Type: class Default: null Valid Values: A concrete subclass of org.apache.kafka.connect.storage.Converter, A class with a public, no-argument constructor Importance: low -
header.converter
HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas.
Type: class Default: null Valid Values: A concrete subclass of org.apache.kafka.connect.storage.HeaderConverter, A class with a public, no-argument constructor Importance: low -
config.action.reload
The action that Connect should take on the connector when changes in external configuration providers result in a change in the connector's configuration properties. A value of 'none' indicates that Connect will do nothing. A value of 'restart' indicates that Connect should restart/reload the connector with the updated configuration properties.The restart may actually be scheduled in the future if the external configuration provider indicates that a configuration value will expire in the future.
Type: string Default: restart Valid Values: [none, restart] Importance: low -
transforms
Aliases for the transformations to be applied to records.
Type: list Default: "" Valid Values: non-null string, unique transformation aliases Importance: low -
predicates
Aliases for the predicates used by transformations.
Type: list Default: "" Valid Values: non-null string, unique predicate aliases Importance: low -
errors.retry.timeout
The maximum duration in milliseconds that a failed operation will be reattempted. The default is 0, which means no retries will be attempted. Use -1 for infinite retries.
Type: long Default: 0 Valid Values: Importance: medium -
errors.retry.delay.max.ms
The maximum duration in milliseconds between consecutive retry attempts. Jitter will be added to the delay once this limit is reached to prevent thundering herd issues.
Type: long Default: 60000 (1 minute) Valid Values: Importance: medium -
errors.tolerance
Behavior for tolerating errors during connector operation. 'none' is the default value and signals that any error will result in an immediate connector task failure; 'all' changes the behavior to skip over problematic records.
Type: string Default: none Valid Values: [none, all] Importance: medium -
errors.log.enable
If true, write each error and the details of the failed operation and problematic record to the Connect application log. This is 'false' by default, so that only errors that are not tolerated are reported.
Type: boolean Default: false Valid Values: Importance: medium -
errors.log.include.messages
Whether to include in the log the Connect record that resulted in a failure. For sink records, the topic, partition, offset, and timestamp will be logged. For source records, the key and value (and their schemas), all headers, and the timestamp, Kafka topic, Kafka partition, source partition, and source offset will be logged. This is 'false' by default, which will prevent record keys, values, and headers from being written to log files.
Type: boolean Default: false Valid Values: Importance: medium
3.8.2 MirrorMaker Source Configs
Below is the configuration of MirrorMaker 2 source connector for replicating topics.-
config.properties.blacklist
Deprecated. Use config.properties.exclude instead.
Type: list Default: null Valid Values: Importance: high -
config.properties.exclude
Topic config properties that should not be replicated. Supports comma-separated property names and regexes.
Type: list Default: follower\.replication\.throttled\.replicas,leader\.replication\.throttled\.replicas,message\.timestamp\.difference\.max\.ms,message\.timestamp\.type,unclean\.leader\.election\.enable,min\.insync\.replicas Valid Values: Importance: high -
topics
Topics to replicate. Supports comma-separated topic names and regexes.
Type: list Default: .* Valid Values: Importance: high -
topics.blacklist
Deprecated. Use topics.exclude instead.
Type: list Default: null Valid Values: Importance: high -
topics.exclude
Excluded topics. Supports comma-separated topic names and regexes. Excludes take precedence over includes.
Type: list Default: .*[\-\.]internal,.*\.replica,__.* Valid Values: Importance: high -
add.source.alias.to.metrics
Deprecated. Whether to tag metrics with the source cluster alias. Metrics have the target, topic and partition tags. When this setting is enabled, it adds the source tag. This configuration will be removed in Kafka 4.0 and the default behavior will be to always have the source tag.
Type: boolean Default: false Valid Values: Importance: low -
config.property.filter.class
ConfigPropertyFilter to use. Selects topic config properties to replicate.
Type: class Default: org.apache.kafka.connect.mirror.DefaultConfigPropertyFilter Valid Values: Importance: low -
consumer.poll.timeout.ms
Timeout when polling source cluster.
Type: long Default: 1000 (1 second) Valid Values: Importance: low -
emit.offset-syncs.enabled
Whether to store the new offset of the replicated records in offset-syncs topic or not. MirrorCheckpointConnector will not be able to sync group offsets or emit checkpoints if emit.checkpoints.enabled and/or sync.group.offsets.enabled are enabled while emit.offset-syncs.enabled is disabled.
Type: boolean Default: true Valid Values: Importance: low -
offset-syncs.topic.location
The location (source/target) of the offset-syncs topic.
Type: string Default: source Valid Values: [source, target] Importance: low -
offset-syncs.topic.replication.factor
Replication factor for offset-syncs topic.
Type: short Default: 3 Valid Values: Importance: low -
offset.lag.max
How out-of-sync a remote partition can be before it is resynced.
Type: long Default: 100 Valid Values: Importance: low -
refresh.topics.enabled
Whether to periodically check for new topics and partitions.
Type: boolean Default: true Valid Values: Importance: low -
refresh.topics.interval.seconds
Frequency of topic refresh.
Type: long Default: 600 Valid Values: Importance: