Kafka 4.0 Without ZooKeeper: KRaft Migration for CDC Pipelines

Apache Kafka 4.0 runs only in KRaft mode. If your brokers still have zookeeper.connect in server.properties, there is no 4.x release you can roll onto them. The upgrade notes say it plainly: clusters in ZooKeeper mode "have to be migrated to KRaft mode" before they can upgrade. For teams running change data capture (CDC), that is two projects. First comes a metadata migration on the Kafka 3.9 bridge release. Then comes a 4.x upgrade that needs Java 17 on Kafka Connect workers, drops client protocol versions older than Kafka 2.1, and removes MirrorMaker 1. This guide covers both for a cluster that carries Debezium or other Kafka Connect pipelines. It explains how to size the controller quorum, how the migration phases run, where each rollback point is, and what to check on the Connect side.

Scope and review date: November 2026. This post covers self-managed Apache Kafka clusters in ZooKeeper mode moving to Kafka 4.x through the 3.9 bridge release, with Kafka Connect and Debezium 3.x pipelines attached. Managed services and Kubernetes operators have their own migration procedures, so follow your provider's documentation there. Configs, commands, and version numbers were checked against the Apache Kafka 3.9 and 4.4 documentation and the Debezium release notes on 9 November 2026. They were not run in a lab, so rehearse them on a staging cluster.

What did Kafka 4.0 remove, and which 4.x release is current?

The Kafka upgrade notes open the 4.0 changes with this: "Apache Kafka 4.0 only supports KRaft mode - ZooKeeper mode has been removed." Metadata that lived in ZooKeeper znodes now lives in the __cluster_metadata log, replicated by a Raft quorum of controllers.

The Kafka downloads page lists 4.4.0 (22 October 2026), 4.3.2 (23 October 2026), and 4.2.2 (29 September 2026) as the supported releases. Every 3.x release, including 3.9.2 (21 February 2026), is archived. Treat 3.9 as a place to migrate, not a place to stay.

# On each broker: which mode is this cluster in?
grep -E '^(process\.roles|zookeeper\.connect|broker\.id|node\.id|inter\.broker\.protocol\.version)=' \
  /etc/kafka/server.properties

Interpretation: a zookeeper.connect line with no process.roles means ZooKeeper mode, and you need the full migration below. A process.roles=broker line means the cluster is already KRaft, and you can skip to the 4.x upgrade section. KRaft clusters older than 3.3 should go to 3.9.x first, because 4.x accepts upgrades only from 3.3.x onwards.

What is the upgrade path from a ZooKeeper cluster to Kafka 4.x?

Three stages, each with its own rolling restarts. The 4.4 KRaft operations page says: "The last bridge release is Kafka 3.9." The 3.9 KRaft documentation, updated in October 2026, says brokers "must be upgraded to software version 3.9.1" and run with inter.broker.protocol.version set to 3.9 before the migration starts.

1. Bridge. Upgrade the ZooKeeper-mode cluster to 3.9.x and raise the inter-broker protocol to 3.9. Use 3.9.2: it adds the KIP-1252 compatibility flag for an AlterConfigPolicy inconsistency between the two modes.
2. Migrate. Move metadata from ZooKeeper to a KRaft controller quorum, still on 3.9.
3. Upgrade. Roll the KRaft cluster from 3.9 to 4.x and finalize the new metadata version. The 4.4 notes accept servers "from any version 3.3.x through 4.3.0", so 3.9 can go straight to 4.4.

# Record the cluster ID and the IBP before you start
bin/zookeeper-shell.sh zk-1:2181 get /cluster/id
grep '^inter.broker.protocol.version=' /etc/kafka/server.properties

Interpretation: the cluster ID returned here must be reused when you format the controllers. If the IBP is not 3.9 on every broker, finish that rolling restart first. The 3.9 docs warn that changing the metadata version during the migration "may break the cluster".

How should you size and place the KRaft controller quorum?

A KRaft controller quorum needs a majority alive to accept metadata changes. The 3.9 and 4.4 docs give the same rules. Run 3 or 5 controllers: 3 tolerate one failure and 5 tolerate two, because tolerating N failures takes 2N + 1 controllers. Combined broker and controller nodes are "not recommended in critical deployment environments". For a typical cluster, the docs estimate that 5 GB of memory and 5 GB of disk for the metadata log directory is sufficient.

Place each controller in a different failure domain, such as one per availability zone, so the quorum survives a zone outage. Controller node.id values share one namespace with broker IDs, so pick a range no broker uses (the docs use 3000 and up).

# controller.properties for a 3-controller static quorum in migration mode (Kafka 3.9)
process.roles=controller
node.id=3000
controller.quorum.voters=3000@ctl-0:9093,3001@ctl-1:9093,3002@ctl-2:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093
# Enable the migration
zookeeper.metadata.migration.enable=true
zookeeper.connect=zk-1:2181,zk-2:2181,zk-3:2181
# Lets the KRaft controller send RPCs to ZooKeeper-mode brokers
inter.broker.listener.name=PLAINTEXT

Interpretation: the 3.9 sample uses a single standalone controller with controller.quorum.bootstrap.servers, a dynamic quorum. A static quorum lists every voter in controller.quorum.voters. Static cannot become dynamic on 3.9, but the 4.4 docs show the switch from release version 4.1 by upgrading kraft.version to 1.

How do you run the ZooKeeper to KRaft migration on Kafka 3.9?

The 3.9 docs name the phases: initial (all brokers in ZooKeeper mode), initial metadata load, hybrid (some brokers still in ZooKeeper mode under a KRaft controller), dual-write (all brokers in KRaft mode, the controller still writes to ZooKeeper), and finalized. Each move below is a rolling restart, so partitions stay available as long as replication factors and min.insync.replicas allow one broker down.

Broker settings for migration mode

Adapted from the 3.9 sample. broker.id must be an explicit non-negative integer even when broker.id.generation.enable is on.

# server.properties, ZooKeeper-mode broker entering migration mode (Kafka 3.9)
broker.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker-0:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
inter.broker.protocol.version=3.9
zookeeper.metadata.migration.enable=true
zookeeper.connect=zk-1:2181,zk-2:2181,zk-3:2181
controller.quorum.voters=3000@ctl-0:9093,3001@ctl-1:9093,3002@ctl-2:9093
controller.listener.names=CONTROLLER

Interpretation: use the same quorum setting as the controllers, and add the controller listener to listener.security.protocol.map. With TLS or SASL, that entry must match the controllers' listener.

Broker settings for step 5

# server.properties, same broker restarted in KRaft mode
process.roles=broker
node.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker-0:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
controller.quorum.voters=3000@ctl-0:9093,3001@ctl-1:9093,3002@ctl-2:9093
controller.listener.names=CONTROLLER
# Removed: inter.broker.protocol.version, zookeeper.metadata.migration.enable,
# zookeeper.connect, control.plane.listener.name
# If you used AclAuthorizer:
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer

Interpretation: node.id must equal the old broker.id; the docs say the broker must keep the same ID. KRaft uses the metadata.version feature flag instead of an IBP.

Where are the rollback points, and how do you revert?

The 3.9 docs include a revert table keyed on the last step you fully completed. Before brokers enter migration mode, you only deprovision the controllers. After that, reverting needs ZooKeeper edits and one or two broker rolls. Once the migration is finalized, the docs say "you cannot revert".

# Revert from step 3 or later, right after deprovisioning the controllers
bin/zookeeper-shell.sh zk-1:2181 get /controller_epoch
bin/zookeeper-shell.sh zk-1:2181 get /migration   # note kraft_controller_epoch
# Only if the KRaft epoch is higher: pick a value above it
bin/zookeeper-shell.sh zk-1:2181 set /controller_epoch 42
bin/zookeeper-shell.sh zk-1:2181 delete /controller
bin/zookeeper-shell.sh zk-1:2181 delete /migration

Interpretation: the epoch check is new in the October 2026 revision of the docs (KAFKA-20488). Partition state znodes carry the KRaft controller epoch during the migration. A ZooKeeper controller elected with a lower epoch fails partition state changes, because it assumes another controller with a higher epoch exists. The value 42 is a placeholder. Run these commands quickly: the cluster has no controller until /controller is deleted.

What breaks when you move from KRaft 3.9 to Kafka 4.x?

On a finalized KRaft 3.9 cluster, the 4.x upgrade is a rolling binary swap plus kafka-features.sh. What breaks is mostly what connects to the cluster:

• Java. Clients and Kafka Streams need Java 11 or later. Brokers, Connect, and tools need Java 17. The 4.4 Java page lists Java 17, 21, and 25 as fully supported and recommends the newest LTS.
• Old protocol versions. Brokers must be 2.1 or later before Java clients go to 4.0, and Java clients (including Connect and Streams) must be 2.1 or later before brokers go to 4.0. Non-Java clients need their own check against KIP-896.
• Removed configs and tools. log.message.format.version, message.format.version, offsets.commit.required.acks, and delegation.token.master.key are gone, and so is MirrorMaker 1.
• Logging. Log4j moved to Log4j2. The notes point to log4j-transform-cli to convert existing Log4j files.
• Group and transaction protocols. The new consumer rebalance protocol (KIP-848) and the transactions server-side defense (KIP-890) turn on when the 4.0 upgrade is finalized. Once consumer groups use the new protocol, the cluster can only be downgraded to 3.4.1 or newer.

# On 3.9, before the 4.x roll: are old protocol versions still in use?
bin/kafka-run-class.sh org.apache.kafka.tools.JmxTool \
  --jmx-url service:jmx:rmi:///jndi/rmi://broker-0:9999/jmxrmi \
  --object-name 'kafka.network:type=RequestMetrics,name=DeprecatedRequestsPerSec,*' \
  --one-time true

Interpretation: KIP-896 added DeprecatedRequestsPerSec in 3.7, tagged by request, API version, and client software name and version. KIP-896 says that if it is 0, "the upgrade is safe." A non-zero value names the client library to upgrade. Check every broker over a full business cycle, because batch jobs and old CDC consumers may connect only once a day.

# After every node runs 4.x and the cluster looks healthy
bin/kafka-features.sh --bootstrap-server broker-0:9092 describe
bin/kafka-features.sh --bootstrap-server broker-0:9092 upgrade --release-version 4.4

Interpretation: describe shows the finalized metadata.version and kraft.version. The upgrade raises them. The 4.0 notes say metadata downgrade is not supported because 4.0 carries metadata changes, so this is the second one-way step in the project.

What changes for Debezium and Kafka Connect pipelines?

Kafka Connect workers are Kafka clients. They reach the cluster through bootstrap.servers and never through ZooKeeper, so the migration itself needs no Connect config change. Connect keeps connector offsets, configs, and status in its three internal topics. Debezium keeps its database schema history in a topic as well. The migration moves cluster metadata only, so those topics and their data stay where they are.

What the pipeline does feel is the broker rolls. Each restart moves partition leaders, and Connect's clients retry. Source connectors deliver at least once unless exactly.once.source.support is enabled, so a task restart between offset flushes can repeat a few change events. Keep sinks idempotent, for example with upserts keyed on the primary key.

# Before each roll: record Debezium's committed source offsets (Connect 3.5+)
curl -s http://connect-0:8083/connectors/inventory-cdc/offsets | jq .
curl -s http://connect-0:8083/connectors/inventory-cdc/status | jq '.tasks[].state'

Interpretation: the offsets response holds the binlog position, LSN, or resume token per source partition. If a connector comes back after a roll at an older position than the one recorded, you will see re-delivery. The Connect user guide documents PATCH and DELETE on the same endpoint, which only work while the connector is stopped with PUT /connectors/{name}/stop.

Moving the Connect workers to 4.x

You can upgrade brokers first and keep Connect workers on a 3.x release, since 3.x clients are newer than 2.1. When the workers move to 4.x, check these:

• Debezium version. The Debezium 3.7 release notes say 3.7.1.Final (6 November 2026) "has been built against Kafka Connect 4.3.1 and has been tested with version 4.3.1 of the Kafka brokers". The Debezium releases page lists Java 17+ for connectors and Kafka Connect 3.1 and later. Debezium 2.7 lists Java 11 and Connect 2.x and 3.x, so 4.x workers are outside its tested range.
• Removed SMT options. whitelist and blacklist were removed from ReplaceField; use include and exclude.
• Removed task APIs. SinkTask.onPartitionsRevoked(Collection), onPartitionsAssigned(Collection), and SourceTask.commitRecord(SourceRecord) are gone. Custom or older third-party connectors that override them need a newer build.
• Producer defaults. The default linger.ms changed from 0 to 5 in 4.0. If end-to-end latency matters, set producer.linger.ms in the worker config and measure.
• MirrorMaker. MirrorMaker 1 is removed, so CDC topics copied to a second cluster need MirrorMaker 2. Its topics.blacklist, groups.blacklist, and config.properties.blacklist options become *.exclude.

# Find connector configs that use removed ReplaceField or MirrorMaker 2 options
for c in $(curl -s http://connect-0:8083/connectors | jq -r '.[]'); do
  curl -s "http://connect-0:8083/connectors/$c/config" \
    | jq -r --arg c "$c" 'to_entries[]
        | select(.key | test("whitelist|blacklist"))
        | "\($c): \(.key)"'
done

Interpretation: every hit is a config to edit before it runs on a 4.x worker, because the removed key no longer applies its filter. MirrorMaker 2 running in dedicated mode reads mm2.properties instead, so grep that file too.

What should you monitor during and after the migration?

ZooKeeper-mode dashboards go quiet after the migration. The KRaft vs ZooKeeper page lists the controller metrics that exist only in ZooKeeper mode, such as ControllerChannelManager queue metrics and ControllerStats rate metrics. During the migration, the 3.9 monitoring docs give you a state gauge to track.

# Quorum health from the command line (3.9 and 4.x)
bin/kafka-metadata-quorum.sh --bootstrap-server broker-0:9092 describe --status
bin/kafka-metadata-quorum.sh --bootstrap-server broker-0:9092 describe --replication

Interpretation: --status summarizes the quorum and --replication lists each voter's and observer's log end offset and lag. Run both before and after every controller restart.

On the Connect side, watch task state and Debezium's MilliSecondsBehindSource metric. A lag spike during a broker roll is expected; a task stuck in FAILED afterwards is not. For Connect metrics, see Kafka Connect: JDBC source, SMTs, dead letter queues, and connector monitoring.

Frequently asked questions

Key takeaways

Plan your Kafka KRaft migration with JusDB

JusDB's Debezium CDC team can review your ZooKeeper-mode cluster and the Connect pipelines on it. We can plan the controller quorum and write a migration runbook with the revert steps for each phase. We can also check your connectors, SMTs, and clients against the 4.x changes before the upgrade.

For the wider cutover, our database migration service can help plan and run the change windows alongside your team. Bring your broker and Connect versions, the connector inventory, and your maintenance windows. Your team approves every production change.

Related reading

• Debezium advanced: handling schema changes, restarts, and offsets
• Debezium 3.0: real-time CDC from PostgreSQL, MySQL, and MongoDB
• Kafka Connect: building database integration pipelines
• Apache Kafka 3.9 documentation: ZooKeeper to KRaft migration