Apache Kafka for Database Engineers: A Practical Introduction

You already understand transactions, durability, replication, and sequential I/O — the same concepts that make a database reliable are exactly what Apache Kafka is built on. The terminology is different, but the engineering tradeoffs are ones you have spent years reasoning about. Most "Kafka for beginners" content treats you like you have never seen a WAL or a replication lag metric, which wastes your time. This guide speaks database first: every Kafka concept is anchored to something you already know.

Kafka from a Database Perspective

A relational database is optimized for random reads and writes against a mutable state. Kafka is optimized for sequential writes and sequential reads against an immutable, ordered log. The fundamental data structure is not a B-tree — it is an append-only log, the same structure that underlies every database WAL you have ever diagnosed.

Where a database answers "what is the current state of row X?", Kafka answers "what events happened, in what order, and what is the offset of the last one a given reader processed?" That inversion of perspective is the main conceptual shift. Everything else — durability, replication, consumer isolation — follows patterns you already recognize.

Core Concepts Mapped to Database Concepts

Topics are Tables

A topic is the logical namespace for a stream of related events, exactly as a table is the logical namespace for related rows. You publish events to a topic; consumers read from a topic. Unlike a table, you do not UPDATE or DELETE rows in a topic — you append new events. State changes are expressed as new records, not mutations (unless you use log compaction, covered below).

Partitions are Shards

Every topic is divided into one or more partitions. Think of them exactly like horizontal shards in a distributed database: each partition is an independent, ordered log stored on a specific broker. Kafka guarantees ordering within a partition, not across partitions — the same guarantee you get with shard-local ordering in a sharded MySQL or Vitess cluster.

The partition key (set on the producer via the message key) determines which partition a record lands in — this is consistent hashing, identical to how a sharding key routes writes to a shard. Choosing a bad partition key causes hot partitions, exactly as a bad sharding key causes hot shards.

Offsets are Sequence Numbers

Within a partition, every record has a monotonically increasing integer called an offset. This is the Kafka equivalent of a database sequence number, a MySQL binlog position, or a PostgreSQL LSN (Log Sequence Number). Consumers track their position in a partition by storing the last committed offset — the equivalent of a replica tracking its binlog position.

Offsets are immutable and never reused within a partition. A consumer can seek backward to any offset and replay events, something impossible with a traditional database queue but trivially possible because Kafka is a log, not a queue that deletes consumed records.

Consumer Groups are Read Replicas with Independent Read Positions

A consumer group is a set of consumers that collectively read a topic. Each partition is assigned to exactly one consumer within the group at a time — think of the group as a single logical reader, spread across workers for parallelism. Two different consumer groups each get their own independent offset tracking: they read the same data without interfering with each other, exactly like two read replicas that each maintain their own replication position against the same binlog stream.

This means you can add a new downstream system (a new consumer group) without touching producers or any existing consumers. The topic is the source of truth; consumers just advance their pointer through it.

Durability Guarantees

Kafka's durability model maps almost perfectly onto MySQL's semi-synchronous replication settings. The key producer configuration is acks:

• acks=0 — fire and forget. No acknowledgment. Equivalent to writing to a buffer and hoping for the best.
• acks=1 — the leader broker acknowledges after writing to its local log. The record can be lost if the leader fails before followers replicate. Equivalent to async replication.
• acks=all (or acks=-1) — the leader waits for all in-sync replicas (ISR) to acknowledge. This is the Kafka equivalent of sync_binlog=1 combined with semi-synchronous replication. Use this for any data you cannot afford to lose.

Replication Factor

Set replication.factor=3 for production topics. With three replicas and min.insync.replicas=2, you can lose one broker without data loss or write unavailability — directly analogous to a three-node MySQL Group Replication cluster tolerating one failure.

Retention vs. WAL

Kafka retains records for a configurable period via retention.ms (default: 7 days) or by size via retention.bytes. Unlike a database WAL — which is purged as soon as the data is checkpointed and no replica needs it — Kafka retention is intentionally longer, because consumers are expected to replay history. Treat retention.ms as the window during which a lagging consumer can still catch up without data loss.

Log Compaction: UPSERT Semantics on an Append-Only Log

With cleanup.policy=compact, Kafka retains only the most recent record for each message key within a topic. Older records with the same key are deleted during compaction. This gives you UPSERT semantics: the compacted topic represents the latest state of each key, while the log itself remains append-only. Compacted topics are commonly used for configuration data, user profile snapshots, or any dataset where "latest value per key" is the useful view.

A null-value record (a tombstone) triggers deletion of a key during compaction — the equivalent of a DELETE statement.

Exactly-Once Delivery

Kafka's default delivery guarantee is at-least-once: if a producer retries a failed send, the broker may write the record twice. This is the same problem as a network timeout on a database INSERT — you do not know if the write succeeded, so you retry, and now you have a duplicate.

Kafka solves this with two mechanisms that together provide exactly-once semantics:

Idempotent Producers

Setting enable.idempotence=true on the producer assigns a unique Producer ID (PID) and per-partition sequence number to every batch. The broker deduplicates retries using these identifiers — exactly like a database using a client-supplied idempotency key to deduplicate retried inserts. Enable this for any producer where duplicate records would cause downstream problems.

# Producer configuration for idempotent, durable delivery
enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5
retries=2147483647

Kafka Transactions

For read-process-write pipelines (consume from topic A, transform, write to topic B, commit offset — atomically), Kafka supports transactions via a transactional.id on the producer. The transaction coordinator ensures the write to topic B and the offset commit to topic A happen atomically. Consumers configured with isolation.level=read_committed will not see records from uncommitted transactions. This is the Kafka equivalent of a two-phase commit across the write and the offset store.

Operations: kafka-topics.sh for DBAs

The kafka-topics.sh utility is your SHOW and ALTER TABLE equivalent for Kafka. Here are the commands most relevant for a DBA approaching Kafka for the first time.

List Topics (SHOW TABLES)

kafka-topics.sh --bootstrap-server broker:9092 --list

Describe a Topic (SHOW CREATE TABLE / SHOW TABLE STATUS)

kafka-topics.sh --bootstrap-server broker:9092 \
  --describe --topic orders

# Output shows: partition count, replication factor,
# leader broker per partition, ISR list, and config overrides

Create a Topic with Durability Settings

kafka-topics.sh --bootstrap-server broker:9092 \
  --create --topic orders \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000 \
  --config cleanup.policy=delete

Check Consumer Group Lag (Replication Lag Equivalent)

kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group payments-service

# LOG-END-OFFSET - CURRENT-OFFSET = LAG per partition
# Sum of LAG across partitions = total consumer lag
# Monitor this like you monitor replication lag

Reset Consumer Group Offset (Point-in-Time Recovery)

# Reset to earliest available offset (full replay)
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --group payments-service \
  --topic orders \
  --reset-offsets --to-earliest --execute

# Reset to a specific timestamp (equivalent to PITR)
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --group payments-service \
  --topic orders \
  --reset-offsets --to-datetime 2026-02-20T00:00:00.000 --execute

KRaft Mode: No More ZooKeeper

Since Kafka 3.3 (and mandatory in Kafka 4.0), Kafka uses KRaft mode — the Kafka Raft Metadata protocol — for cluster metadata and leader election, replacing the ZooKeeper dependency entirely. From an operational standpoint, this eliminates a separate distributed coordination cluster you had to manage, monitor, and keep version-compatible with Kafka. If you are deploying a new Kafka cluster in 2026, start with KRaft. There is no operational reason to use ZooKeeper-mode Kafka for new deployments.

KRaft moves the cluster metadata (topic configurations, partition assignments, controller election) into an internal Kafka topic (__cluster_metadata) replicated across a set of controller nodes. The failure model is the same Raft quorum you see in etcd or MySQL Group Replication: a majority of controllers must be available to elect a leader and accept metadata writes.

Key Takeaways

Go Deeper with JusDB

JusDB publishes practical, engineering-first guides for database professionals working across the modern data stack — from PostgreSQL internals and MySQL replication to distributed systems like Kafka and ClickHouse. If this walkthrough was useful, explore the rest of the JusDB blog for similarly grounded content: no fluff, no oversimplification, written for engineers who already know what a WAL is.

Have a specific Kafka operational question, a production incident you are diagnosing, or a data architecture decision you are working through? Reach out to the JusDB team — we work with engineering teams on exactly these kinds of problems.