Transactional Outbox Pattern: Reliable Event Publishing from Databases

You've seen this bug before: the order is saved to PostgreSQL, but the order.created event never reaches Kafka. The payment service waits. The warehouse never picks. The customer emails support. This class of failure — silent, intermittent, and devastating in production — has a name: the dual-write problem. It occurs whenever your application tries to atomically commit to two independent systems — a database and a message broker — without a distributed transaction binding them. The transactional outbox pattern eliminates this failure mode by reducing every event publish to a plain database write inside your existing transaction.

The Dual-Write Problem

Consider a checkout service. When an order is placed, it must persist the order row and publish an order.created event so downstream services can react. The naive implementation looks like this:

def place_order(order):
    db.execute("INSERT INTO orders (...) VALUES (...)", order)
    kafka.produce("order.created", order.to_json())   # danger zone

There are at least three failure windows here:

• The database write succeeds, but the process crashes before kafka.produce is called. The event is lost.
• kafka.produce is called, but the broker is temporarily unavailable. The event is lost unless you add retry logic — and even then, retry logic can fail.
• The Kafka write succeeds, but the subsequent db.commit rolls back (e.g., a constraint violation). The event has been published for data that does not exist.

The only robust solution is to make "publishing an event" synonymous with "writing a row to your database."

The Outbox Pattern Solution

The outbox pattern works by introducing a dedicated outbox table in your existing relational database. Instead of calling the broker directly from application code, you insert an event record into this table inside the same database transaction as your business data write. A separate, independent process — the relay — reads unprocessed outbox rows and forwards them to the broker. Once successfully forwarded, the row is marked as processed.

Because the outbox insert shares a transaction with the business write, you get exactly the atomicity you need from the database engine. Either both the order row and the outbox row commit, or neither does. The relay is now decoupled: it can fail, restart, and retry without ever producing a phantom event for a rolled-back transaction.

Implementation

Outbox Table Schema

Create the outbox table alongside your business tables. Keep it in the same database so the same transaction context applies.

-- PostgreSQL
CREATE TABLE outbox (
    id             UUID        PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type VARCHAR(128) NOT NULL,          -- e.g. 'Order', 'Payment'
    aggregate_id   VARCHAR(255) NOT NULL,          -- e.g. the order UUID
    event_type     VARCHAR(128) NOT NULL,          -- e.g. 'order.created'
    payload        JSONB       NOT NULL,           -- full event payload
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    processed_at   TIMESTAMPTZ NULL               -- NULL = unprocessed
);

CREATE INDEX idx_outbox_unprocessed
    ON outbox (created_at)
    WHERE processed_at IS NULL;

-- MySQL equivalent
CREATE TABLE outbox (
    id             CHAR(36)      NOT NULL PRIMARY KEY,
    aggregate_type VARCHAR(128)  NOT NULL,
    aggregate_id   VARCHAR(255)  NOT NULL,
    event_type     VARCHAR(128)  NOT NULL,
    payload        JSON          NOT NULL,
    created_at     DATETIME(6)   NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
    processed_at   DATETIME(6)   NULL,
    INDEX idx_outbox_unprocessed (created_at, processed_at)
);

Transactional Insert

The critical invariant: the outbox insert must happen inside the same transaction as the business write. In application code this looks like:

BEGIN;

INSERT INTO orders (id, customer_id, total, status, created_at)
VALUES (
    'a1b2c3d4-e5f6-7890-abcd-ef1234567890',
    42,
    9999,
    'pending',
    now()
);

INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload)
VALUES (
    gen_random_uuid(),
    'Order',
    'a1b2c3d4-e5f6-7890-abcd-ef1234567890',
    'order.created',
    '{
        "order_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
        "customer_id": 42,
        "total": 9999,
        "status": "pending",
        "occurred_at": "2026-02-22T10:00:00Z"
    }'
);

COMMIT;

If the transaction rolls back for any reason — constraint violation, deadlock, application crash — both rows are discarded. No orphaned events. No missing events.

Debezium for Outbox: CDC-Based Relay

Debezium is a Change Data Capture (CDC) platform that tails the database write-ahead log (WAL for PostgreSQL, binlog for MySQL) and streams row-level change events to Kafka. Combined with Debezium's built-in Outbox Event Router Single Message Transform (SMT), this is the most operationally efficient relay strategy.

Instead of polling the outbox table with repeated SELECT queries, Debezium reads inserts directly from the database log. The Outbox Event Router SMT intercepts each INSERT on the outbox table and re-routes it to a Kafka topic derived from aggregate_type, using aggregate_id as the message key. The raw outbox table change event is replaced by a clean domain event on the wire.

{
  "name": "outbox-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "db.example.com",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "secret",
    "database.dbname": "appdb",
    "table.include.list": "public.outbox",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.table.field.event.id": "id",
    "transforms.outbox.table.field.event.key": "aggregate_id",
    "transforms.outbox.table.field.event.payload": "payload",
    "transforms.outbox.route.by.field": "aggregate_type",
    "transforms.outbox.route.topic.replacement": "outbox.${routedByValue}.events"
  }
}

With this configuration, an Order outbox row automatically publishes to the Kafka topic outbox.Order.events, keyed by the order UUID. Ordering within a partition is preserved for all events belonging to the same aggregate because Kafka partitions by message key.

Polling Publisher Alternative

If CDC infrastructure is not available or practical, a polling publisher is a straightforward alternative. A background worker runs on a fixed interval, selects unprocessed outbox rows, publishes them to the broker, and marks them processed.

-- Step 1: Claim a batch of unprocessed rows (PostgreSQL)
-- Use SKIP LOCKED to allow multiple relay instances without contention
SELECT id, aggregate_type, aggregate_id, event_type, payload
FROM outbox
WHERE processed_at IS NULL
ORDER BY created_at ASC
LIMIT 100
FOR UPDATE SKIP LOCKED;

-- Step 2: After successfully publishing each event to the broker,
-- mark it processed
UPDATE outbox
SET processed_at = now()
WHERE id = 'a1b2c3d4-e5f6-7890-abcd-ef1234567890';

The FOR UPDATE SKIP LOCKED clause is essential when running multiple relay instances for high-throughput scenarios. Each instance acquires row-level locks on a different batch, preventing duplicate publishes from concurrent workers.

Ordering Guarantees

The outbox pattern preserves per-aggregate ordering when you use aggregate_id as the Kafka message key. Kafka routes all messages with the same key to the same partition, and within a partition messages are totally ordered. Events for order a1b2c3d4 will always be consumed in the sequence they were inserted into the outbox table.

Cross-aggregate ordering — events for different orders interleaved — is not guaranteed, which is the correct behavior for most event-driven architectures. If your consumers need strict global ordering, you will need a single-partition topic, which sacrifices throughput significantly.

Outbox Cleanup

Processed outbox rows accumulate quickly in high-volume systems. Without cleanup, the outbox table becomes a significant source of storage bloat and index maintenance overhead. Schedule a periodic cleanup job — a cron task, a database job scheduler, or a pg_cron job in PostgreSQL.

-- Delete outbox rows processed more than 7 days ago
-- Run this as a scheduled job; avoid deleting in one large batch
DELETE FROM outbox
WHERE processed_at IS NOT NULL
  AND processed_at < now() - INTERVAL '7 days';

-- For MySQL
DELETE FROM outbox
WHERE processed_at IS NOT NULL
  AND processed_at < NOW() - INTERVAL 7 DAY
LIMIT 5000;  -- batch limit to avoid long-running lock contention

Retain processed rows for at least as long as your slowest downstream consumer's replay window. If a consumer needs to reprocess events from the last 24 hours, deleting outbox rows after 1 hour will break that capability.

Key Takeaways

Manage Your Outbox at Scale with JusDB

The transactional outbox pattern is straightforward in development, but running it reliably in production requires careful attention to index health, vacuum behavior, partition maintenance, and relay lag monitoring. JusDB gives backend engineers and DBAs the visibility they need to keep outbox-based systems healthy: query performance insights, table bloat analysis, index utilization tracking, and slow query alerts — all without adding instrumentation to your application code.

Whether you're running PostgreSQL or MySQL, JusDB surfaces the database-level signals that matter most for event-driven architectures: growing unprocessed outbox backlogs, degraded index scans on the WHERE processed_at IS NULL filter, or cleanup jobs taking longer than expected. Start a free trial at jusdb.com and connect your database in under five minutes.