The idea: get an event out of a transaction, once
The Order service saves an order in its own database. The Payment service has to hear about it and charge the customer. The message goes through Kafka. Two things must hold, even when a process crashes at the worst moment: no event is lost (every saved order is eventually charged) and no event is applied twice (nobody is charged twice).
The page has three parts: the Order service with its database orders_db, a message relay, the Kafka topic order-events, and the Payment service with payments_db. Press Place Order, Relay Poll and Consume in that order, or run a demo. The crash select makes one component die at a chosen point of its next action. It restarts at its following action.
The dual-write problem
The obvious code writes twice: commit the order, then send the event to Kafka. These are two writes to two systems, and no transaction covers both. If the process dies between them, the order exists but the event never leaves (Demo 2: dual write loses the event). Swapping the order of the two writes does not help. If you send first and then the database transaction fails or rolls back, Payment charges an order that does not exist.
Retrying in the app does not fix it either: after a crash, nothing remembers that a send is still owed.
The transactional outbox
Make the event part of the database write. The app inserts the order row and a row in an outbox table in one local transaction:
BEGIN;
INSERT INTO orders (id, status) VALUES (1, 'NEW');
INSERT INTO outbox (aggregate_id, event_type, message_id, payload, published_at)
VALUES ('order 1', 'OrderPlaced', 'evt-1', '{...}', NULL);
COMMIT;
The database guarantees both rows or neither. A crash right after COMMIT loses nothing: the event is already stored and waits to be sent (Demo 3: same crash, with an outbox). The app itself never talks to Kafka.
The message relay: polling or log tailing
A separate process, the message relay, moves outbox rows to the broker. The page uses a polling publisher. On every poll it runs:
SELECT * FROM outbox WHERE published_at IS NULL ORDER BY id LIMIT 4- send each row to Kafka (
acks=all) - wait for the acks
UPDATE outbox SET published_at = now() WHERE id IN (…)
Steps 2 and 4 are again two writes to two systems. If the relay dies after Kafka acked but before the UPDATE, the rows still say NULL. The next poll sends them again, and the topic now holds the same message_id twice (Demo 4). That trade is deliberate: the relay can lose nothing, so it may send more than once. That is at-least-once delivery. Exactly-once cannot be had on the sending side alone.
The other kind of relay is log tailing, also called change data capture (CDC). A connector such as Debezium reads the database's own change log (the MySQL binlog, the PostgreSQL WAL) and turns every committed outbox insert into a Kafka record. Nothing polls the table, latency is lower and the database does less work. Debezium ships an outbox event router for exactly this. It is at-least-once too: after a crash the connector restarts from the last log position it stored, which can be before records it had already sent. See Change Data Capture for this relay animated: the WAL, the replication slot, offsets and the duplicates a connector crash causes.
| polling publisher | log tailing (CDC) | |
|---|---|---|
| reads | the outbox table, with a query | the database's transaction log |
| latency | up to one poll interval | close to the commit |
| load on the database | a query every interval, plus an UPDATE | reads the log; the outbox rows can be deleted at once |
| remembers progress in | published_at | the connector's stored log position |
| delivery | at-least-once | at-least-once |
| needs | nothing beyond SQL | log access and a connector to run |
The inbox: an idempotent consumer
Duplicates also come from the consumer side. A consumer commits its Kafka offset only after it has handled a record. If it crashes between its database COMMIT and the offset commit, it gets the same records again after a restart (Demo 6). See Kafka Consumer Groups for the same thing during a rebalance.
So the consumer must make a second copy harmless. The inbox (also called a processed messages or deduplication table) has the message_id as its primary key. The consumer inserts into it in the same transaction as its own work:
BEGIN;
INSERT INTO inbox (message_id) VALUES ('evt-1'); -- duplicate key → ROLLBACK, skip
INSERT INTO payments (order_id, amount) VALUES (1, 100);
COMMIT;
-- then commit the Kafka offset
Either the payment and the inbox row are both saved, or neither is. A copy that arrives later fails on the primary key and is skipped (Demo 5: relay crash, inbox skips the copy). Without the inbox the same copy charges the order a second time (Demo 4: relay crash, no inbox: charged twice).
Checking the inbox and doing the work in two separate transactions is not enough. A crash between them brings back the same gap. Some work is idempotent by nature (SET status = 'PAID') and needs no inbox, but "charge 100" is not.
At-least-once + dedup = effectively once
Put together: the outbox means no event is lost, the relay and Kafka may deliver it more than once, and the inbox means it takes effect once. This is often called effectively-once processing. The network is not exactly-once, but every effect happens once.
| approach | event lost? | applied twice? | cost |
|---|---|---|---|
| dual write | yes, on a crash between the writes | yes, with retries | none, until it breaks |
| outbox only | no | yes, after a relay or consumer crash | an outbox table and a relay |
| outbox + inbox | no | no | also an inbox table per consumer |
| Kafka transactions (EOS) | no | no, for Kafka-to-Kafka processing | only covers records and offsets inside Kafka, not your database |
Ordering
The relay reads rows ORDER BY id and sends them one partition at a time, so events leave in the order they were committed (Demo 7: three orders, kept in order). The record key is the aggregate id (the order), so every event of one order lands in the same partition and is read in order. Two relays polling the same table at once would break this. In practice there is one active relay, picked by a lock or leader election, or each relay owns a disjoint slice of the aggregate ids. A duplicate can still arrive after a newer event, and the inbox drops it.
Housekeeping
Both tables grow forever unless cleaned. Published outbox rows can be deleted after a while (with CDC, right away). Inbox rows only need to be kept for as long as a duplicate can still arrive. That dedup window has to be longer than the longest redelivery: topic retention, consumer downtime, a replay from an old offset.
See also Saga vs Two-Phase Commit: a saga's steps are messages between services, and each step needs exactly this outbox and inbox to be safe. And How Kafka Moves a Message for acks, partitions and offsets.
Every reply and event in a saga needs this; see Saga: Orchestration vs Choreography for a whole saga built from such messages.
With event sourcing the event stream itself can serve as the outbox; see Event Sourcing and CQRS for commands, an append-only event store and projected read models.
See also Delivery Semantics: why a dedup key turns at-least-once delivery into an effect that happens once.
What the page leaves out
Several relay instances and how one is chosen; the CDC connector's internals and its stored log positions; the payload and its schema; records that keep failing (retries, dead-letter topics); outboxes in databases without multi-row transactions (some NoSQL stores keep the events inside the same document); several partitions; and the Payment service sending its own reply event through its own outbox.