The Outbox Pattern: Commit and Publish Atomically

Two lines of perfectly ordinary code:

await db.order.create({ data: order });        // committed
await broker.publish('OrderCreated', order);   // ← crash here

The order exists. Nobody was told. No inventory reserved, no confirmation email, no analytics event — and nothing anywhere will ever notice, because from the database’s point of view the write succeeded and from the broker’s point of view nothing was ever attempted.

This is the dual write problem, and swapping the order of the two lines just changes which way it breaks: publish first and a crash before the commit announces an order that does not exist.

Why you cannot just be careful

Retry the publish? The process died. There is nothing left to retry with.

Wrap both in a transaction? The broker is not in your database’s transaction. There is nothing to enrol.

Two-phase commit? Technically possible, in practice avoided — it needs a transaction manager, it holds locks across a network round trip, and most modern brokers do not support XA at all.

The problem is structural: two systems, no shared atomicity. So stop trying to make two writes atomic and make it one write.

The outbox

The event goes into a table in the same database, in the same transaction as the data:

CREATE TABLE outbox (
  id            bigserial PRIMARY KEY,
  aggregate_id  uuid        NOT NULL,
  event_type    text        NOT NULL,
  payload       jsonb       NOT NULL,
  created_at    timestamptz NOT NULL DEFAULT now(),
  published_at  timestamptz
);

await db.$transaction(async (tx) => {
  const order = await tx.order.create({ data });
  await tx.outbox.create({ data: {
    aggregateId: order.id,
    eventType: 'OrderCreated',
    payload: order,
  }});
});

Both rows commit or neither does. There is no window. If the process dies anywhere, the database is in one of exactly two states, and both are correct.

A separate publisher then moves events out:

const batch = await db.outbox.findMany({
  where: { publishedAt: null },
  orderBy: { id: 'asc' },
  take: 100,
});

for (const event of batch) {
  await broker.publish(event.eventType, event.payload);
  await db.outbox.update({ where: { id: event.id }, data: { publishedAt: new Date() } });
}

At-least-once, which means consumers must be idempotent

The outbox guarantees the event is never lost. It explicitly does not guarantee it is delivered once.

So every consumer must be able to process the same event twice without effect — dedupe on the event id, or make the handler naturally idempotent. This is the same requirement idempotency keys exist for, and the same reason exactly-once is a myth at the delivery layer.

If your consumers are not idempotent, the outbox has not made your system reliable. It has made it reliably duplicated.

Polling or change data capture

Polling. A worker queries for unpublished rows on an interval. Simple, no new infrastructure, easy to reason about. Costs a query every interval and adds latency equal to roughly half the poll period.

Run it with SELECT … FOR UPDATE SKIP LOCKED so multiple publishers can share the work without processing the same row twice — and so a stuck row does not block the queue.

Change data capture. Debezium (or equivalent) tails the write-ahead log and publishes changes as they are committed. No polling load on the database, lower latency, and it cannot miss a row. The cost is real: Kafka Connect, replication slots, schema handling, and a new thing to monitor.

Start with polling. Move to CDC when the poll interval or the query load becomes a measured problem — not before.

Watch your replication slots if you do use CDC: an inactive slot stops Postgres from recycling WAL, and the disk fills. It is the most common way a CDC setup takes down the database it was reading.

Ordering, honestly

Rows come out of the outbox in id order, which is close to the order they were created. Two caveats:

• With concurrent transactions, sequence values are assigned before commit, so a transaction with a lower id can commit after one with a higher id. A naive WHERE id > last_seen publisher can skip a row that had not committed when it ran.
• Ordering across the broker is only preserved within a partition. Key by aggregate id so all events for one order land on one partition and stay ordered relative to each other.

For most systems, per-aggregate ordering is what actually matters, and global ordering is a requirement people assume they need and rarely do.

Housekeeping

The outbox table grows forever unless you tend it. Delete published rows beyond your retention window — a partitioned table by day makes this a DROP (see partitioning) rather than a large DELETE. Index on published_at so the publisher’s query stays cheap as the table grows, and alert on the oldest unpublished row, because a stalled publisher is invisible otherwise: the application keeps working perfectly and nothing downstream hears anything.

What you have actually built

Not reliable messaging. The broker is as reliable as it ever was.

You have made the decision to publish part of the same fact as the data change. Once the transaction commits, the event exists — the only remaining question is when it gets sent, and that is a liveness problem you can monitor and retry. Losing it is no longer possible, which converts a silent correctness bug into a visible operational one.