Transactional Outbox Pattern Problems That Start on Day Two
The transactional outbox pattern fixes dual writes and adds relay stalls, lag, churn, poison messages and skipped rows. How to detect each one in production.
Quick Answer
The transactional outbox pattern writes each outgoing message to an outbox table in the same database transaction as the business change, and a relay publishes it later. That removes the dual-write problem but adds new production failures. Relays stall silently, lag turns into stale downstream state, the table churns, poison messages block the queue, polling misses rows under concurrency, and direct publishes creep back in. Each one needs its own metric.
The transactional outbox pattern is a way to update a database and publish a message without losing either one. The service writes the message into an outbox table inside the same transaction as the business data, and a separate relay reads that table and sends the messages to the broker.
Most tutorials stop once the relay publishes its first message. We think the interesting part starts the day after launch, when the relay becomes a piece of production infrastructure that can fail while every API health check stays green.
In this post we cover what the pattern guarantees, six failures that appear in production, the metrics we put on every outbox, how polling and log-based relays differ, and when the pattern is more than a system needs.
What the Transactional Outbox Pattern Guarantees and What It Does Not
AWS describes the problem it solves in its prescriptive guidance on the transactional outbox. A dual write happens when one operation involves both a database write and a message notification, and either half can fail on its own. Putting the message in the same transaction means that if the data is saved, the message is saved too.
That is the guarantee, and it is narrower than it sounds. We list what it does not cover before building anything on top of it.
- Not exactly once. The relay can publish a message and crash before recording that it did.
- Not immediate. There is always a gap between the commit and the relay publishing the row.
- Not ordered by default. Concurrent transactions can commit in a different order than their rows suggest.
Chris Richardson's pattern description is direct about the first point. "The Message relay might publish a message more than once," so consumers have to be idempotent. Making that true at the point of the write is its own discipline, which we cover in exactly-once is a property of the write.
Six Transactional Outbox Pattern Failures That Start on Day Two
These are the failures we plan for on any outbox in production. Each one leaves the business transaction succeeding, which is why each one can run for days before anyone notices.
1. The Relay Stalls
The relay is a polling worker or a change data capture process, and either can stop. A crash loop on one malformed payload, lock contention on the outbox table or an unhandled exception in a single-instance relay will all halt publishing while writes continue.
The signal is the age of the oldest unpublished row. A growing row count can mean a busy period, but an oldest-row age that keeps climbing means nothing is moving.
2. Lag Turns Into Stale Downstream State
Eventual consistency is only acceptable while the eventual part is short. When write volume outgrows a single-threaded relay, search indexes, caches and other services start acting on old data, and users see records that are missing or out of date.
We treat outbox lag as a service level objective, measured as the time from commit to publish, and alert when it exceeds what the business can tolerate.
3. The Table Churns
Every message is an insert followed by an update that marks it sent, or by a delete. On PostgreSQL that churn leaves dead rows for vacuum to clean up, and a high-volume outbox can become one of the busiest tables in the database. We watch dead tuples and vacuum activity on it, and our approach to reading autovacuum is in the autovacuum decision procedure.
Polling adds its own cost. Gunnar Morling, writing for Decodable in Revisiting the Outbox Pattern, notes that polling "can be resource-intensive, creating spikes of high load on the database upon each polling attempt."
4. A Poison Message Blocks the Queue
A poison message fails every time the relay tries it, such as a payload larger than the broker accepts. If the relay retries until success to protect ordering, that one row blocks every message behind it.
We move a row aside after a fixed number of attempts, into a quarantine table with the error attached, so the rest can flow. That table needs an owner and an alert, or it becomes a place where failures are forgotten, as we describe in the dead letter queue manual.
5. Polling Misses Rows Under Concurrency
A relay that selects rows with an ID or timestamp above the last one it saw assumes rows become visible in that order. They do not. Morling's article puts it plainly, citing Martin Kleppmann. With parallel transactions, "it can't be guaranteed that the order of messages in the outbox table" matches "the order of commits."
A transaction that took ID 100 but commits after the transaction holding ID 101 can be skipped for good if the relay has already moved past 100. We either read from the transaction log, or poll for unpublished rows instead of rows above a high-water mark.
6. The Dual Write Comes Back
A new feature publishes straight to the broker from inside a request handler, because its author did not know about the outbox. The safety the pattern bought is gone for that event type, and nothing fails until the broker is briefly unavailable.
We catch this in code review and with a periodic comparison between business records and published events. Any event type that appears on the broker without a matching outbox row is a direct publish.
When one of these failures does surface, root cause analysis needs evidence that is spread across the outbox table, the relay's logs and the broker. It helps to connect it read-only to your logs and database so an investigation can read all three at once.
The Metrics and Alerts We Put on Every Outbox
An outbox without these signals is a queue nobody is watching. This is the starting set we instrument before go-live.
- Unpublished row count, which shows the backlog.
- Age of the oldest unpublished row, which is the single best stall signal.
- Commit-to-publish lag at the 99th percentile, measured against an agreed objective.
- Relay publish errors, split by error type so poison messages stand out.
- A relay heartbeat, so a dead process pages even when the table is quiet.
- Quarantined row count, with an owner who reviews it.
The thresholds depend on the business. A payment notification and a search index update tolerate very different delays, and we set each alert from that tolerance rather than from a default.
How Polling Relays and Log-Based Relays Differ
The relay design decides which failures are likely. We choose it with the trade-offs below in view.
| Relay design | Strength | Main risk |
|---|---|---|
| Polling the outbox table | Simple to build and to reason about | Load spikes, churn and rows skipped under concurrency |
| Change data capture from the transaction log | Commit-ordered and no polling load | A connector and replication slot to operate and monitor |
| Logical decoding messages without a table | No outbox table to vacuum | Tied to PostgreSQL logical decoding and its tooling |
The third option uses pg_logical_emit_message, which the PostgreSQL administration functions docs describe as a way to pass generic messages to logical decoding plugins through WAL, with a parameter that makes the message part of the current transaction. Downstream, consumer lag on the broker becomes the next thing to watch, which we cover in Kafka consumer lag shapes.
When the Outbox Pattern Is More Than You Need
We do not reach for an outbox by default. If a service has no consumers outside its own database, a plain transaction already gives it consistency, and an outbox only adds a relay to operate.
Some systems can also tolerate an occasional lost notification, such as a non-critical analytics event, and a best-effort publish with logging is cheaper to run. Others suit event sourcing better, which the pattern page lists as an alternative solution.
The outbox earns its cost when another service acts on the message and a lost or duplicated event would cause real harm. That is where its failure modes are worth instrumenting.
Instrument Your Transactional Outbox Pattern Before It Stalls
We would add the oldest-row age, the lag objective and the relay heartbeat first, then decide between polling and log-based relaying with the concurrency risk in mind. Those three signals turn a silent stall into a page while the backlog is still small.
A transactional outbox pattern is reliable only while someone watches the relay. When it does stall, Operate helps find out why. It brings the evidence from logs, databases and code together through read-only adapters, and proposes any code fix as a patch file an engineer reviews and applies.
Frequently Asked Questions
It is a pattern for updating a database and publishing a message reliably. We write the message to an outbox table in the same transaction as the data change, then a relay publishes it to the broker. If the transaction commits, the message is guaranteed to be published at least once.
We use it when another service acts on an event and losing that event would cause real harm, such as payments, orders or inventory changes. If nothing outside the service depends on the message, or an occasional loss is acceptable, a simpler publish is usually enough.
The outbox is the design, and change data capture is one way to run its relay. A CDC tool reads the database's transaction log and publishes outbox rows in commit order, while a polling relay queries the table. We prefer CDC at higher volume because it avoids polling load and skipped rows.
The outbox makes one service's database change and message publish consistent. A saga coordinates a long-running business process across several services through a sequence of steps and compensations. We usually see them together, because each saga step needs its messages published reliably, which is the outbox's job.
It adds a relay to operate, extra write load and churn on the database, publishing lag, and at-least-once delivery that forces consumers to be idempotent. In our experience the biggest cost is that relay failures are silent unless the outbox is instrumented from the start.
About the author
Operate Team
The team behind Operate
Operate Team builds Operate, enterprise AI agents for production operations. Operate answers production questions from Slack and Microsoft Teams, investigates alerts, and reviews performance, security and cost. It finds the root cause with evidence and proposes a fix for an engineer to review, with controlled access to your systems.
