Exactly Once Delivery Is a Property of the Write

Exactly once delivery is the promise that each message reaches its consumer one time, with no loss and no duplicate. It is real inside the systems that offer it, and it stops at their boundary. The duplicate that charges a customer twice is usually created on the far side of that boundary.

We see the same misunderstanding in a lot of incident reviews. A team chose a broker for its exactly-once setting, a duplicate order appeared anyway, and the first hour went on checking broker configuration that was never at fault.

In this post we cover what the guarantee covers in Kafka and SQS, the three places a duplicate is born, four steps that make the write itself exactly once, effects you do not own, how to detect duplicates nothing alerts on, and when idempotency keys are the wrong tool.

Nothing Was Delivered Twice and the Order Was Saved Twice

Take a webhook receiver that processes orders. It receives a request, commits the order to the database, and then a post-save step throws, so it returns HTTP 500. The caller sees a failure and does what every retry policy tells it to do.

The retry arrives, the receiver commits the order again, and there are now two orders. We would describe the sequence this way. One message was delivered once, acknowledged zero times and recorded twice.

Every component behaved as documented. The broker or HTTP client delivered once per attempt, the caller retried a failure, and the database accepted two valid inserts. The only missing piece was a rule at the write that said the second insert was the same event.

What Exactly Once Delivery Covers in Kafka and SQS

Kafka's guarantee is narrower than its name suggests. Confluent's delivery semantics documentation says Kafka supports exactly-once delivery in Kafka Streams and uses transactional producers and consumers between Kafka topics. For other systems it points to Kafka Connect offset management, and otherwise, in its words, "by default Kafka guarantees at-least-once delivery."

The same page names the hard part. When writing to an external system, coordinating the consumer's position with the data it wrote is challenging, typically handled by storing the position alongside the data. That is a design you build, not a flag you set.

Amazon SQS FIFO queues offer what AWS calls exactly-once processing. If you retry SendMessage within the 5-minute deduplication interval, SQS does not add a duplicate to the queue. We read that as protection on the way in, scoped to a time window.

System	What the guarantee covers	What it does not cover
Kafka transactions and Streams	Read-process-write between Kafka topics	A consumer's write to Postgres, an API call, an email
Kafka default	At-least-once delivery	Duplicate effects at the destination
SQS FIFO	Duplicate sends within the 5-minute deduplication interval	A retry after the interval, or a consumer that commits and crashes before deleting

Three Places a Duplicate Is Born

We sort duplicates by where they start, because only one of the three is a delivery problem. The other two live outside the messaging layer, which is why tuning the broker does not remove them.

1. Delivery Sends the Same Message Twice

The broker or client resends a message it already delivered, for example after a consumer rebalance. This is the case exactly-once settings address, and in a well-configured system it is the least common one we meet.

2. The Acknowledgement Gap After a Commit

The destination commits and then the response is lost to a timeout, a connection reset or a 500 from later code. The caller cannot tell a lost request from a lost response, so retrying is the only safe move it has. A client deadline set just under the write path's p99 turns a latency regression into a duplicate-write regression, with no error that points at it.

3. Replay That Duplicates on Purpose

Someone redrives a dead letter queue, runs a backfill or resets consumer offsets during recovery. These are sanctioned actions, and they produce duplicates by design. Our dead letter queue operating manual treats redrive as a change for that reason, because it exercises the duplicate path at the moment a team is least able to notice it.

Four Steps That Make the Write Itself Exactly Once

An idempotent write is one where applying the same event twice leaves the same result as applying it once. We get there with four decisions, and skipping any one of them reopens the gap.

1. Pick a Key the Producer Owns

The idempotency key has to identify the business event and stay the same across every retry. An order ID, a payment intent ID or an upstream event ID works. A UUID generated per attempt does not, because each retry then looks like a new event and the mechanism is defeated before it starts.

The key should come from the producer, not the consumer. If the consumer derives it from arrival time or a message offset, a replay through a different path produces a different key for the same event.

2. Let a Constraint Decide, Not a Lookup

Check-then-insert is a race. Two concurrent retries can both look, both see nothing, and both insert. We let the database decide instead, with a unique constraint on the key and an insert that tolerates a conflict.

In PostgreSQL that is INSERT ... ON CONFLICT DO NOTHING. The INSERT documentation says DO NOTHING simply avoids inserting a row as its alternative action, and that the DO UPDATE form guarantees an atomic insert or update outcome even under high concurrency. The constraint is the arbiter, so the race disappears.

3. Record the Effect in the Same Transaction

When the effect is more than one row, we keep a processed-events table keyed on the event ID and write to it inside the same transaction as the business change. Either both commit or neither does, so the record of the effect cannot drift from the effect.

This is the consumer-side mirror of the transactional outbox pattern. The outbox makes publishing atomic with the write on the producer side. The processed-events table makes consuming atomic with the write on the receiving side.

4. Return the First Response on a Retry

A retry should get the answer the first attempt would have returned, not a generic "already exists" error that the caller's state machine may treat as a failure. That means storing the original response, or enough to rebuild it, against the key.

Stripe's idempotent requests documentation is a clear public example. It saves the status code and body of the first request for a key, and "subsequent requests with the same key return the same result, including 500 errors."

Effects You Do Not Own Need an Intent Record

We can make our own database write idempotent. We cannot make an email send or a third-party call idempotent unless that provider accepts an idempotency key, and many do not. That is the honest limit of the argument.

For those effects we record the intent first, keyed on the event, then perform the call, then record completion. If the process dies between the call and the completion record, we have a reconciliation problem rather than a silent duplicate, and a reconciliation job can find it. Passing our own key through to providers that accept one closes most of the remaining gap.

Duplicate Writes Never Page Anyone

Duplicates do not throw exceptions. They show up as a customer charged twice, a ledger that does not balance or a support ticket a week later. We detect them with invariants, not error rates.

• Compare the count of stored effects with the count of distinct source event IDs over the same window.
• Scan for repeated business keys in tables that should hold one row per event.
• Track retry rate per endpoint next to its p99, since the acknowledgement gap grows when the two converge.
• Diff counts before and after every redrive or offset reset.

When one of those checks fails, the useful evidence is spread across the queue, the application logs and the table itself. It is worth being able to connect an investigator read-only to your logs and database so the comparison does not depend on who remembers where each count lives. Retries that pile up for the same reason are also the start of a retry storm, so we read the two signals together.

When Idempotency Keys Are the Wrong Tool

Some writes are idempotent already. Setting a status to shipped, overwriting a profile field or writing a computed snapshot gives the same result however often it runs, and adding a key table to them adds cost and a failure mode without removing a risk.

Keys also expire. SQS deduplicates for five minutes, and Stripe lets keys be removed once they are at least 24 hours old. A dead letter queue redriven three days later sails past both, which is why the durable guarantee has to sit in your own constraint with a retention that outlives your longest replay.

Finally, an idempotency key is not a fix for a bad retry policy. If a caller retries a request that is still running, we have two copies in flight, and the right answer is a longer deadline or a status check before retrying, not only a key at the bottom.

Test Exactly Once Delivery Where the Write Happens

We would add two integration tests that almost nobody has. Send the same event twice in quick succession, and kill the connection immediately after the commit but before the response. If the system passes both, its behavior under duplicates is known, and exactly once delivery stops being a broker setting and becomes a property you have checked.

The fix, when the tests fail, is usually a schema change, which deserves the same care as any production migration. Operate investigates production issues through read-only adapters to logs, databases and repositories, and proposes any code fix as a patch file an engineer reviews and applies.