Most production duplicates are not delivery failures. The destination commits, fails to acknowledge, and the caller correctly retries. No broker guarantee covers that.

The Queue Delivered It Once. Your Database Recorded It Twice. Exactly-Once Is a Property of the Write, Not of the Broker.

TL;DR: Most production duplicates are not delivery failures; they occur when a destination commits a write but fails to acknowledge it, causing the caller to correctly retry. Because the broker cannot see inside your database transaction, "exactly-once" delivery guarantees cannot prevent these duplicates. The property of exactly-once execution must be enforced at the point of the write using idempotency keys and database constraints.

Nothing was delivered twice

Consider a common scenario in modern workflow automation. A developer sets up a webhook receiver to process incoming orders. The destination system receives the request, successfully saves the order to the database, but then—due to a transient network blip or a post-save logic error—returns an HTTP 500 Internal Server Error.

The caller, seeing the 500, behaves exactly as it should: it retries the request. The destination receives the retry, saves the order again, and now you have two identical orders.

According to a verified reproduction by Shabon on Substack, this exact behavior was observed in a controlled n8n test environment. The destination saved the event and then failed to respond; the HTTP Request node retried, and one logical event became two stored effects. Every component behaved exactly as specified in its documentation. Nothing was delivered twice by the broker. One thing was delivered once, acknowledged zero times, and recorded twice.

Where people think the guarantee lives

There is a persistent belief among engineering teams that "exactly-once" is a feature you select when you pick a message broker or a workflow engine. If you use Kafka, you might rely on its exactly-once semantics (EOS).

To be clear: Kafka’s exactly-once is real. It works for the "read-process-write" case where both the source and destination are Kafka, utilizing transactional producer capabilities. However, its scope is strictly bounded. According to Confluent's documentation, coordinating a consumer's position with a write to an external system is notoriously difficult.

The moment your consumer writes to a PostgreSQL database, calls an external API, or publishes to a cloud queue, the side effect falls outside the broker's transaction boundary. The standard remedy in distributed systems literature is to accept at-least-once delivery to the external system and prevent duplicate effects on the receiving side. The guarantee was never yours to buy from a vendor; it is yours to enforce at the database.

Key Takeaway: Exactly-once is not a delivery guarantee you configure on a broker, but a property of the write that you must enforce at the destination.

Three places a duplicate is born

To solve the problem, we must first categorize where duplicates actually originate. Only one of these is a delivery failure.

Delivery: The broker itself sends the same message twice. This is the least common scenario in modern, well-configured brokers, yet it is the only one "exactly-once" settings address.
Acknowledgement: The destination commits the write and then fails to acknowledge it (via timeout, connection reset, or a 500 error). The caller cannot distinguish a lost request from a lost response. No broker setting can touch this.
Replay: A human drains a Dead Letter Queue (DLQ), runs a backfill, or rolls back a deployment that resets consumer offsets. These are sanctioned, deliberate actions that produce duplicates by design.

Because two of these three categories live entirely outside the messaging layer, the logic to handle them must live where the data is written.

The acknowledgement gap, which is most of your duplicates

The "Acknowledgement Gap" is the source of the most persistent production issues. When a client times out, it is often because the server is under load. If that timeout is tuned just below the p99 of your write path, a minor latency regression turns into a duplicate-effect regression.

This is operationally nasty because there is no "error" in the logs that points to the root cause. The server thinks it succeeded; the client thinks it failed. Common shapes of this gap include:

TCP connection resets after the COMMIT but before the response.
A load balancer dropping a long-running connection.
A client-side deadline that is shorter than the server's database transaction timeout.

Replay is a feature and it duplicates on purpose

We often treat duplicates as accidents, but in operations, we create them on purpose. When an incident occurs, we might reprocess a window of events or drain a Dead Letter Queue. These actions are usually performed under pressure by humans.

If your system relies on the broker to prevent duplicates, a manual offset reset will bypass all your protections. The duplicate-write path is most likely to be exercised at the exact moment the team is least able to notice it—during the recovery phase of an outage.

What an idempotent write actually is

An idempotency key is the industry-standard tool for solving this, but implementation details matter. A simple "check-then-write" (look for the ID, then insert if missing) is not idempotent under concurrency; it is a race condition.

Real idempotency requires one of the following:

Database Constraints: A UNIQUE constraint on a business key (e.g., order_id).
Upserts: Using INSERT ... ON CONFLICT DO NOTHING.
Processed-Events Table: Recording the event ID in a dedicated table within the same transaction as the business logic. This makes the effect and the record of the effect atomic.

Crucially, the idempotency key must be chosen by the producer and remain stable across retries. If you generate a new UUID every time you retry a request, you have defeated the mechanism. Furthermore, you must store the original response. If a retry occurs, the system should return the same success message it generated the first time, rather than a generic "already exists" error, to ensure the caller's state machine stays in sync.

Effects you do not own

There is an honest limit to this argument. You can make a database write idempotent, but you cannot easily make an email send or a third-party API call idempotent unless that provider supports idempotency keys.

When dealing with side effects you don't own, the pattern must be:

Record the intent to act idempotently in your database.
Perform the external effect.
Record the completion.

If the system crashes between step 2 and 3, you have a reconciliation problem. It is better to admit this gap than to pretend a broker guarantee closes it.

Nothing will page you about this

Duplicate writes do not throw exceptions. They surface as a customer being charged twice or a reconciliation job that doesn't balance. Because there is no "Failed" status, detection must be based on invariant checks:

Comparing the count of stored effects against the count of distinct source event IDs.
Scanning for duplicate business keys in tables that should be unique.

This failure class is exactly the kind of unglamorous, standing question that never gets staffed. It is a proactive check that a system like Operate can perform on a schedule, opening a case only when stored effects and distinct source event IDs diverge.

The test nobody has in CI

To ensure your system is resilient, you need an integration test that no one likes to write:

Send the same event twice in rapid succession.
Simulate a "commit-then-fail" by killing the connection immediately after a database write but before the return.

If your system fails these tests, your behavior under duplicate delivery is unknown. In production, "unknown" is rarely the same as "correct." The fix is usually a schema change—a constraint or a processed-events table. These migrations against production databases are high-stakes; they are precisely the kind of changes that Operate drafts as PRs for humans to review and approve, ensuring the fix is as safe as the detection was precise.

Sources & further reading

According to Confluent, exactly-once is scoped to the broker and requires careful coordination when writing to external stores.
Shabon's n8n reproduction demonstrates how HTTP 500 retries lead to duplicate database records.
Research from OneUptime confirms that while the semantics are well-documented, the operational reality of managing side effects remains a significant challenge for SREs.
As noted in industry discussions on LinkedIn, automation that swallows exceptions often masks the very retries that cause these duplications.