Idempotency: The Most Important Pattern in Distributed Systems

Introduction

Here is the cruelest fact in distributed systems: when a request times out, the client cannot know whether it happened. The response may have died on the way back after the work was done. There are exactly two options, give up (and maybe lose the order) or retry (and maybe duplicate it). Every reliable system chooses retry, which means every reliable system must make retries safe.

That property is idempotency: performing an operation twice has the same effect as performing it once. It sounds academic until you see a customer charged twice because a load balancer flaked at the wrong millisecond. This post covers how to actually build it, client-generated keys, a Postgres implementation, and the three edge cases that separate real implementations from decorative ones.

Why "Exactly Once" Is a Lie (and What's Actually On Offer)

Delivery guarantees come in two honest flavors: at-most-once (fire and forget; failures lose work) and at-least-once (retry until acknowledged; failures duplicate work). "Exactly-once delivery" over an unreliable network is impossible, the Two Generals problem is not waiting for a sufficiently clever framework.

What is achievable is exactly-once processing: deliver at-least-once, then make processing duplicates harmless. Idempotency is the mechanism. This reframing matters because it puts the responsibility where it belongs, in your data layer, not in your message broker's marketing page.

Some operations are naturally idempotent: PUT /users/42/email with a full value, DELETE /sessions/abc, any "set X to Y". The dangerous ones are creations and increments, POST /orders, charge $50, append event, where each execution mints new state. Those need explicit protection.

Idempotency Keys: The Client Names the Operation

The pattern (used by every serious payment API): the client generates a unique key per logical operation, not per HTTP attempt, and sends it with the request:

POST /v1/orders
Idempotency-Key: 9f1c2e34-7ab1-4f0e-a1d2-2b9c8d1a4e55

A retry of the same operation reuses the same key. The server's contract: the first request with a key executes; every subsequent request with that key returns the stored response of the first execution, same status code, same body, without re-executing anything.

The subtle half of the contract is scoping: the key identifies this user's intent to perform this operation once. Generate it when the user confirms the action (one key per checkout press, reused across network retries), scope it per-caller on the server, and give keys a TTL (24 hours is typical) so storage doesn't grow forever.

A Real Implementation in Postgres

The naive version, "SELECT to check if the key exists, then INSERT", has a race: two concurrent retries both see nothing, both execute. The database has to arbitrate. A unique constraint does it correctly:

CREATE TABLE idempotency_keys (
    key          text        NOT NULL,
    caller_id    bigint      NOT NULL,
    status       text        NOT NULL DEFAULT 'in_progress',  -- in_progress | done
    response_code int,
    response_body jsonb,
    created_at   timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (caller_id, key)
);

def handle_create_order(caller_id, key, payload):
    claimed = db.execute(
        """INSERT INTO idempotency_keys (key, caller_id)
           VALUES (%s, %s)
           ON CONFLICT (caller_id, key) DO NOTHING
           RETURNING key""",
        key, caller_id,
    )

    if claimed:                               # we own this operation
        result = create_order(payload)        # the actual business logic
        db.execute(
            """UPDATE idempotency_keys
               SET status='done', response_code=%s, response_body=%s
               WHERE caller_id=%s AND key=%s""",
            result.code, result.body, caller_id, key,
        )
        return result

    record = db.fetch_one(
        "SELECT status, response_code, response_body FROM idempotency_keys "
        "WHERE caller_id=%s AND key=%s", caller_id, key,
    )
    if record.status == "done":
        return Response(record.response_code, record.response_body)  # replay
    return Response(409, {"error": "operation already in progress, retry later"})

The atomic INSERT ... ON CONFLICT DO NOTHING RETURNING is the whole trick: exactly one concurrent caller "claims" the key; everyone else either replays the stored response or learns the operation is still running.

The Edge Cases That Separate Real from Decorative

1. Crash between claim and completion. The server claims the key, starts the work, and dies. The key is stuck in_progress forever, and retries get 409 until heat death. Fix: store a lease deadline with the claim (locked_until = now() + interval '90 seconds'); a retry that finds an expired in-progress claim may re-claim it, which also means your business logic must itself be safe to re-run, typically by doing the final write transactionally with the key update.

2. Same key, different payload. A client bug reuses a key with a changed request body. Silently replaying the stored response for a different request hides the bug. Store a hash of the request with the key; on mismatch return 422 Unprocessable, "this key was used for a different request."

3. Idempotency ends at your process boundary, extend it to your side effects. If create_order charges a card via a payment provider, your deduplication doesn't protect the charge, a crash after charging but before recording still double-charges on re-run. Propagate the key downstream: pass your idempotency key (or a derivative) as the payment provider's idempotency key, so the whole chain shares one notion of "this operation." Idempotency composes only if every effectful hop participates.

The same logic applies to queue consumers: at-least-once delivery from SQS/Kafka means your handler will see duplicates, and the message ID (or a business key from the event) plays the role of the idempotency key against a processed-events table, ideally updated in the same transaction as the state change it guards. This is exactly the discipline that makes at-least-once streaming pipelines behave like exactly-once ones.

Takeaways

• Timeouts make retries mandatory; retries make idempotency mandatory. It's a chain, not a preference.
• "Exactly-once" is achievable only as at-least-once delivery + idempotent processing.
• Client-generated keys, claimed atomically via a unique constraint, with stored responses for replay.
• Handle the three real-world cases: crashed in-progress claims (leases), key reuse with different payloads (request hashing), and downstream side effects (propagate the key).
• Design every consumer of every queue as if it will see duplicates, because it will.

Idempotency is unusual among architectural properties: it's cheap to add early, brutal to retrofit, and invisible exactly when it's working. Which is to say, it's what reliability actually looks like.