Idempotency & Exactly-Once Effects

Networks retry — so your API will see duplicate requests. We design for at-least-once delivery and make handlers idempotent so duplicates produce the same effect as one call.

Separate delivery guarantee (how many times a message arrives) from effect guarantee (how many times the world changes). Interview answer: "We use at-least-once delivery with idempotent handlers — that gives us effectively-once side effects."

Distributed systems duplicate messages — idempotency keys and dedup make retries safe. Three lenses:

Walk payment retry as interview steps. Step 1 — Client sends: POST with Idempotency-Key UUID. Step 2 — Server check: lookup key in idempotency store. Step 3 — First seen: process payment, store result keyed by idempotency key. Step 4 — Duplicate: return cached result without re-charging. Step 5 — TTL: expire keys after 24h; state at-least-once + idempotent consumer model.

Idempotent handler skeleton

def charge_payment(idempotency_key: str, amount: int):
    cached = dedup_store.get(idempotency_key)
    if cached:
        return cached  # duplicate request — return same response

    if not dedup_store.setnx(idempotency_key, "processing", ttl=86400):
        return wait_for_inflight(idempotency_key)  # concurrent duplicate

    try:
        result = payment_processor.charge(amount)
        dedup_store.set(idempotency_key, result, ttl=86400)
        return result
    except Exception:
        dedup_store.delete(idempotency_key)  # allow retry on failure
        raise

At-most-once, at-least-once, exactly-once effects — delivery semantics we compare:

• Delivery vs effect guarantees — what the transport promises vs what your system must enforce:

• Dedup store options — pick by durability and retention requirements:

• Idempotency key rules: client-generated UUID v4 or deterministic hash of (user_id + operation + payload fingerprint); scope keys per tenant; TTL ≥ max client retry window (typically 24h for payments).
• Concurrent duplicates: two requests with the same key in flight — use SETNX "processing" state or DB unique constraint to serialize; second waiter polls or blocks briefly.

Safe retries trade storage for dedup records and design discipline — we state both:

Chasing true end-to-end exactly-once (Kafka transactional EOS + DB 2PC) adds latency and operational complexity. In most interviews, at-least-once + idempotent sink is the pragmatic answer unless the prompt explicitly demands ledger-grade EOS.

Duplicate charges, lost updates, and idempotency store races — we volunteer mitigations:

Payments, webhooks, and queue consumers require idempotency in HLD discussions:

Add idempotency keys when POST operations must survive client retries or message redelivery:

Idempotency connects to message queues, saga compensations, and API contract design:

• #11 Distributed Transactions: 2PC vs Saga — saga compensations must be idempotent; choreography retries duplicate forward steps.
• #30 Microservices Patterns — transactional outbox publishes events at-least-once; consumers dedupe by event_id.
• #32 API Contract & Integration Design — idempotency key header contract, 409 vs 200 on replay, TTL policy.
• #17 Message Queues Fundamentals — at-least-once vs exactly-once delivery semantics at the broker layer.
• #24 Payment Gateway (problem) — idempotency keys on charge endpoints and webhook settlement dedup.
• #103 Workflow Orchestration (Temporal) (problem) — activity idempotency keys for effectively-once side effects.
• #78 Banking Ledger System (problem) — ledger entries keyed by idempotency token to prevent double-posting.
• #102 CDC Pipeline (problem) — idempotent sinks (upsert by PK) for at-least-once stream delivery.

Transactional Outbox + Idempotent Consumers

The transactional outbox pattern (see concept #30) writes business state and an outbox event row in one local DB transaction. A CDC relay (Debezium) or polling publisher delivers events to Kafka at least once. Downstream consumers achieve effectively-once processing by:

1. Storing processed event_id in a dedup table (or Redis SET with TTL).
2. Skipping handler logic if event_id already exists.
3. Using idempotent sink writes: INSERT ON CONFLICT DO UPDATE, Elasticsearch upsert by doc_id, Redis SET.

This stacks cleanly with concept #11: saga orchestrators emit compensating events through the same outbox, and compensations carry their own idempotency keys so rollback retries do not double-refund.

Idempotency Key Storage Schema

CREATE TABLE idempotency_keys (
  key           VARCHAR(64) PRIMARY KEY,
  request_hash  VARCHAR(64) NOT NULL,  -- reject key reuse with different body
  status        ENUM('processing','completed','failed'),
  response_body JSONB,
  created_at    TIMESTAMPTZ DEFAULT now(),
  expires_at    TIMESTAMPTZ NOT NULL
);
CREATE INDEX idx_idempotency_expires ON idempotency_keys(expires_at);

Reject requests where the same key arrives with a different payload hash (HTTP 422). Background job deletes expired rows. For high QPS, front Redis with SETEX idempotency:{key} and async flush to Postgres for audit.

Non-Idempotent Operations Made Safe

• Counter increment → store "applied event_ids" set; only increment if event_id not seen.
• Send email → dedup by (template_id, recipient, idempotency_key); email provider may also dedupe.
• Reserve seat → conditional UPDATE WHERE seats_available > 0 + unique (show_id, seat_id) constraint; retry returns existing reservation for same key.
• Append to event log → event sourcing (problem #34 Event Sourcing System) uses deterministic event IDs so replays are no-ops.

Kafka Exactly-Once Processing (When to Mention)

Kafka's transactional producer (enable.idempotence=true) + transactional consume-process-produce gives EOS within the Kafka cluster. External side effects (DB, HTTP) still need idempotent handlers. Interview framing: "EOS in Kafka prevents duplicate writes to output topics; we still idempotency-key external API calls."