Exactly Once
checkout lab
Same checkout, same failures. Only one version never charges twice or loses an order.
An $80 order runs through 4 services. Three retry strategies face the same 10,000 failure scenarios.
of calls hit a fault
Loading the engine…
Never retries
at most once
A dropped call stays dropped.
naive · fire and forget
checkout 64
Charged, never shipped
$80.00 charged · 0 shipments
3,131 of 10,000 checkouts went wrong
Retries blindly
at least once
A retry after a timeout or crash charges the card again.
naive · retry loop
checkout 64
Charged twice
$160.00 charged · 2 shipments
2,932 of 10,000 checkouts went wrong
Retries safely
exactly once
Every call carries an idempotency key, so the payments service ignores the repeat.
production patterns
checkout 64
Charged once, shipped once
$80.00 charged · 1 shipment
0 of 10,000 checkouts went wrong
51 waiting for a person 3 refunded or cancelled
What happened to checkout 64
Each square is one checkout. Select a red square to see a failure.
Never retries
- 40 msinventory commits reserve_stock
- 80 mspayments commits charge_card
- 80 msorders crashes before it reads the charge_card reply
- 80 msthe run ends with no ending
Retries blindly
- 40 msinventory commits reserve_stock
- 80 mspayments commits charge_card
- 80 msorders crashes before it reads the charge_card reply
- 580 msorders restarts and starts over: its progress was only in memory
- 580 msorders resends reserve_stock with no key
- 620 msorders resends charge_card with no key
- 660 mspayments replies 503 to charge_card
- 757 mspayments commits charge_card
- 797 msthe network delivers create_shipment twice
- 797 msshipping commits create_shipment
- 837 msthe run ends complete
- +5 more lines in the full log
Retries safely
- 80 mspayments commits charge_card
- 80 msorders crashes before it reads the charge_card reply
- 580 msorders restarts and resumes from its progress log← progress log
- 580 msorders resends charge_card with the same key 9f88cceb← idempotency key
- 620 mspayments replies 503 to charge_card
- 813 msorders backs off 193 ms before charge_card #3← 3 tries
- 813 msorders resends charge_card with the same key 9f88cceb← idempotency key
- 853 mspayments drops a repeat of charge_card · key 9f88cceb← idempotency key
- 893 msthe network delivers create_shipment twice
- 893 msshipping commits create_shipment
- 893 msshipping drops a repeat of create_shipment · key 32f7087d← idempotency key
- +3 more lines in the full log
Faults and logs for checkout 64
Select a cell, or type a checkout number from 0 to 9,999.
Never retries · at most once
Retries blindly · at least once
Retries safely · exactly once
First failure in each lane, shrunk
The engine finds and shrinks the first failure in each lane once it loads.
I'm Steven Mays, a lead backend and AI engineer with 15+ years in production systems. I built this lab to test retry code the way production tests it: with the network failing on purpose.
What exactly once means here
The network in this lab drops and repeats messages, as real networks do, so no lane gets exactly-once delivery. The exactly-once lane gets at-least-once delivery plus idempotent receivers, which makes each effect happen once. A retry or a repeat can't charge the card a second time. The guarantee holds only while each service remembers the keys it has seen, and the "Service forgets it already did the work" slider breaks that assumption on purpose.
The lab runs in your browser on a Rust engine compiled to WebAssembly. It makes no server calls and uses no model.
How the safe version works
Each protection below answers one fault in the safe lane and links its sources.
Idempotent Receiver
- Fault
- Timeout after commit, duplicate delivery
- Protection
- Idempotency key on every call, so the receiver drops repeats
- Sources
Idempotent Receiver, Enterprise Integration Patterns (Hohpe, Woolf)
Idempotent Receiver, Patterns of Distributed Systems (Joshi)
Write-Ahead Log
- Fault
- Crash after a step commits
- Protection
- Record each step in a durable log, and resume from the log
- Source
Write-Ahead Log, Patterns of Distributed Systems (Joshi)
Timeouts
- Fault
- Slow or failing service
- Protection
- Timeouts
- Source
Stability Patterns, Release It!, 2nd edition (Nygard)
Backoff and retry budget
- Fault
- Retries pile up
- Protection
- Randomized exponential backoff, and a retry budget of up to 3 attempts per request
- Sources
Addressing Cascading Failures, Site Reliability Engineering (Google)
Handling Overload, Site Reliability Engineering (Google)
Saga
Dead Letter Channel
- Fault
- A compensation can't be delivered, a step's outcome is unknown, or the confirmation can't be sent
- Protection
- Dead Letter Channel, where a person finishes the run
- Source
Dead Letter Channel, Enterprise Integration Patterns (Hohpe, Woolf)
Reading list
- Idempotent Receiver, Enterprise Integration Patterns (Hohpe, Woolf)
- Idempotent Receiver and Write-Ahead Log, Patterns of Distributed Systems (Joshi)
- Stability Patterns, Release It!, 2nd edition (Nygard)
- Addressing Cascading Failures, Site Reliability Engineering (Google)
- Handling Overload, Site Reliability Engineering (Google)
- Sagas, SIGMOD 1987 (Garcia-Molina, Salem)
- Saga, Microservices Patterns, chapter 4 (Richardson)
- Dead Letter Channel, Enterprise Integration Patterns (Hohpe, Woolf)
Lab rules
The checkout
orders runs one $80.00 checkout across 4 services, one step at a time:
reserve_stockoninventory.release_stockundoes it.charge_cardonpayments.refund_chargeundoes it.create_shipmentonshipping. A shipped order is never undone.send_confirmationonemail. It is retried, never undone.
The faults
Each checkout number is a seed: it fixes every fault, keyed by step and delivery number. All 3 retry strategies read the same faults, so a checkout replays exactly.
- 503: the service rejects the request and commits nothing. That's a rule of this lab. A real 503 doesn't promise it.
- Timeout after commit: the service commits, and the reply misses the 800 ms deadline. Every lane waits the same 800 ms.
- Lost before commit: the network loses the request before the service sees it. Nothing commits, and the caller sees the same timeout as a lost reply.
- Duplicate delivery: the network delivers the request twice.
- Crash after commit: the service commits, and
orderscrashes before it reads the reply. - Forgotten key: the service loses the record of a key it has seen, so a repeat with that key commits again. It breaks the safe lane on purpose, to show where the guarantee ends.
Retries back off on the simulated clock. Faults key on the delivery number, not the time, so backoff never changes a verdict.
The 3 retry strategies
Each strategy is a lane in the lab, named for the delivery guarantee its code gives.
- Never retries (at most once) calls each service once. A 503 or a crash drops a step. The customer is charged but never shipped, or the order disappears.
- Retries blindly (at least once) retries until each call succeeds. It sends no keys and keeps its progress in memory. After a timeout or a crash it resends a step that already committed, and a duplicate applies twice. The customer can be charged twice.
- Retries safely (exactly once) makes the same retries with an idempotency key on every call, a durable progress log, a retry budget of 3 attempts, and compensation. When a step fails for certain, the run refunds the charge and releases the stock. When the outcome is unknown, a person finishes it: the run parks the step and its key in a dead-letter queue, and the redrive sends it with the same key.
Invariants
A checker reads the 4 services’ ledgers after every run. A red cell breaks one of these:
- The card is charged at most once.
- If the order shipped, the card was charged exactly once.
- If the order didn't ship, the net charge is $0 and the stock is released.
- At most 1 confirmation goes out.
- Every run ends complete, compensated, or in the dead-letter queue.
Some red cells break a valid-ending check, such as “stock never reserved”, which goes beyond these 5 invariants. Each ending must leave the ledgers it promises. A dead-lettered run is checked when it parks and again after the redrive.