At 10:00 on launch day a client's flash sale opened to 30,000 waiting customers. By 10:06 the support inbox had 140 emails saying the same thing: I was charged, but I have no order.
The obvious suspects
The payment gateway's dashboard showed every callback delivered with a 200 response. Our API was under load, so we scaled from 4 to 12 pods. The failure rate stayed at roughly 8%, which told us capacity was not the problem.
The real clue was in the timestamps. For every missing order, the browser redirect and the gateway webhook had arrived within 40 milliseconds of each other.
Two writers, one row
Both handlers read the pending order, saw it unpaid, and tried to mark it paid. One of them also cleared the cart and, in a bad branch, rolled back the order it had just written. We fixed it with an idempotency key per payment intent and a row lock on the order.
await db.transaction(async (tx) => {
const order = await tx.orders
.where({ paymentIntent: key })
.forUpdate().first();
if (order.status === "paid") return order; // second writer exits
return tx.orders.markPaid(order.id);
});We have run four sales on the same system since. Zero dropped orders. Every payment handler we write now starts from the assumption that it will be called twice.
Why this happens
Nothing was actually broken. The browser redirect and the payment gateway's webhook are two independent clients, and under load they can both reach your API within milliseconds of each other. A "check status, then write" handler has a gap between the read and the write, and two requests can both pass the check before either has written. No amount of extra servers closes that gap — it only makes the race faster.
Applying this beyond payments
The same shape shows up anywhere two writers can act on the same row: coupon redemption, inventory decrements, webhook processing in general. Whenever you have a "check-then-act" operation that a client might call more than once — by retry, by double-click, or by two independent systems — the fix is the same checklist:
- Pick a key that identifies the logical operation, not the HTTP request — in this case the payment intent, not the request ID.
- Lock or constrain at the database level, not in application memory, so it holds even across multiple server instances.
- Write the handler so calling it twice is harmless, not just unlikely.
We now add idempotency key support to any endpoint that touches money or inventory as a default, not an afterthought, and treat a duplicate-key hit in the logs as a signal the protection is working — not a bug.