evanson
← all writing tech

The idempotency key that did absolutely nothing

Three independent defenses against double-charging a customer. A user tapped three times and got billed three times. All three defenses were present, correct, and useless at once.

2 min read
  • #idempotency
  • #databases
  • #bugs

I had three independent defenses against billing a customer twice for one order: a client-generated idempotency key sent with every attempt, a server-side check for that key before creating anything, and a UNIQUE index on the key column as a final backstop. Belt, suspenders, and a second belt.

A user on a flaky connection tapped “place order.” It seemed to hang, so they tapped again. Twice. They were billed three times for one delivery.

Here’s the part that kept me up at night: every one of those three defenses was present, correct in isolation, and completely useless at the same moment.

Three safety nets, three no-ops

The bug was two quiet traps stacked on top of each other, and either one alone would have been survivable.

  • The ORM only saves fields you allow-list. The idempotency key wasn’t on that list, so the model dutifully wrote NULL into the key column on every insert. The value arrived from the client, sat in the request, and got dropped on the floor on the way to the database.
  • A UNIQUE index treats every NULL as distinct. This is correct SQL behavior and exactly the wrong thing here. Two rows with NULL keys don’t collide. Three rows with NULL keys don’t collide. The index I was trusting to catch duplicates considered all three orders perfectly unique.

So the server check found nothing, because there was never a non-null key to find. The unique index allowed everything, because nulls don’t conflict. The keys the client worked so hard to generate never survived the first hop.

Why it hid so well

Each piece looked right on its own. The client code was correct. The controller logic was correct. The migration that added the unique index was correct. You could read all three in a pull request and approve every one. The failure lived only in the seam between the ORM’s allow-list and the database’s null semantics, which is exactly the kind of place no single file ever shows you.

Idempotency you never proved by firing the same request three times is not idempotency. It’s a comment that says idempotency.

Prove it or you don’t have it

The fix was trivial once I saw it: add the key to the allow-list, and make the column NOT NULL so the database refuses to store the ambiguous state at all. But the fix isn’t the lesson. The lesson is that I believed this was handled because every part was named correctly. “idempotencyKey.” “unique index.” The words were all there, so my brain checked the box.

Now, anything that’s supposed to be idempotent doesn’t get to be “done” until I’ve actually fired the same request three times in a row and watched the second and third bounce. Not reasoned about it. Sent it, and looked. The number of safety nets in your code tells you nothing. The only thing that counts is what survives the third tap.

Email