Debug concurrent data changes

Data and Concurrency Debugging

Races have deterministic causes that show up nondeterministically. Find the shared state, then control the timing until the failure happens every time.

Pin down the mismatch

• State exactly what is wrong: which records and fields, expected versus actual, since when. Compare with the source of truth (payment provider, ledger, audit log, event history).
• Build a timeline for the affected records from logs and audit data: every write, by which request, job or user, with timestamps and correlation ids. Interleaved writes to the same record are the first suspect.
• Ask of every read: could it hit a different place than the write? Most "it saved but shows the old value" bugs are written to one place and read from another.

Usual causes

• Check then act: two requests both see "no such email" and both create a user.
• Read, modify, write: two requests read stock 1 and both write 0; one update is lost.
• Acting on a stale status: two workers both see "paid", both call the shipping API.
• Duplicate delivery without deduplication: a retried request, a resent webhook, a queue message delivered twice, overlapping scheduled runs (see background-jobs for the job-side rules).
• Partial failure: step two of three failed with no transaction or compensation, leaving the record half done.
• Consistency gaps: replica lag (typically 10–500 ms), a cache stale until its TTL, a search index that refreshes seconds later, a read model waiting for an event to be consumed.
• Isolation: under read committed, the default of most databases, two transactions can read the same row and both write. Repeatable read and serializable prevent more anomalies but make conflicting transactions fail, so the caller must retry.
• Client side: an older response arriving after a newer one, double clicks, two tabs, an optimistic update that was never rolled back.

Reproduce it on purpose

• Run the competing operations truly in parallel on the same record (two clients, two browser contexts, two workers), with unique test data per run.
• Control the timing instead of sleeping: hold one request or transaction at the critical point (a breakpoint, a hook, an intercepted request), release the other first, then try the reverse order.
• Widen the window: add latency, throttle CPU or network, loop the scenario many times in parallel. Find the threshold by halving the delay between the two operations.
• Assert on the final settled state (counts, sums, uniqueness) and repeat the test; one failure in many runs is a failure.

Fixes, simplest first

1. A unique constraint that rejects the duplicate.
2. An atomic update (SET stock = stock - 1 WHERE id = ? AND stock > 0) with a check of the affected row count.
3. An upsert instead of a lookup followed by an insert.
4. A conditional status change (WHERE status = 'paid') claimed before the side effect.
5. Optimistic locking with a version column; report or retry the conflict.
6. A row lock (SELECT ... FOR UPDATE) in a short transaction when work must happen between the read and the write.
7. A distributed lock only as a last resort, with an expiry, and a fencing token when it guards external work.

• Reads that must show the user's own write: return the written data, or read from the primary right after the write; update the cache on write instead of deleting it.
• Make retried operations idempotent so duplicate deliveries do nothing.

Before you finish

• The mismatch is stated with record ids and a timeline.
• The race or gap was reproduced deterministically, or why not is explained.
• The fix uses the simplest mechanism that closes the window, with a test that runs the operations concurrently.
• Data damaged by the bug is listed, with a repair plan for the person to approve.