Executable bug reproductions

Executable Reproduction Tests

When to Use

Use whenever a reproduced bug can be expressed as an automated test case (unit, integration, API, or Playwright/E2E) or a self-contained reproduction script using the project's existing test runner.

Properties of a High-Signal Reproduction Test

1. Fails for the Right Reason (Target-Specific Assertion)

A reproduction test is useless if it fails because of a missing mock, a syntax error, an unseeded database table, or a generic timeout instead of the actual product defect:

• Verify Code-Path Execution: Confirm the test actually reaches the defective function, SQL query, or UI handler before failing.
• Assert the Specific Expected Contract:
  • Do not write weak assertions like expect(() => run()).not.toThrow() when the bug is a wrong return value, or expect(res.status).toBe(500) (which cements the buggy behavior!).
  • Write the assertion for the correct expected behavior (expect(res.status).toBe(200) and expect(body.total).toBe(150)), so the test fails today with the exact bug signature and will pass unmodified once the fix is applied.

2. Self-Contained Setup & Teardown

• Seed only the minimal fixture data required by the reproduction inside the test (beforeEach / factory helper / temp directory).
• Never depend on external network availability, live third-party credentials, or leftover state from previous test files.
• Clean up all temporary files, timers, child processes, and database records in afterEach / try ... finally.

3. Deterministic Control Over Time, Randomness & Concurrency

Eliminate flakiness in reproduction tests:

• Time-Dependent Bugs: Use fake timers or injected clocks (vi.setSystemTime(...), sinon.useFakeTimers, frozen Clock interface) rather than sleep(2000).
• Race Conditions: Use explicit synchronization barriers (Deferred promises, deterministic request interceptors, controlled stream chunks) rather than hoping setTimeout(..., 10) hits the race window.
• Intermittent Bugs: If a bug remains probabilistic (p < 1), run a bounded loop or stress harness with fixed seeds and report the exact failure rate (k failures in N runs).

4. Non-Intrusive Instrumentation Rule

• Never silently mutate production application code to force a test to fail.
• If temporary diagnostic logging or a test hook is needed to prove a timing window, keep it strictly isolated and clearly marked as diagnostic instrumentation to be removed.