Grade technical research evidence

Evidence Grading & Research Verification

Credible technical research never blurs marketing claims, code reading, live experiments, and personal inference. Every conclusion must carry its exact evidence grade.

The Four Evidence Tiers

Label every major capability or finding explicitly:

1. Claimed (Documentation / README / Marketing): Stated in docs, blog posts, or comments, but not yet verified in source code or execution.
2. Source-Verified (Code-Traced): Traced through the actual implementation at a specific version/commit (repo@commit:path#L10-L45), confirming the mechanism and edge-case handling.
3. Empirically Verified (Executed / Tested): Proven by running the project's test suite or a minimal, reproducible experiment in our target runtime environment.
4. Inferred (Researcher Judgment): A reasoned architectural conclusion or extrapolation — stated with its assumptions and what would falsify it.

Workflow

1. Verify decisive claims with the smallest possible experiment (Willison & Beck):
   • When a recommendation hinges on a critical claim (e.g., "supports streaming tool calls without buffering", "handles 50k nodes under 50 ms", "works in Node 22 ESM without native builds"), write and run a 20–50 line isolated verification script if the environment allows.
   • Preserve the exact command, input fixture, and observed output in the research report so any teammate can reproduce it in seconds.
2. Audit third-party benchmarks critically:
   • Never repeat a README benchmark chart at face value. Check the benchmark source code for:
     • Dataset size and realism, warm-up iterations, and whether competitors were configured fairly.
     • Hardware, concurrency model, and whether error checking or durability was disabled to inflate numbers.
3. Track open unknowns and verification triggers:
   • If an environment constraint prevents live testing (e.g., requires an external enterprise account or specific hardware), mark the item as Source-Verified / Untested at Runtime and define the exact verification step for phase 1 of implementation.
4. Stop when decision sufficiency is reached:
   • Once the evidence clearly separates the winning option from the alternatives and bounds the integration risks, conclude the research and deliver the recommendation.

Verification Checklist

• Every key recommendation cites primary source files and version/commit identifiers.
• Project claims, code-traced facts, executed tests, and inferences are clearly distinguished.
• No repository URLs, file paths, APIs, or benchmark results are fabricated or guessed.
• Remaining unknowns have a concrete, low-cost verification plan.