Choose by question
Evidence ladder
Higher lanes answer broader questions, but they do not invalidate the lower ones. A production observation still needs a pinned contract, and a conformance test still cannot predict user outcomes.Current public artifacts
Terminal-Bench pilot snapshot
The reliability-oriented arm used more tokens and steps while producing the
same aggregate pass count. The study supports failure analysis and design
questions for this sample, not a claim that either arm is generally superior.
Select the next guide
Interpret results
Read counts, rates, resources, uncertainty, missing trials, and limitations
without overstating the result.
Design an evaluation
Define the decision, task sample, controls, metrics, evidence, stop rules,
and acceptance threshold before running.
Reproduction workflow
Trace the versioned publication, regenerate checked docs artifacts, and
plan a compatible or intentionally different experiment.
Metrics glossary
Use consistent definitions for accuracy, unresolved trials, latency,
throughput, token use, effects, and evidence classes.