Read outcomes before percentages
The pilot reports 36 trials per arm:
The aggregate difference is zero percentage points. Do not describe this as an
accuracy improvement, regression, or tie across all possible tasks. It is an
equal aggregate count in this sample.
Keep failure classes visible
Collapsing unresolved and parse errors into one “failed” bucket hides useful engineering information:- passed: the trial met the benchmark grader’s success rule;
- unresolved: the system did not produce a graded solution within the protocol;
- parse error: the trial output could not be interpreted by the evaluation pipeline.
Read resource use as a tradeoff
These values show that the reliability loop did more work in the pilot. They
do not prove production cost because provider pricing, caching, deployment,
concurrency, and task mix were not measured as a production workload.
Check experimental comparability
Before comparing arms, confirm:- Same task sample and attempt count.
- Same model, provider, reasoning configuration, and prompt.
- Same tools and environment image.
- Same token, step, wall-clock, and concurrency budgets.
- Same grader and outcome definitions.
- Same missing-trial and exclusion policy.
- Randomized or interleaved arm order when timing matters.
- Compatible cache and warmup state.
Ask what uncertainty is absent
The publication reports a small pilot, not a powered confirmatory study. It does not publish an independent replication or a causal confidence interval for a general population of tasks. Avoid using decimal precision to imply certainty the design does not support. For your own study, predefine:- the task population you want to generalize to;
- the minimum meaningful effect;
- the number of independent tasks and attempts;
- the paired or unpaired analysis;
- the treatment of missing and invalid trials;
- the stopping rule;
- the correction for repeated comparisons.
Separate admission, completion, and effect
These are different dependent variables. A run can be admitted and fail, can finish without producing the intended provider effect, or can produce an effect that does not improve the user’s outcome.Review checklist
When someone shares a result, ask:Language to use
Supported: “In this 12-task pilot, both arms passed 17 of 36 trials; the reliability-layer arm used more tokens and steps.” Not supported: “Praxa makes agents more accurate,” “Praxa is production proven,” or “the reliability layer is faster.”Methodology
Review the published controls, threats to validity, and comparison
requirements in detail.