Skip to main content
Reproduction has two useful forms:
  • artifact reproduction regenerates tables and charts from the published evidence;
  • experimental reproduction reruns the protocol in a compatible environment.
The first checks analysis integrity. The second tests whether the observed result recurs.

​
Published lineage

The docs summary records:

​
1. Inspect the versioned release

Require a clean detached checkout. Read the paper, paper/data, and paper/protocols before attempting to run anything. Do not assume current default-branch scripts are identical to the tagged protocol.

​
2. Preserve an environment receipt

Record:
Store secret values outside the receipt. Record only the credential class, scope, and issuer required to interpret the run.

​
3. Regenerate documentation artifacts

From the docs repository:
docs:generate:check fails when a checked generated page or chart differs from its source JSON. This proves deterministic documentation generation, not an experimental rerun.

​
4. Decide compatibility

An intentionally different experiment is valuable. Label it accurately and do not combine its aggregate with the pilot.

​
5. Run with fail-closed collection

Before the full run, prove the collector records:
  1. every scheduled trial;
  2. terminal status and grader outcome;
  3. unresolved and parse-error states;
  4. missing trials explicitly;
  5. token, step, and elapsed-resource fields;
  6. exclusions with reasons;
  7. provider readback when effect claims are made;
  8. environment and source fingerprints.
If the model or provider quota becomes unavailable for one arm, stop or mark the run incomplete. Do not compare a fully executed arm with a quota-truncated arm.

​
6. Validate raw-to-aggregate parity

For every reported metric:
  • recompute it from raw trial records;
  • verify the denominator;
  • confirm missing and excluded trials;
  • compare task-level and aggregate results;
  • reject metrics whose source metadata disagrees.
The published pilot omits pass-at-k because aggregate and raw metadata did not agree for one arm. Omission is the correct outcome when provenance is unresolved.

​
7. Publish a claim boundary

Include a plain-language statement such as:
This run measures the named systems on the recorded task sample and environment. It does not establish general superiority, production reliability, or provider-effect success outside the included verifier data.

​
Reproduction checklist

  • Versioned source and artifacts
  • Locked environment and dependency receipt
  • Exact task and attempt inventory
  • Model/provider/prompt settings
  • Budgets and stop rules
  • Cache, warmup, and arm-order policy
  • Raw outcomes including missing trials
  • Deterministic aggregation code
  • Secret and customer-data review
  • Independent reviewer
  • Limitations and unsupported claims

Published data and reproduction

Open the exact public artifacts and generated chart sources.

Design an evaluation

Create a new protocol when your model, tasks, treatment, or deployment differs from the pilot.
Last modified on August 14, 2026