Benchmark data and reproduction

The charts in these docs are generated from a checked, machine-readable summary of the public preprint-v1.1.0 artifacts.

Trace the evidence

Artifact	Purpose
Preprint release	Versioned publication boundary
Paper data directory	Public evidence inputs
Local summary	Values used by this docs build
Outcomes chart	Trial-result visualization
Resource chart	Token, step, verification, and rollback comparison
Coverage chart	Repository-local coverage visualization

Regenerate the docs artifacts

From the docs repository:

npm run docs:generate
npm run docs:generate:check
npm run docs:verify

The check mode fails if a generated product guide, service-status page, or benchmark chart differs from its checked data source.

Reproduce the pilot

Follow the protocol and environment instructions in the public benchmark repository. Record:

exact source and package revisions;
model and provider configuration;
task and attempt inventory;
budgets and stop conditions;
environment image and cache state;
every exclusion and missing trial;
raw outcomes before aggregate calculation.

Do not compare a new run to the published pilot unless the comparison conditions are compatible. A reproduction with a different model or environment is still useful, but it is a new experiment.

Never put provider credentials, customer data, private traces, or secret prompts in a public reproduction artifact.

Last modified on August 14, 2026

Methods

Performance and reproduction

Benchmark data and reproduction

Trace the evidence

Regenerate the docs artifacts

Reproduce the pilot

​Trace the evidence

​Regenerate the docs artifacts

​Reproduce the pilot

Trace the evidence

Regenerate the docs artifacts

Reproduce the pilot