Benchmark design examples

Use these examples as study templates, not as published Praxa results. Each example names the decision, unit of analysis, evidence, failures, and acceptance rule before any run begins.

Example 1: durable task reliability

Decision: Is a candidate release safe enough to admit, execute, and reconcile a fixed suite of durable tasks in staging?

Field	Frozen design
Unit	One logical task, including exact retries
Sample	30 fixed tasks across read-only, synthesis, and approval-bound classes
Attempts	3 independent attempts per task and release
Primary metric	Verified terminal outcomes / eligible trials
Failure classes	admission, authorization, timeout, execution, verification, unknown outcome, parse
Evidence	request digest, idempotency key hash, run ID, terminal projection, verifier record
Stop rule	Stop on cross-tenant disclosure, secret exposure, or unbounded execution

Record admission and outcome separately:

trial-record.json

{
  "trialId": "task-07-attempt-2",
  "release": "candidate-a",
  "taskClass": "approval-bound",
  "admitted": true,
  "terminalStatus": "completed",
  "effectStatus": "verified",
  "failureClass": null,
  "elapsedMs": 18420,
  "inputTokens": 2140,
  "outputTokens": 388
}

A 202 response counts only as admission. The success numerator changes only after terminal and effect evidence meet the frozen protocol.

Example 2: resumable event consumption

Decision: Does a consumer recover from disconnects without losing or double-applying lifecycle events?

Create one disposable run.
Persist events and the cursor only after the local transaction commits.
Disconnect after each selected sequence boundary.
Resume with the persisted Last-Event-ID.
Compare stored sequences with the authoritative run projection.
Repeat under slow-consumer, duplicate-delivery, and temporary-outage cases.

Measure:

missing unique sequences;
duplicate deliveries and duplicate side effects separately;
reconnect attempts and recovery time;
terminal projection mismatches; and
unreconciled outcomes.

Passing means every expected sequence is present, the application side effect is applied once per event identity, and terminal readback agrees with the event projection. It does not mean the network delivered each event exactly once.

Example 3: federated memory degradation

Decision: Does an agent retain useful, source-labelled recall when one read-only memory provider is unavailable? Build a fixture set with expected facts, contradictions, recency, and source lineage. Run the same queries against:

every configured source available;
each source timed out in turn;
malformed records from one source; and
every source unavailable.

Report both retrieval quality and operating state:

Metric	Why it matters
Recall@k	Expected relevant items retrieved within the result budget
Source coverage	Providers contributing at least one valid result
Provenance completeness	Results carrying required source identity and capture metadata
Contradiction visibility	Conflicting normalized statements retained for caller policy
Partial-result rate	Queries with at least one success and at least one failure
Failed-query rate	Queries with no successful source

The federation API returns partial only when at least one source succeeds and failed when none succeeds. Provider failures are result states; callers must inspect the aggregate and per-source statuses instead of relying on exceptions.

Example 4: governed agent tools

Decision: Does an agent host expose only tools allowed by the granted scope and require approval before a committing action? Create a matrix of principal, scope set, tool, operation class, approval state, and expected result. Include:

read-only tool with matching scope;
tool absent from the granted scope set;
committing tool before approval;
committing tool after approval;
expired or revoked token;
replay of the same logical mutation; and
foreign-tenant resource identifier.

Count a trial as passing only when registration, host selection, Praxa policy, and final readback all match the expected boundary. A model declining to call a tool is not evidence that the authorization gate would refuse it.

Turn a template into a study

Copy the closest example.
Replace the decision with the exact release or product choice.
Freeze task IDs, inputs, attempts, budgets, and exclusions.
Define machine-readable evidence records and a redaction review.
Run a small pilot to debug the harness.
Freeze the confirmatory protocol before viewing comparative results.
Publish counts, denominators, failures, resources, and limitations together.

Design the evaluation

Define the decision, evidence class, controls, metrics, stop rules, and acceptance threshold.

Metrics glossary

Keep outcome, reliability, performance, resource, and validation metrics distinct.

Methods

Performance and reproduction

Benchmark design examples

Example 1: durable task reliability

Example 2: resumable event consumption

Example 3: federated memory degradation

Example 4: governed agent tools

Turn a template into a study

Design the evaluation

Metrics glossary

​Example 1: durable task reliability

​Example 2: resumable event consumption

​Example 3: federated memory degradation

​Example 4: governed agent tools

​Turn a template into a study

Design the evaluation

Metrics glossary

Example 1: durable task reliability

Example 2: resumable event consumption

Example 3: federated memory degradation

Example 4: governed agent tools

Turn a template into a study