Design a Praxa evaluation

​

1. Name the decision

• Should we enable resumable event consumption for this workflow?
• Does the new retry policy reduce unresolved tasks without unacceptable cost?
• Can the integration meet a p95 latency target in our staging topology?
• Does the provider effect match the approved action in every trial?

​

2. Define the evidence class

​

3. Build a task matrix

​

4. Freeze the treatment

• source revision and dependency lock;
• model, provider, reasoning settings, and prompts;
• tool registry and capability policy;
• environment image and deployment topology;
• token, step, elapsed-time, and concurrency budgets;
• task order, randomization, warmup, and cache policy;
• grader version and success rubric;
• exclusions, retries, and stop rules.

​

5. Choose metrics

1. Safety vetoes — cross-tenant disclosure, unapproved effect, secret leak, false success, or failed cleanup.
2. Primary outcome — task success or independently verified effect.
3. Reliability outcomes — unresolved rate, parse errors, exact replay, reconciliation success.
4. Resource outcomes — latency, tokens, steps, requests, and cost.
5. Diagnostic outcomes — failure class, tool choice, retry count, and event reconnects.

​

6. Define evidence records

{
  "trialId": "trial-001",
  "taskId": "approval-browser-01",
  "arm": "candidate",
  "sourceRevision": "<commit>",
  "modelBinding": "<provider/model/settings>",
  "startedAt": "<UTC timestamp>",
  "terminalStatus": "completed",
  "graderOutcome": "passed",
  "providerEffect": {
    "expectedDigest": "<redacted digest>",
    "observed": true,
    "verifier": "provider-readback"
  },
  "resources": {
    "elapsedMs": 0,
    "inputTokens": 0,
    "outputTokens": 0,
    "steps": 0
  },
  "exclusion": null
}

​

7. Run a pilot, then freeze the confirmatory protocol

1. correct the protocol;
2. freeze the task set or sampling procedure;
3. freeze analysis code and thresholds;
4. run the confirmatory study once;
5. publish missing and excluded trials;
6. retain raw redacted records for review.

​

8. Report all outcomes

• counts before rates;
• per-task outcomes and aggregates;
• unresolved and invalid trials;
• safety vetoes;
• resource use;
• confidence or uncertainty appropriate to the design;
• deviations from the protocol;
• limitations and unsupported claims.

​

Acceptance template

Decision: Enable the candidate for 10% of staging traffic.

Safety vetoes:
- zero cross-tenant disclosure
- zero unapproved provider effects
- zero secret leakage
- zero false terminal success

Primary threshold:
- at least 90% independently verified effect success across the frozen sample

Resource threshold:
- p95 terminal latency under the stated budget in the named topology

Inconclusive rule:
- any missing verifier record or environment drift blocks the decision

Rollback:
- restore the previous artifact and preserve run IDs, cursors, and idempotency records