> ## Documentation Index
> Fetch the complete documentation index at: https://docs.praxa.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Praxa Benchmarks: Measuring Agentic Mission Performance
> The Praxa benchmarks suite measures mission latency, event streaming throughput, SDK retry performance, and gateway reliability under load.
The praxa-benchmarks suite provides reproducible performance measurements for the Praxa platform, helping you understand expected latency and throughput characteristics before production deployment. Because Praxa operates as a credential-free agentic mission platform — with the Integration Gateway enforcing tenant, consent, and budget validation on every request — accurate performance data helps you size resource budgets, tune retry parameters, and design mission fan-out patterns with confidence. All benchmark scenarios run against a real Integration Gateway using the `@praxa/sdk` client, so results reflect the full end-to-end path including token acquisition, contract validation, and SSE stream delivery.
## What We Measure
The benchmarks suite covers four categories of measurements, each targeting a distinct layer of the Praxa request lifecycle.
### Mission Lifecycle Latency
This category measures wall-clock time across the key state transitions of a single mission:
* **Time to first event** — the interval between calling `createMission()` and receiving the first `AuraSseEvent` from `missionEvents()`. This captures gateway acceptance latency, initial execution scheduling, and SSE stream setup overhead.
* **Time to `mission.completed`** — the full elapsed time from `createMission()` to the terminal event on the SSE stream. This reflects the actual work duration for a given `goalSpec` and `resourceBudget` combination.
* **`getMission()` polling latency** — round-trip time for a status-check read on an in-flight or completed mission, which exercises the `GET /v8/missions/{runId}` safe-read path.
Each scenario uses a fixed, minimal `resourceBudget` so that results are dominated by platform overhead rather than task complexity.
### Event Stream Throughput
This category measures the rate at which `AuraSseEvent` objects are delivered through the SSE stream to the `missionEvents()` async generator. Praxa's SSE implementation uses a buffered UTF-8 line reader with a 1 MiB per-line and per-event ceiling; throughput measurements reveal how close to that ceiling a realistic mission workload comes. The benchmark emits synthetic missions with varying event frequencies to produce events-per-second figures at p50, p95, and p99 percentiles.
### SDK Retry Performance
This category quantifies the overhead introduced when the automatic retry mechanism activates on transient failures. The `@praxa/sdk` client retries requests on HTTP 408, 425, 429, 500, 502, 503, and 504 responses, preserving the idempotency key and request body unchanged across all attempts. Benchmarks measure:
* **Retry decision latency** — time from a retryable response to the first byte of the next attempt.
* **Backoff accuracy** — measured sleep duration versus the configured `retryBaseDelayMs × 2^(attempt−1)` target.
* **Total added latency** — cumulative delay budget consumed by retries before a successful response.
### Concurrent Mission Scaling
This category evaluates platform behavior when you submit multiple missions in parallel using `Promise.all`. Key metrics include:
* **Fan-out submission time** — wall-clock time to receive `MissionProjection` responses for N concurrent `createMission()` calls.
* **`maximumParallelism` interaction** — whether the gateway's per-mission `maximumParallelism` budget affects submission acceptance latency or only internal execution scheduling.
* **Idempotency key collision resistance** — verification that unique keys per parallel mission prevent cross-contamination of retry state.
## Benchmark Methodology
All benchmark scenarios follow a consistent methodology to ensure you can compare results across gateway versions and deployment environments.
**Test harness.** Benchmarks use the `@praxa/sdk` `PraxaClient` directly against a live Integration Gateway. No mock fetch adapters are used in benchmark mode — results reflect real network, TLS handshake, gateway validation, and execution overhead.
**Iteration count and statistics.** Each scenario runs a configurable number of warm-up iterations (discarded) followed by N measurement iterations. The benchmark harness reports p50, p95, and p99 latency percentiles, arithmetic mean, and standard deviation. Iteration counts default to 50 for latency benchmarks and 200 for throughput benchmarks to give stable tail percentiles.
**Resource budget isolation.** Every benchmark scenario specifies a fixed `ResourceBudget` — identical `maximumSteps`, `maximumToolCalls`, `maximumElapsedMs`, and `maximumParallelism` values — so that budget validation overhead is constant across runs. Scenarios that explore budget effects vary exactly one parameter at a time.
**Reproducibility constraints.** Because results depend on your gateway deployment, network path, and the complexity of each `goalSpec`, the benchmarks suite records the `PRAXA_BASE_URL`, SDK version (`AURA_OPENAPI_VERSION = "8.1.0"`), and contract fingerprint (`AURA_CONTRACT_VERSION = "aura-integration-gateway-v8.1"`) alongside every result file. Do not compare results across different contract versions or gateway deployments without accounting for those variables.
The praxa-benchmarks repository is under active development. Benchmark scenarios, harness configuration, and result schemas may change before the first stable release. Watch [praxa-labs/praxa-benchmarks](https://github.com/praxa-labs/praxa-benchmarks) on GitHub for release announcements.
## Running the Benchmarks
The `praxa-benchmarks` package is being developed and will be published to the `praxa-labs/praxa-benchmarks` GitHub repository. Once available, you can run the full suite against your own Integration Gateway deployment with the following steps.
```bash theme={null}
git clone https://github.com/praxa-labs/praxa-benchmarks
cd praxa-benchmarks
```
```bash theme={null}
npm install
```
Set `PRAXA_BASE_URL` to the HTTPS origin of your Integration Gateway and `PRAXA_ACCESS_TOKEN` to a short-lived delegated OAuth token with at least the `missions:write` and `missions:read` scopes. Provider credentials must not appear in these variables — the gateway holds those internally.
```bash theme={null}
export PRAXA_BASE_URL=https://your-gateway.example
export PRAXA_ACCESS_TOKEN=
```
```bash theme={null}
PRAXA_BASE_URL=https://your-gateway.example PRAXA_ACCESS_TOKEN= npm run bench
```
Results are written to `results/` as JSON files containing raw latency samples and computed percentile summaries.
The benchmark runner validates your `PRAXA_BASE_URL` as an HTTPS origin with no embedded credentials before issuing any requests — the same check performed by `PraxaClient` itself. Plain HTTP origins are rejected immediately.
## Interpreting Results
When you receive benchmark output, a few reference points help you judge whether your deployment is performing as expected.
**Mission lifecycle latency.** Time-to-first-event p50 values under 500 ms typically indicate a healthy gateway with no queue backlog. p99 values exceeding 2–3 s may point to cold-start scheduling delays or token validation overhead accumulating under load. If your time-to-first-event p99 is significantly higher than p50, consider whether your `accessToken` function is re-fetching a token on every call — caching the token for its remaining lifetime dramatically reduces that variance.
**Retry overhead.** If your p99 mission creation latency is an outlier compared to p50, check whether retries are activating. A single retry with `retryBaseDelayMs: 100` adds at least 100 ms plus network round-trip time. At the default `maximumAttempts: 3`, worst-case backoff reaches 100 + 200 + 400 = 700 ms of sleep time before the final attempt. If retry-induced tail latency is unacceptable for your use case, reduce `maximumAttempts` to 2 or 1 for latency-sensitive paths where you can tolerate handling failures in application code.
**Factors that affect results.** The following variables have the largest influence on benchmark numbers across deployments:
| Factor | Effect |
| ----------------------------------------- | ------------------------------------------------------------------------ |
| Network round-trip time to gateway | Directly adds to every latency measurement |
| Token acquisition time in `accessToken()` | Adds per-request overhead; mitigated by caching |
| Gateway deployment region | Affects both latency and capacity |
| `maximumElapsedMs` budget | Caps individual mission duration; does not affect submission latency |
| `maximumParallelism` | Affects internal scheduling; may affect concurrent-submission throughput |
| Mission `goalSpec` complexity | Drives time-to-completed; excluded from submission latency benchmarks |
**What good throughput looks like.** The SSE stream reader processes events as fast as your network delivers bytes. For missions that emit many fine-grained events, the bottleneck is typically network bandwidth rather than the SDK's buffered line reader. If you observe event delivery rates far below your expected mission update frequency, check whether a proxy or load balancer is buffering the SSE response body before forwarding it — many HTTP/1.1 intermediaries buffer `text/event-stream` responses by default.
Detailed guidance on SDK client overhead, retry backoff behavior, SSE stream memory footprint, and concurrent mission submission patterns.