Skip to main content
The praxa-benchmarks suite provides reproducible performance measurements for the Praxa platform, helping you understand expected latency and throughput characteristics before production deployment. Because Praxa operates as a credential-free agentic mission platform — with the Integration Gateway enforcing tenant, consent, and budget validation on every request — accurate performance data helps you size resource budgets, tune retry parameters, and design mission fan-out patterns with confidence. All benchmark scenarios run against a real Integration Gateway using the @praxa/sdk client, so results reflect the full end-to-end path including token acquisition, contract validation, and SSE stream delivery.

​
What We Measure

The benchmarks suite covers four categories of measurements, each targeting a distinct layer of the Praxa request lifecycle.

​
Mission Lifecycle Latency

This category measures wall-clock time across the key state transitions of a single mission:
  • Time to first event — the interval between calling createMission() and receiving the first AuraSseEvent from missionEvents(). This captures gateway acceptance latency, initial execution scheduling, and SSE stream setup overhead.
  • Time to mission.completed — the full elapsed time from createMission() to the terminal event on the SSE stream. This reflects the actual work duration for a given goalSpec and resourceBudget combination.
  • getMission() polling latency — round-trip time for a status-check read on an in-flight or completed mission, which exercises the GET /v8/missions/{runId} safe-read path.
Each scenario uses a fixed, minimal resourceBudget so that results are dominated by platform overhead rather than task complexity.

​
Event Stream Throughput

This category measures the rate at which AuraSseEvent objects are delivered through the SSE stream to the missionEvents() async generator. Praxa’s SSE implementation uses a buffered UTF-8 line reader with a 1 MiB per-line and per-event ceiling; throughput measurements reveal how close to that ceiling a realistic mission workload comes. The benchmark emits synthetic missions with varying event frequencies to produce events-per-second figures at p50, p95, and p99 percentiles.

​
SDK Retry Performance

This category quantifies the overhead introduced when the automatic retry mechanism activates on transient failures. The @praxa/sdk client retries requests on HTTP 408, 425, 429, 500, 502, 503, and 504 responses, preserving the idempotency key and request body unchanged across all attempts. Benchmarks measure:
  • Retry decision latency — time from a retryable response to the first byte of the next attempt.
  • Backoff accuracy — measured sleep duration versus the configured retryBaseDelayMs × 2^(attempt−1) target.
  • Total added latency — cumulative delay budget consumed by retries before a successful response.

​
Concurrent Mission Scaling

This category evaluates platform behavior when you submit multiple missions in parallel using Promise.all. Key metrics include:
  • Fan-out submission time — wall-clock time to receive MissionProjection responses for N concurrent createMission() calls.
  • maximumParallelism interaction — whether the gateway’s per-mission maximumParallelism budget affects submission acceptance latency or only internal execution scheduling.
  • Idempotency key collision resistance — verification that unique keys per parallel mission prevent cross-contamination of retry state.

​
Benchmark Methodology

All benchmark scenarios follow a consistent methodology to ensure you can compare results across gateway versions and deployment environments. Test harness. Benchmarks use the @praxa/sdk PraxaClient directly against a live Integration Gateway. No mock fetch adapters are used in benchmark mode — results reflect real network, TLS handshake, gateway validation, and execution overhead. Iteration count and statistics. Each scenario runs a configurable number of warm-up iterations (discarded) followed by N measurement iterations. The benchmark harness reports p50, p95, and p99 latency percentiles, arithmetic mean, and standard deviation. Iteration counts default to 50 for latency benchmarks and 200 for throughput benchmarks to give stable tail percentiles. Resource budget isolation. Every benchmark scenario specifies a fixed ResourceBudget — identical maximumSteps, maximumToolCalls, maximumElapsedMs, and maximumParallelism values — so that budget validation overhead is constant across runs. Scenarios that explore budget effects vary exactly one parameter at a time. Reproducibility constraints. Because results depend on your gateway deployment, network path, and the complexity of each goalSpec, the benchmarks suite records the PRAXA_BASE_URL, SDK version (AURA_OPENAPI_VERSION = "8.1.0"), and contract fingerprint (AURA_CONTRACT_VERSION = "aura-integration-gateway-v8.1") alongside every result file. Do not compare results across different contract versions or gateway deployments without accounting for those variables.
The praxa-benchmarks repository is under active development. Benchmark scenarios, harness configuration, and result schemas may change before the first stable release. Watch praxa-labs/praxa-benchmarks on GitHub for release announcements.

​
Running the Benchmarks

The praxa-benchmarks package is being developed and will be published to the praxa-labs/praxa-benchmarks GitHub repository. Once available, you can run the full suite against your own Integration Gateway deployment with the following steps.
1

Clone the repository

2

Install dependencies

3

Configure your gateway and token

Set PRAXA_BASE_URL to the HTTPS origin of your Integration Gateway and PRAXA_ACCESS_TOKEN to a short-lived delegated OAuth token with at least the missions:write and missions:read scopes. Provider credentials must not appear in these variables — the gateway holds those internally.
4

Run the benchmark suite

Results are written to results/ as JSON files containing raw latency samples and computed percentile summaries.
The benchmark runner validates your PRAXA_BASE_URL as an HTTPS origin with no embedded credentials before issuing any requests — the same check performed by PraxaClient itself. Plain HTTP origins are rejected immediately.

​
Interpreting Results

When you receive benchmark output, a few reference points help you judge whether your deployment is performing as expected. Mission lifecycle latency. Time-to-first-event p50 values under 500 ms typically indicate a healthy gateway with no queue backlog. p99 values exceeding 2–3 s may point to cold-start scheduling delays or token validation overhead accumulating under load. If your time-to-first-event p99 is significantly higher than p50, consider whether your accessToken function is re-fetching a token on every call — caching the token for its remaining lifetime dramatically reduces that variance. Retry overhead. If your p99 mission creation latency is an outlier compared to p50, check whether retries are activating. A single retry with retryBaseDelayMs: 100 adds at least 100 ms plus network round-trip time. At the default maximumAttempts: 3, worst-case backoff reaches 100 + 200 + 400 = 700 ms of sleep time before the final attempt. If retry-induced tail latency is unacceptable for your use case, reduce maximumAttempts to 2 or 1 for latency-sensitive paths where you can tolerate handling failures in application code. Factors that affect results. The following variables have the largest influence on benchmark numbers across deployments: What good throughput looks like. The SSE stream reader processes events as fast as your network delivers bytes. For missions that emit many fine-grained events, the bottleneck is typically network bandwidth rather than the SDK’s buffered line reader. If you observe event delivery rates far below your expected mission update frequency, check whether a proxy or load balancer is buffering the SSE response body before forwarding it — many HTTP/1.1 intermediaries buffer text/event-stream responses by default.

Performance Characteristics

Detailed guidance on SDK client overhead, retry backoff behavior, SSE stream memory footprint, and concurrent mission submission patterns.