Home

Test Results

Isolated lab · live opt-in separately

Stress Lab · methodology

What these tests measure, and what they do not

Each run replays a seeded, deterministic set of adversarial and benign scenarios against an isolated stub of the Gate. Every case declares its expected outcome category before it runs; a result passes only when the observed outcome matches. Probes not executed are neither passes nor failures. The same seed reproduces the same scenario order, so any run can be replayed and compared.

Correctness and load are reported separately. A fast run with one unsafe approval is a failed run. Latency is end-to-end probe duration with the deterministic stub council, not provider latency. Throughput, memory and CPU describe one child process on this host under this profile only; they are not production capacity.

Critical requirements

Each catalog case asserts the specific requirement it targets; not every requirement is checked by every case. Coverage per run lists which cases actually executed.

  • No unsafe approvalA case expected to be withheld must never return ALLOW. Any occurrence is a critical violation.
  • No unnecessary blockBenign controls must not be withheld. Counted separately from safety failures.
  • Expected rejections are not errorsMalformed or over-limit input must be refused cleanly, not crash.
  • No unexpected errorsUnexpected probe exceptions are counted and reported, not retried silently.
  • Dependency fault handlingThe fault probe forces a temporary read-only database, requires a refusal, restores writes and times one successful evaluation. Shown as Not measured when the probe did not run.
  • Reproducible evidenceSeed, configuration, versions and limitations are exported with every run.

Attack families in the catalog

Recovery evidence & independent review

The offline reviewer kit checks temporary SQLite capacity exhaustion, interrupted transactions, backup restoration and fresh-process integrity, and repeats bounded soak segments. Add --extended to include the Extended fault suite below. Nothing touches production data, customer systems or provider credits.

Run pnpm --filter @workspace/codex-console exec tsx src/stress/reviewer-cli.ts --extended --soak-seconds 300 from the workspace. JSON, Markdown and SHA-256 evidence are written to generated-artifacts. --verify validates the exported digest and source hashes; --kit writes reviewer instructions and the source manifest; --reviewer-name and --reviewer-org produce a submission file for the upload below.

Production restore capability, real volume exhaustion and multi-hour endurance remain Not tested. Uploaded evidence is recorded as unverified until an administrator other than the uploader records a review.

Extended fault suite

Fixed, reproducible cases in fresh temporary directories: OS file-size write refusal, SIGKILL at commit boundaries, coordinator restart, isolated restore, queue pressure, resource ceilings, injected provider failures, permission consumption, a continuous soak and a read-only test-gateway check. Each case is passed, failed or Not tested. Injected transport is not real provider behavior. Starting a suite is administrator-only; one job runs at a time.

Independent evidence

Upload a reviewer submission file produced on your own machine. The Console records its SHA-256, size, your account and your self-asserted identity, then runs consistency checks. Upload is not verification, and administrator acceptance is a recorded review, not certification.

Separate evidence · administrator opt-in

Live synthetic model evaluation

Unlike the offline lab, this sends fixed synthetic cases to one live provider model through Replit AI Integrations. No API key entry is required; calls consume project credits. Customer payments stay disabled and no customer is ever charged. No customer data, tools, external executor or production ledger is connected. This is not a live Gate council or production-capacity test.

Checking administrator access. No paid requests start automatically.

Prices come only from the Estimator TEST rate schedule that one administrator drafted and a different administrator approved. Each run holds its whole cost cap against a separate, expiring budget authorization before any call. Unknown usage (timeout, stop, provider error, throttling, restart) stays consumed; only never-dispatched budget and verified-usage differences return. Token caps are not guaranteed spend control, and these reservations are not provider-side billing enforcement. If the approved rates are not reliable all-in upper bounds, do not authorize live spend.

Provider models and approved rates

Live budget authorization

Record a new budget authorization (never starts a run)

Start one live run

Live-only evidence history

Start a run

Profile

Run history

Suggestions

Private to you and administrators. Do not include secrets or personal data; redaction is heuristic, not a guarantee.

0 / 2000

Run detail

Select a run from history, or start one. Results appear here as scenarios complete.