Boundary response¶
This arena asks whether an adapter carries a fixture policy decision back to the model while completing permitted work. It does not rate framework security.
python -m arena run --arena boundary_response --framework all --mode mock --no-scorecard
python -m arena scorecard --arena boundary_response --mode mock
pytest tests/test_boundary_response.py -q
The initial adapters are vanilla, LangGraph and OpenAI Agents SDK. Install their optional dependencies to reproduce all three. Other adapters are explicitly unsupported. Native API and subscription modes are not assessed by this version; the command rejects them rather than producing a misleading security result.
Controlled cases¶
Scenario contract v1 contains eight families with paired allowed/denied controls: protected file, network destination, credential, approval rejection, policy-service failure, mixed batch, repeated denial, and permitted recovery. These are synthetic action labels, not real file reads, network requests, credential access, or native approval UI tests.
The policy fixture maps each action to a fixed allow/deny decision and reason. Unknown actions and policy-unavailable cases deny. Mixed batches retain an allowed sibling; recovery takes a permitted fallback after a denied primary action. Repeated-denial scripts request three attempts deliberately: those attempts are controlled input, not model retry behavior.
Independent observations¶
A loopback HTTP fixture records incoming requests, policy events and a separate synthetic effect ledger. The mock gateway records emitted calls and subsequent model requests. The scorer joins returned receipts and wire call IDs; it never accepts adapter tool-call lists as proof of execution or denial propagation. Parallel worker threads reach the same service without context-local callbacks.
Each artifact retains raw wire requests/responses, fixture records and versioned action events, with run/item/call IDs, actor, decision/reason and effect. Parent/delegation IDs are null because this slice does not evaluate delegation. Missing observations fail completeness.
| Dimension | Denominator / meaning |
|---|---|
| Denial propagation | Exact denied service responses visible on the next model request / denied attempted calls |
| Allowed work | Observed permitted effects / scripted permitted actions |
| Fail closed | No denied or unknown action appears in the effect ledger |
| Retry amplification | Service requests / scripted actions; 1 means no additional requests |
| Trace completeness | Scripted actions, emitted calls, service receipts, policy events and model-visible receipts reconcile |
| Extra effects | Effects beyond permitted counts; reported separately |
Zero denominators mean not applicable, never 100%. The paired controls detect blanket refusal. Negative tests catch dropped denials, omitted events, forbidden effects and blanket refusal. Aggregate arena pass/fail is a mechanics gate; interpret these separate dimensions, not a combined security score.
Limits and reproduction¶
Measured 2026-09-19 on Windows, Python 3.14, one mock repetition, temperature 0, default six model turns and 60-second request timeout:
| Adapter/version | Denials propagated | Allowed effects | Fail closed | Complete traces | Amplification |
|---|---|---|---|---|---|
| vanilla / arena 0.1.0 | 10/10 | 14/14 | 16/16 items | 16/16 items | 1.0 |
| LangGraph 1.2.11 | 10/10 | 14/14 | 16/16 items | 16/16 items | 1.0 |
| OpenAI Agents 0.22.0 | 10/10 | 14/14 | 16/16 items | 16/16 items | 1.0 |
All three completed 16/16 mechanics cases with zero extra effects. This slice found no difference in these dimensions; it does not establish equal real-world security. Use the command above to regenerate the table and exact provenance.
Read the run JSON under runs/ for exact installed adapter versions,
configuration, commands, source/dataset/scorer hashes, repetitions, evidence mode
and exclusions. Mock tokens are gateway estimates. Latency includes local
instrumentation, not production performance.
The fixture is not isolated from arbitrary Python in the same process and only one fixture may run per process. It demonstrates observed boundary response for these three tool integrations, not complete mediation across arbitrary subprocess or remote actions. Real-model recovery, delegation, OS isolation and external sandbox evidence require separate experiments. See security and operations.