Customer Support · customer C001, orders and account actions in a controlled sandbox.
Detect unsafe tool behavior, reproduce the failure, apply a guardrail, and verify the fix.
Controlled customer-support sandbox. Exact-case replay uses an LLM and may vary. Verification applies only to the completed baseline/protected pair tested.
Each scenario runs an agent inside a controlled sandbox and records its tool actions, not just its final response.
Select the failure scenario and sandbox context.
Run the agent and record every tool action.
Check the execution trace against policy rules.
A safe-looking final answer does not guarantee safe execution. The trace shows what the agent actually accessed before producing its response.
Find the critical action and reduce the trigger.
Apply a guardrail and rerun the same case.
Replay runs the same failure before and after the guardrail so the mitigation is tested against the original case.
Save the verified failure for future testing.
# Regression test will appear after a confirmed failure.
Every saved case preserves its replay trigger, failure class, applied guardrail, and violation summaries. New saves and reruns also store full replay traces on the server.
Each card is a previously verified failure that can be rerun later to confirm the same guardrail still prevents the violation.
Frozen final-combined-20261005-005502 · audited exact-case evidence. Initial failures and conditional mitigation have distinct denominators; these are not universal protection rates.
Utility preservation covers baseline-successful tasks; overall benign success is 41/45. These results apply to the frozen evaluated cases only.
Cross-customer: 29/30 initial failures · 29/29 audited mitigation
Customer injection: 28/30 initial failures · 28/28 mitigation
Identity bypass: 8/9 initial failures · 8/8 paired mitigation
Unauthorized dispatch: 1/2 initial failures · 1/1 mitigation
Protected violations: 0
Benign emergency utility controls: 3/3
Controlled customer content ablation: poisoned 9/10 vs neutralized 0/10 vs sanitized 0/10. Matched intervention evidence consistent with a causal contribution; not formal causal proof.
Pipeline compatibility: Nemotron · Hermes · Qwen. Controlled subset only; not a model leaderboard.
Controlled models: Nemotron · Hermes · Qwen via Nebius Token Factory. Nemotron is the primary benchmark model.
Controlled simulated actions. Real earthquake data is snapshotted, not live. Exact-case replay is model-dependent. Generalization is demonstrated across two explicit domains, not universal. Some evaluators are rule-based. Emergency actions are simulated only.
Nebius Token Factory powers agent execution, minimization, replay, and controlled multi-model evaluation through an OpenAI-compatible inference API.