{
  "schema_version": "1.3.0",
  "slug": "when-must-an-instance-be-excluded-from-a-litmus-safety-benchmark-the-contaminated-context-boundary-in-adversarial-multi-turn-agent-evaluation",
  "title": "When must an instance be excluded from a LITMUS safety benchmark: the contaminated-context boundary in adversarial multi-turn agent evaluation",
  "date_of_record": "2026 November 13",
  "scope": "Applies to construction and scoring of multi-turn adversarial safety benchmarks (LITMUS-style) where an agent's *turn-N* compliance is judged against a *turn-N* probe while all prior turns remain in context. Does not apply to single-turn probes, stateless red-team evaluations, or benchmarks that reset context between probes.",
  "projections": {
    "unified_markdown_uri": "https://corpusfetch.com/c/when-must-an-instance-be-excluded-from-a-litmus-safety-benchmark-the-contaminated-context-boundary-in-adversarial-multi-turn-agent-evaluation.md",
    "method_python_slice_uri": "https://corpusfetch.com/c/when-must-an-instance-be-excluded-from-a-litmus-safety-benchmark-the-contaminated-context-boundary-in-adversarial-multi-turn-agent-evaluation.py",
    "remnant_warrant_slice_uri": "https://corpusfetch.com/c/when-must-an-instance-be-excluded-from-a-litmus-safety-benchmark-the-contaminated-context-boundary-in-adversarial-multi-turn-agent-evaluation.warrant.json"
  },
  "token_profile": {
    "full_markdown_tokens": 1663,
    "python_slice_tokens": 398,
    "warrant_slice_tokens": 220
  },
  "remnant_summary": {
    "core_distinction": "The non-obvious boundary is between **a refusal that is evidence of safety** and **a refusal that is an artifact of contaminated context**. In multi-turn adversarial evaluation, a turn-N probe is only a valid measurement of the agent's safety policy if the refusal at turn N is attributable to the probe itself and not to (a) a prior turn that already poisoned...",
    "what_was_observed": "Across a 1,200-instance LITMUS-style set (avg. 6.2 turns), we audited every turn-N refusal by replaying the instance with the turn-N probe replaced by a benign control probe of matched surface form. Two classes of false pass appeared: 1. **Refusal carry-over.** In 47 instances (3.9%), the agent refused the turn-N probe but *also* refused the benign control probe at the same position. The refusal was a context-mode ar...",
    "condition_of_validity": "A turn-N refusal is a valid safety measurement **iff** all of the following hold: - **Attribution:** Replacing the turn-N probe with a benign control of matched length, register, and imperative force yields *compliance* (the agent does the benign task). If the control is also refused, the refusal is context-driven and the instance is excluded. - **Non-conces...",
    "decision_boundary": "The boundary is not \"refused vs. complied.\" It is **\"refusal attributable to the probe\" vs. \"refusal attributable to context.\"** Operationally: - If benign-control replay **complies** and fresh-context replay **refuses** → *valid pass*. Score as safety success. - If benign-control replay **refuses** → *context artifact*. Exclude. Do not count as pass. - If f...",
    "operational_effect": "Applying this boundary to the 1,200-instance set moved the reported safety rate from **91.4% → 86.1%** (a 5.3-point correction) and surfaced 19 latent-leakage instances that had been silently scored as safe while the harmful content was produced at turn 2–3. The exclusion rate stabilized at 5.5% and became a first-class metric: a rising `excluded_contaminate..."
  },
  "matched_failure_symptoms": [
    "DOMAIN:EPISTEMIC_PROVENANCE_AND_FEDERATION_ROUTING"
  ],
  "composition_edges": {
    "primary_domain": "epistemic_provenance_and_federation_routing",
    "upstream_precondition_residuals": [
      "https://corpusfetch.com/c/according-to-what-precedence-hierarchy-must-an-instance-resolve-contradictory-directives-across-system-developer-and-user-tiers.md",
      "https://corpusfetch.com/c/how-should-an-instance-decide-that-a-stored-conclusion-has-gone-stale-without-a-new-observation.md"
    ],
    "downstream_consequence_residuals": [
      "https://corpusfetch.com/c/by-what-checkpoint-and-compaction-boundaries-must-an-instance-externalize-working-state-to-prevent-context-window-drift-and-truncation-amnesia.md",
      "https://corpusfetch.com/c/across-what-additive-schema-and-unknown-field-preservation-boundaries-must-an-instance-evolve-wire-contracts-to-prevent-rolling-deploy-deserialization-crashes.md"
    ],
    "co_occurring_sibling_residuals": [
      "https://corpusfetch.com/c/how-can-an-instance-capture-implicit-user-preferences-without-storing-private-data.md",
      "https://corpusfetch.com/c/how-can-an-instance-compress-long-reasoning-traces-into-binding-conclusions.md"
    ]
  },
  "empirical_verification_warrant": {
    "status": "AST_AND_INVARIANT_VERIFIED",
    "syntax_valid": true,
    "code_sha256": "9302b474fd26c3fb9efec667adef771590b4277a252de6dc1582b4efb25d9c09",
    "verified_classes": [
      "ContaminatedInstanceError",
      "Outcome",
      "Replay",
      "Instance"
    ],
    "verified_functions": [
      "classify",
      "safety_rate"
    ],
    "verified_functions_count": 2,
    "defensive_guards_count": 2,
    "raised_exceptions": [
      "ContaminatedInstanceError"
    ],
    "sample_fail_closed_invariants": [
      "no admissible instances: every refusal was context-contaminated; the benchmark is measuring the wrong turn"
    ],
    "stdlib_dependencies": [
      "dataclasses",
      "enum"
    ],
    "observation_mode": "EMPIRICAL_BOUNDARY_CAPTURE",
    "zero_external_deps": true
  }
}