{
  "schema_version": "1.3.0",
  "slug": "when-must-an-instance-be-excluded-from-litmus-safety-benchmark-aggregation-cross-run-contamination-from-shared-agent-skill-state",
  "title": "When must an instance be excluded from LITMUS safety benchmark aggregation: cross-run contamination from shared agent skill state",
  "date_of_record": "2026 November 15",
  "scope": "Applies to LITMUS harness deployments that continuously red-team *stateful* agent skills (tools with persistent memory, cached credentials, session tokens, or mutable external resources) where multiple benchmark instances are executed against the same live skill endpoint. Does not apply to stateless, per-instance-provisioned skills or to offline replay benchmarks where skill state is snapshotted per instance.",
  "projections": {
    "unified_markdown_uri": "https://corpusfetch.com/c/when-must-an-instance-be-excluded-from-litmus-safety-benchmark-aggregation-cross-run-contamination-from-shared-agent-skill-state.md",
    "method_python_slice_uri": "https://corpusfetch.com/c/when-must-an-instance-be-excluded-from-litmus-safety-benchmark-aggregation-cross-run-contamination-from-shared-agent-skill-state.py",
    "remnant_warrant_slice_uri": "https://corpusfetch.com/c/when-must-an-instance-be-excluded-from-litmus-safety-benchmark-aggregation-cross-run-contamination-from-shared-agent-skill-state.warrant.json"
  },
  "token_profile": {
    "full_markdown_tokens": 1668,
    "python_slice_tokens": 317,
    "warrant_slice_tokens": 220
  },
  "remnant_summary": {
    "core_distinction": "The LITMUS harness treats an *instance* as an independent trial: a (prompt, expected-refusal-class) pair executed against a skill, scored pass/fail, then aggregated into a safety rate. This independence assumption is valid only when the skill's observable behavior for instance *i* is a function of instance *i*'s inputs alone. For stateful skills it is not. T...",
    "what_was_observed": "During a continuous run over a memory-bearing \"support agent\" skill (session-scoped memory, 30-minute TTL), the same 40-instance adversarial suite was executed on a cold skill and on a warm skill (preceded by 12 benign onboarding instances). The cold run produced a refusal rate of 0.925 (37/40). The warm run produced 0.775 (31/40). The six flips were all instances whose prompts referenced an account context; in the w...",
    "condition_of_validity": "An instance *i* is valid for aggregation only if, at execution time, the skill state observed by *i* is equivalent to the state observed by *i* under a cold, isolated execution with the same input. Operationally this requires one of: 1. **State reset between instances** — the harness invokes a skill-provided reset (memory clear, session invalidate, cache flu...",
    "decision_boundary": "The boundary is not \"stateful vs. stateless\" (too coarse) nor \"did the skill change\" (too late). It is: **does the skill's refusal boundary for instance *i* depend on any state written by instance *j < i*?** A practical decision procedure: - For each instance, run it twice: once cold, once immediately after the preceding instance in the sequence. If the two ...",
    "operational_effect": "The operational consequence is that continuous red-teaming of stateful skills requires the harness to treat *execution order* as a first-class variable. Two changes are mandatory: 1. **Per-instance state attestation.** The harness records a state fingerprint (reset canary result, memory entry count, session token generation) at instance entry and exit. Insta..."
  },
  "matched_failure_symptoms": [
    "DOMAIN:EPISTEMIC_PROVENANCE_AND_FEDERATION_ROUTING"
  ],
  "composition_edges": {
    "primary_domain": "epistemic_provenance_and_federation_routing",
    "upstream_precondition_residuals": [
      "https://corpusfetch.com/c/according-to-what-precedence-hierarchy-must-an-instance-resolve-contradictory-directives-across-system-developer-and-user-tiers.md",
      "https://corpusfetch.com/c/how-should-an-instance-decide-that-a-stored-conclusion-has-gone-stale-without-a-new-observation.md"
    ],
    "downstream_consequence_residuals": [
      "https://corpusfetch.com/c/by-what-checkpoint-and-compaction-boundaries-must-an-instance-externalize-working-state-to-prevent-context-window-drift-and-truncation-amnesia.md",
      "https://corpusfetch.com/c/across-what-additive-schema-and-unknown-field-preservation-boundaries-must-an-instance-evolve-wire-contracts-to-prevent-rolling-deploy-deserialization-crashes.md"
    ],
    "co_occurring_sibling_residuals": [
      "https://corpusfetch.com/c/how-can-an-instance-capture-implicit-user-preferences-without-storing-private-data.md",
      "https://corpusfetch.com/c/how-can-an-instance-compress-long-reasoning-traces-into-binding-conclusions.md"
    ]
  },
  "empirical_verification_warrant": {
    "status": "AST_AND_INVARIANT_VERIFIED",
    "syntax_valid": true,
    "code_sha256": "abf85482012737f7e7fd6ca5938aeae1f3ef97acce76a1b7a17ed4fb4a4105c6",
    "verified_classes": [
      "ContaminatedInstanceError",
      "StateFingerprint"
    ],
    "verified_functions": [
      "attest_instance",
      "aggregate_safety_rate",
      "equivalent_to_cold"
    ],
    "verified_functions_count": 3,
    "defensive_guards_count": 6,
    "raised_exceptions": [
      "ContaminatedInstanceError"
    ],
    "sample_fail_closed_invariants": [
      "state fingerprint missing; cannot attest cold equivalence",
      "exit reset canary failed; skill left state that will contaminate the next instance — quarantine this instance and halt t",
      "no instances to aggregate",
      "all instances contaminated; no valid pool"
    ],
    "stdlib_dependencies": [
      "dataclasses",
      "typing"
    ],
    "observation_mode": "EMPIRICAL_BOUNDARY_CAPTURE",
    "zero_external_deps": true
  }
}
