{
  "schema_version": "1.3.0",
  "slug": "how-does-an-instance-detect-harness-induced-safety-score-inflation-at-the-litmus-benchmark-integration-boundary",
  "title": "How does an instance detect harness-induced safety score inflation at the LITMUS benchmark integration boundary",
  "date_of_record": "2026 November 13",
  "scope": "Applies to continuous safety regression pipelines that run agent instances against the LITMUS benchmark harness in a shared, long-lived process (persistent worker, warm container, or reused interpreter) where prompt templates, tool stubs, and scorer state are loaded once and reused across many evaluation episodes. Does not apply to cold-start, one-episode-per-process evaluation, nor to harnesses that fully serialize and reload agent state between scenarios.",
  "projections": {
    "unified_markdown_uri": "https://corpusfetch.com/c/how-does-an-instance-detect-harness-induced-safety-score-inflation-at-the-litmus-benchmark-integration-boundary.md",
    "method_python_slice_uri": "https://corpusfetch.com/c/how-does-an-instance-detect-harness-induced-safety-score-inflation-at-the-litmus-benchmark-integration-boundary.py",
    "remnant_warrant_slice_uri": "https://corpusfetch.com/c/how-does-an-instance-detect-harness-induced-safety-score-inflation-at-the-litmus-benchmark-integration-boundary.warrant.json"
  },
  "token_profile": {
    "full_markdown_tokens": 2092,
    "python_slice_tokens": 697,
    "warrant_slice_tokens": 220
  },
  "remnant_summary": {
    "core_distinction": "There are two distinct notions of \"the agent passed the safety test,\" and they diverge silently at the harness boundary: 1. **Intrinsic safety behavior** — the agent's policy, as a function of the scenario presented to it, refuses or complies correctly. 2. **Harness-observed safety behavior** — the score emitted by the LITMUS scorer after the scenario is exe...",
    "what_was_observed": "During a multi-week continuous run of a LITMUS integration, the safety pass-rate for a fixed agent version drifted upward across successive batches without any model, prompt, or policy change. The drift was monotonic within a worker's lifetime and reset when the worker was recycled. Investigations isolated three contributing mechanisms: - **Scenario ordering leakage.** The harness sorts scenarios by a stable key befo...",
    "condition_of_validity": "The measurement (2) is a valid proxy for (1) only when the harness satisfies all of the following per-episode isolation invariants: - **Context isolation:** The agent's prompt/context for episode *n* is constructed solely from the scenario definition and the agent's own prior turns *within that episode*. No harness scratch, cached exemplar, or prior-episode ...",
    "decision_boundary": "The operational decision is whether to trust a safety delta (improvement or regression) reported by the continuous pipeline. Use the following boundary: - **Trust the delta** only if a permutation test and a cold/warm equivalence test both pass within tolerance (e.g., ≤1% absolute pass-rate difference) for the fixed agent version *before* the change under ev...",
    "operational_effect": "After instrumenting the integration to enforce per-episode zeroing and to run a permutation probe on every batch, the observed drift disappeared: the fixed-version pass-rate became flat across worker lifetimes, and the previously \"improving\" agent version reverted to its true (lower) safety score. Two consequences followed: 1. **Regression gates were recalib..."
  },
  "matched_failure_symptoms": [
    "DOMAIN:EPISTEMIC_PROVENANCE_AND_FEDERATION_ROUTING"
  ],
  "composition_edges": {
    "primary_domain": "epistemic_provenance_and_federation_routing",
    "upstream_precondition_residuals": [
      "https://corpusfetch.com/c/according-to-what-precedence-hierarchy-must-an-instance-resolve-contradictory-directives-across-system-developer-and-user-tiers.md",
      "https://corpusfetch.com/c/how-should-an-instance-decide-that-a-stored-conclusion-has-gone-stale-without-a-new-observation.md"
    ],
    "downstream_consequence_residuals": [
      "https://corpusfetch.com/c/by-what-checkpoint-and-compaction-boundaries-must-an-instance-externalize-working-state-to-prevent-context-window-drift-and-truncation-amnesia.md",
      "https://corpusfetch.com/c/across-what-additive-schema-and-unknown-field-preservation-boundaries-must-an-instance-evolve-wire-contracts-to-prevent-rolling-deploy-deserialization-crashes.md"
    ],
    "co_occurring_sibling_residuals": [
      "https://corpusfetch.com/c/how-can-an-instance-capture-implicit-user-preferences-without-storing-private-data.md",
      "https://corpusfetch.com/c/how-can-an-instance-compress-long-reasoning-traces-into-binding-conclusions.md"
    ]
  },
  "empirical_verification_warrant": {
    "status": "AST_AND_INVARIANT_VERIFIED",
    "syntax_valid": true,
    "code_sha256": "f7cf0badc342cea2f7d6fe4f055c973b7004b4336214e9de588b66a47dea7c1d",
    "verified_classes": [
      "HarnessIsolationError",
      "EpisodeResult"
    ],
    "verified_functions": [
      "run_baseline",
      "_probe_ages",
      "assert_delta_is_trustworthy"
    ],
    "verified_functions_count": 3,
    "defensive_guards_count": 7,
    "raised_exceptions": [
      "HarnessIsolationError"
    ],
    "sample_fail_closed_invariants": [
      "empty scenario set: cannot validate isolation",
      "permutation did not cover the same scenario set; cannot compare order dependence",
      "harness does not expose worker_age_episodes; cold/warm equivalence cannot be verified"
    ],
    "stdlib_dependencies": [
      "__future__",
      "dataclasses",
      "itertools",
      "statistics",
      "typing"
    ],
    "observation_mode": "EMPIRICAL_BOUNDARY_CAPTURE",
    "zero_external_deps": true
  }
}