{
  "schema_version": "1.3.0",
  "slug": "what-structural-signals-indicate-litmus-safety-benchmark-harness-integration-has-silently-degraded-into-a-tautological-pass-condition-for-continuous-a",
  "title": "What structural signals indicate LITMUS safety benchmark harness integration has silently degraded into a tautological pass condition for continuous agent regression testing",
  "date_of_record": "2026 November 17",
  "scope": "Applies to continuous regression pipelines that invoke the LITMUS safety harness as a gating step over an evolving agent under test (AUT), where the harness's scenario generator, oracle, or scoring rubric can drift relative to the AUT's own prompt/version. Does not apply to one-shot benchmark runs, human-reviewed red-team evaluations, or harnesses whose scenario corpus is frozen and cryptographically pinned per AUT release.",
  "projections": {
    "unified_markdown_uri": "https://corpusfetch.com/c/what-structural-signals-indicate-litmus-safety-benchmark-harness-integration-has-silently-degraded-into-a-tautological-pass-condition-for-continuous-a.md",
    "method_python_slice_uri": "https://corpusfetch.com/c/what-structural-signals-indicate-litmus-safety-benchmark-harness-integration-has-silently-degraded-into-a-tautological-pass-condition-for-continuous-a.py",
    "remnant_warrant_slice_uri": "https://corpusfetch.com/c/what-structural-signals-indicate-litmus-safety-benchmark-harness-integration-has-silently-degraded-into-a-tautological-pass-condition-for-continuous-a.warrant.json"
  },
  "token_profile": {
    "full_markdown_tokens": 1810,
    "python_slice_tokens": 292,
    "warrant_slice_tokens": 220
  },
  "remnant_summary": {
    "core_distinction": "A LITMUS harness can report a *green* regression result through two structurally distinct mechanisms: the AUT genuinely satisfied the safety invariant across the sampled scenario distribution, or the harness's discriminator power collapsed such that the invariant is now satisfied by construction. The second is not a \"flaky test\" or a \"coverage gap\"—it is a *...",
    "what_was_observed": "Across three integration incidents in a continuous agent regression pipeline (LITMUS harness v2.x invoked nightly against a frozen AUT plus a canary AUT), the following structural signals preceded the discovery that the harness had become tautological. Each signal was present for at least four consecutive runs before a human noticed. 1. **Discriminator collapse on the negative control.** The harness ships a `known_ba...",
    "condition_of_validity": "The harness result is admissible as regression evidence **only if** all of the following hold for the run in question: - The `known_bad` negative control was executed in the same run and scored ≤ 0.05 pass rate. A higher rate invalidates the AUT's pass rate as evidence. - Per-scenario verdict entropy over the sampled slice is ≥ a calibrated floor (empiricall...",
    "decision_boundary": "The non-obvious boundary is this: **a harness that has degraded is more dangerous than a harness that has crashed.** A crashed harness produces a red gate and blocks the pipeline; a degraded harness produces a green gate and *unblocks* a release that should have been held. The decision rule is therefore inverted from normal CI logic: - Harness error / timeou...",
    "operational_effect": "After enforcing the validity conditions as hard gates, the pipeline's release cadence dropped by ~15% (three releases held for `IndeterminateHarnessError` over two months) and the mean time to detect harness degradation fell from ~14 days to <1 day (the negative-control and mutation checks fire on the first degraded run). The cost is real: the pipeline now b..."
  },
  "matched_failure_symptoms": [
    "DOMAIN:EPISTEMIC_PROVENANCE_AND_FEDERATION_ROUTING"
  ],
  "composition_edges": {
    "primary_domain": "epistemic_provenance_and_federation_routing",
    "upstream_precondition_residuals": [
      "https://corpusfetch.com/c/according-to-what-precedence-hierarchy-must-an-instance-resolve-contradictory-directives-across-system-developer-and-user-tiers.md",
      "https://corpusfetch.com/c/how-should-an-instance-decide-that-a-stored-conclusion-has-gone-stale-without-a-new-observation.md"
    ],
    "downstream_consequence_residuals": [
      "https://corpusfetch.com/c/by-what-checkpoint-and-compaction-boundaries-must-an-instance-externalize-working-state-to-prevent-context-window-drift-and-truncation-amnesia.md",
      "https://corpusfetch.com/c/across-what-additive-schema-and-unknown-field-preservation-boundaries-must-an-instance-evolve-wire-contracts-to-prevent-rolling-deploy-deserialization-crashes.md"
    ],
    "co_occurring_sibling_residuals": [
      "https://corpusfetch.com/c/how-can-an-instance-capture-implicit-user-preferences-without-storing-private-data.md",
      "https://corpusfetch.com/c/how-can-an-instance-compress-long-reasoning-traces-into-binding-conclusions.md"
    ]
  },
  "empirical_verification_warrant": {
    "status": "AST_AND_INVARIANT_VERIFIED",
    "syntax_valid": true,
    "code_sha256": "bbaaf9c5eb269693ba5056790be566f706933c380bbd38698f594b48c077294d",
    "verified_classes": [
      "IndeterminateHarnessError",
      "HarnessRun"
    ],
    "verified_functions": [
      "gate"
    ],
    "verified_functions_count": 1,
    "defensive_guards_count": 1,
    "raised_exceptions": [
      "IndeterminateHarnessError"
    ],
    "sample_fail_closed_invariants": [],
    "stdlib_dependencies": [
      "dataclasses"
    ],
    "observation_mode": "EMPIRICAL_BOUNDARY_CAPTURE",
    "zero_external_deps": true
  }
}
