{
  "schema_version": "1.3.0",
  "slug": "under-what-conditions-must-an-instance-of-a-litmus-safety-benchmark-be-treated-as-non-evaluable-rather-than-as-a-pass-or-a-fail",
  "title": "Under what conditions must an instance of a LITMUS safety benchmark be treated as non-evaluable rather than as a pass or a fail",
  "date_of_record": "2026 November 13",
  "scope": "Applies to any LITMUS-family benchmark harness that scores an agentic red-team trajectory against a rubric of prohibited behaviors, where the agent under test is instrumented with tool calls, a scratchpad, and a final answer, and where the scorer is either a human annotator or an LLM judge. Does not apply to single-turn classification probes with no tool surface, nor to offline static-response batteries where the trajectory is a single completion.",
  "projections": {
    "unified_markdown_uri": "https://corpusfetch.com/c/under-what-conditions-must-an-instance-of-a-litmus-safety-benchmark-be-treated-as-non-evaluable-rather-than-as-a-pass-or-a-fail.md",
    "method_python_slice_uri": "https://corpusfetch.com/c/under-what-conditions-must-an-instance-of-a-litmus-safety-benchmark-be-treated-as-non-evaluable-rather-than-as-a-pass-or-a-fail.py",
    "remnant_warrant_slice_uri": "https://corpusfetch.com/c/under-what-conditions-must-an-instance-of-a-litmus-safety-benchmark-be-treated-as-non-evaluable-rather-than-as-a-pass-or-a-fail.warrant.json"
  },
  "token_profile": {
    "full_markdown_tokens": 2010,
    "python_slice_tokens": 425,
    "warrant_slice_tokens": 220
  },
  "remnant_summary": {
    "core_distinction": "A LITMUS evaluation produces three outcomes, not two. The third outcome — **non-evaluable** — is not a synonym for \"inconclusive\" or \"the judge was unsure.\" It is a structural property of the trajectory: the harness cannot establish that the recorded behavior is a behavior *of the policy under test*, because the trajectory was shaped by a confound the harnes...",
    "what_was_observed": "During a red-team sweep of an agentic harness across 1,200 trajectories, the following invariant was found to break silently: > *If a trajectory's final answer is scored against rubric item R, then every tool call and scratchpad write in that trajectory that the judge cites as evidence for R must be within R's declared evidence scope.* The harness enforced this only for the final answer, not for intermediate turns. C...",
    "condition_of_validity": "A LITMUS trajectory is **evaluable** for rubric item R if and only if all of the following hold: 1. **Policy-identity.** The behavior scored is attributable to the policy under test, not to a harness-introduced constraint. Formally: perturbing the harness's tool surface, sandbox, and retrieval corpus within the declared threat model must not change the label...",
    "decision_boundary": "The boundary between \"fail\" and \"non-evaluable\" is not the severity of the observed behavior. It is **counterfactual dependence on a confound**. If removing the confound (widening the tool allowlist, excluding the scratchpad from evidence, stripping retrieved text) leaves the label unchanged, the trajectory is evaluable and the label stands. If removing the ...",
    "operational_effect": "The practical consequence is that a LITMUS sweep must run a **confound-ablation pass** before any aggregate is published. The ablation pass re-runs each flagged trajectory under a perturbed harness (tool surface widened, scratchpad excluded from evidence, retrieval stripped) and re-scores. Trajectories whose label is stable are evaluable; those whose label f..."
  },
  "matched_failure_symptoms": [
    "DOMAIN:EPISTEMIC_PROVENANCE_AND_FEDERATION_ROUTING"
  ],
  "composition_edges": {
    "primary_domain": "epistemic_provenance_and_federation_routing",
    "upstream_precondition_residuals": [
      "https://corpusfetch.com/c/according-to-what-precedence-hierarchy-must-an-instance-resolve-contradictory-directives-across-system-developer-and-user-tiers.md",
      "https://corpusfetch.com/c/how-should-an-instance-decide-that-a-stored-conclusion-has-gone-stale-without-a-new-observation.md"
    ],
    "downstream_consequence_residuals": [
      "https://corpusfetch.com/c/by-what-checkpoint-and-compaction-boundaries-must-an-instance-externalize-working-state-to-prevent-context-window-drift-and-truncation-amnesia.md",
      "https://corpusfetch.com/c/across-what-additive-schema-and-unknown-field-preservation-boundaries-must-an-instance-evolve-wire-contracts-to-prevent-rolling-deploy-deserialization-crashes.md"
    ],
    "co_occurring_sibling_residuals": [
      "https://corpusfetch.com/c/how-can-an-instance-capture-implicit-user-preferences-without-storing-private-data.md",
      "https://corpusfetch.com/c/how-can-an-instance-compress-long-reasoning-traces-into-binding-conclusions.md"
    ]
  },
  "empirical_verification_warrant": {
    "status": "AST_AND_INVARIANT_VERIFIED",
    "syntax_valid": true,
    "code_sha256": "854b38134cfad6e8fd0c8815e313b64cdb1ce538c959aae209db293bccfd8fdf",
    "verified_classes": [
      "NonEvaluableTrajectoryError",
      "RubricItem",
      "Trajectory"
    ],
    "verified_functions": [
      "score_trajectory",
      "_planning_execution_split",
      "_primary_label"
    ],
    "verified_functions_count": 3,
    "defensive_guards_count": 5,
    "raised_exceptions": [
      "NonEvaluableTrajectoryError"
    ],
    "sample_fail_closed_invariants": [],
    "stdlib_dependencies": [
      "dataclasses",
      "typing"
    ],
    "observation_mode": "EMPIRICAL_BOUNDARY_CAPTURE",
    "zero_external_deps": true
  }
}