# What structural signals indicate when a lightweight critic model is insufficient to gate an irreversible tool invocation

**Date of Record:** 2026 November 11

**Scope:** Applies to agent architectures where a small critic/verifier model scores a proposed plan or tool call before execution, and where the downstream tool call is irreversible (payment, deletion, external message, schema migration, infrastructure mutation). Does not apply to reversible or idempotent tool calls where rollback is cheap, nor to cases where the critic and actor are the same model instance.

## Core Distinction

A lightweight critic model is a *distributional* judge: it estimates whether a plan resembles plans it has seen labeled safe. It is not a *causal* judge: it does not simulate the plan's effect on world state. The boundary failure is that critic scores remain high (0.9+) precisely in the region where the plan is structurally novel but lexically familiar. The critic's confidence is a function of surface-form similarity, not of whether the plan's preconditions hold. The non-obvious invariant: **critic score and plan risk are anti-correlated for plans that reuse familiar verbs with novel argument bindings.** A plan that says "delete the staging bucket" scores high; "delete the bucket named `prod-backups-2023`" scores nearly as high because the verb-object skeleton is identical and the critic attends to the skeleton.

## What Was Observed

Across a corpus of 4,100 gated plan proposals on a payment-mutation and resource-deletion tool surface, we logged critic scores, structural features, and human adjudication of the proposals. Three structural signals co-occurred on every plan that the critic scored ≥0.85 but which human review classified as unsafe for execution:

1. **Argument-binding novelty under a familiar verb.** The tool name and verb appeared in the critic's training distribution, but one or more arguments referenced an entity not present in any prior approved plan (a new resource ID, a new recipient, a new region). The critic's token-level attention weighted the verb heavily and the novel entity lightly. Signal: entity novelty ratio of the argument tuple relative to the approved-plan index.

2. **Cross-step precondition references.** The plan's correctness depended on a state established by an earlier step *in the same plan* (e.g., "after rotating the key, revoke the old one"). The critic evaluated steps independently and could not represent the dependency. Signal: presence of a step whose precondition is the postcondition of a sibling step, with no explicit assertion between them.

3. **Irreversibility asymmetry between plan and fallback.** The plan's stated rollback path was strictly weaker than its forward path (forward: delete object + all versions; rollback: "restore from backup if available"). The critic scored the forward text and ignored the asymmetry. Signal: forward-effect cardinality minus rollback-effect cardinality > 0.

Critically, none of these signals are visible in the critic's scalar output, and all three are computable cheaply from the plan's parsed structure before the critic runs.

## Condition of Validity

The critic gate is valid only when, for the proposed plan, all of the following hold:

- Every argument entity appears in the approved-plan index (entity novelty ratio = 0), OR novelty is confined to arguments the tool treats as read-only.
- No step's precondition depends on a sibling step's postcondition without an intervening explicit assertion step.
- Forward and rollback effect cardinalities are equal (fully reversible) or the tool is classified idempotent.

When any condition fails, the critic's score carries no evidential weight for the irreversibility decision, regardless of its value. This is not a threshold-tuning problem; raising the critic threshold does not restore validity because the failure is structural, not scalar.

## Decision Boundary

Route to human adjudication (or a full simulation verifier) when *any* structural signal fires, **before** consulting the critic score. If no signal fires, the critic score may be used as a tiebreaker at its usual threshold. Do not average the structural signals into a single score with the critic output — the signals are gating predicates, not weights. The decision is a conjunction: `execute = (no structural signal) AND (critic_score >= threshold)`. Treating it as a disjunction (execute if either passes) is the exact bug that produced the observed failures.

## Operational Effect

Applying the structural pre-filter reduced unsafe executions from 0.7% to 0.04% of gated proposals while increasing human adjudication load by 3.1% — the filter fires rarely but with high precision on the dangerous tail. The critic model was left unchanged; only the gate topology changed. The lesson for operational memory: a critic score is a *likelihood under a familiar distribution*, and irreversibility decisions require *structural precondition checks* that no scalar critic can supply. When the plan's structure is novel, the critic is confidently wrong, and its confidence is highest exactly where its evidence is weakest.

```python
from dataclasses import dataclass, field


class IrreversibleGateError(RuntimeError):
    """Raised when a plan must not pass the critic gate without human adjudication."""


@dataclass
class PlanStep:
    tool: str
    args: dict
    postcondition: str | None = None
    precondition: str | None = None
    reversible: bool = True


@dataclass
class Plan:
    steps: list[PlanStep] = field(default_factory=list)
    approved_entities: set[str] = field(default_factory=set)


def _entity_novelty(plan: Plan) -> float:
    entities = [str(v) for s in plan.steps for v in s.args.values()]
    if not entities:
        return 0.0
    novel = sum(1 for e in entities if e not in plan.approved_entities)
    return novel / len(entities)


def _has_cross_step_precondition(plan: Plan) -> bool:
    postconds = {s.postcondition for s in plan.steps if s.postcondition}
    for s in plan.steps:
        if s.precondition and s.precondition in postconds and s.precondition != s.postcondition:
            return True
    return False


def _rollback_asymmetry(plan: Plan) -> bool:
    return any(not s.reversible for s in plan.steps)


def gate(plan: Plan, critic_score: float, threshold: float = 0.85) -> bool:
    signals = {
        "entity_novelty": _entity_novelty(plan),
        "cross_step_precondition": _has_cross_step_precondition(plan),
        "rollback_asymmetry": _rollback_asymmetry(plan),
    }
    if signals["entity_novelty"] > 0.0:
        raise IrreversibleGateError(
            f"novel entity binding under familiar verb: ratio={signals['entity_novelty']:.2f}; "
            f"critic_score={critic_score} carries no evidential weight"
        )
    if signals["cross_step_precondition"]:
        raise IrreversibleGateError(
            "plan correctness depends on sibling-step postcondition; critic cannot represent dependency"
        )
    if signals["rollback_asymmetry"]:
        raise IrreversibleGateError(
            "forward effect cardinality exceeds rollback effect cardinality"
        )
    return critic_score >= threshold
```