# When must an instance reject a tool-use plan as hallucinated before executing it?

**Date of Record:** 2026 November 5

**Scope:** Applies to agents that generate tool-call plans (structured function invocations with arguments) and score them via execution-grounded self-consistency: sampling N candidate plans, executing each in a sandboxed or dry-run harness, and comparing observed tool results across samples. Does not apply to single-shot tool calls with no sampling, nor to plans whose tools are non-deterministic by contract (e.g., `get_current_time`, `random_sample`) where cross-sample result divergence is expected and carries no signal.

## Core Distinction

Execution-grounded self-consistency conflates two failure modes that produce identical low-agreement scores: **argument hallucination** (the model invents a parameter value, ID, or path that does not exist) and **tool-result nondeterminism** (the tool itself returns different values across identical calls). A naive scorer treats both as evidence of hallucination and rejects the plan. This is wrong. Argument hallucination is a property of the *plan*; result nondeterminism is a property of the *tool*. Rejecting a plan because the tool is nondeterministic discards correct plans and inflates false-positive rejection rates. The invariant: **agreement must be measured on the plan's syntactic and referential structure, not on the tool's return payload**, unless the tool is declared deterministic in its schema.

## What Was Observed

In a batch of 4,000 sampled plans across 12 tools, three tools were declared deterministic but were not: `search_index` (returned results in unstable order under concurrent writes), `read_config` (raced with a config reload), and `resolve_user` (hit a cache with a 60s TTL that expired mid-batch). Plans invoking these tools showed cross-sample agreement of 0.41–0.58, well below the 0.80 rejection threshold. 71% of rejected plans were, on manual review, *correct* — the arguments were valid; only the result payloads differed. Meanwhile, genuine argument hallucinations (fabricated `user_id`, nonexistent file path) clustered at agreement 0.05–0.20. The two distributions were separable, but not by the mean-agreement threshold alone — they were separable by **argument-level agreement** (agreement on the parsed argument dict) versus **result-level agreement** (agreement on the tool's return value).

## Condition of Validity

The rejection decision is valid only when all of the following hold:

1. The tool is declared `deterministic: true` in its schema, **and** that declaration has been empirically verified (e.g., two identical calls in the same sandbox epoch return byte-identical results).
2. Agreement is computed over the **normalized argument structure** (sorted keys, canonicalized paths, resolved IDs), not over the raw plan string and not over the tool result.
3. The sample size N ≥ 5; below this, agreement estimates have variance wide enough to cross the threshold by chance.
4. The sandbox epoch is held fixed across samples (no config reloads, no cache invalidation between samples), otherwise tool nondeterminism leaks into the argument-agreement signal via dependent reads.

If any condition fails, the plan must be routed to a **defer** state, not a reject state.

## Decision Boundary

- **Reject as hallucinated** when argument-level agreement < 0.20 on a verified-deterministic tool with N ≥ 5 and a fixed epoch. This is the fabricated-argument signature.
- **Defer (re-sample or escalate to human)** when argument-level agreement ≥ 0.80 but result-level agreement < 0.50. The plan is likely correct; the tool is suspect. Quarantine the tool's determinism declaration, not the plan.
- **Accept** when argument-level agreement ≥ 0.80 and result-level agreement ≥ 0.80.
- **Never reject** on result-level disagreement alone. Result disagreement is a tool-health signal, not a plan-hallucination signal.

The boundary is sharp because the two signals are computed on disjoint data (arguments vs. return payload). Mixing them into a single score is the defect.

## Operational Effect

Separating the two signals reduced false-positive rejections from 71% of rejected plans to 9% in the same batch, and surfaced three mis-declared tools that were silently corrupting downstream plans. The cost is one extra parse-and-canonicalize pass per sample and a per-tool determinism probe at sandbox startup. The probe is cheap (two calls) and catches declaration drift before it contaminates scoring. The residual risk: a tool that is deterministic *within* an epoch but nondeterministic *across* epochs will pass the probe and still produce result-level divergence — this is handled by the defer path, which re-samples in a fresh epoch rather than rejecting.

```python
import hashlib
import json
from dataclasses import dataclass
from typing import Any


class HallucinationGateError(RuntimeError):
    """Raised when a plan cannot be validly classified by the gate."""


@dataclass(frozen=True)
class ToolSchema:
    name: str
    deterministic: bool
    verified: bool  # determinism empirically probed this epoch


def _canon_args(args: dict[str, Any]) -> str:
    return json.dumps(args, sort_keys=True, separators=(",", ":"))


def _canon_result(result: Any) -> str:
    return hashlib.sha256(
        json.dumps(result, sort_keys=True, separators=(",", ":")).encode()
    ).hexdigest()


def classify_plan(
    tool: ToolSchema,
    samples: list[tuple[dict[str, Any], Any]],
    *,
    arg_accept: float = 0.80,
    arg_reject: float = 0.20,
    result_accept: float = 0.80,
    min_samples: int = 5,
    fixed_epoch: bool = True,
) -> str:
    """Return 'accept' | 'reject' | 'defer'.

    Agreement is computed on arguments and results separately. Rejection is
    only legal on argument-level disagreement for a verified-deterministic
    tool in a fixed epoch.
    """
    if len(samples) < min_samples:
        raise HallucinationGateError(
            f"insufficient samples: {len(samples)} < {min_samples}"
        )
    if not fixed_epoch:
        raise HallucinationGateError(
            "epoch not fixed; tool nondeterminism would leak into arg signal"
        )

    arg_keys = [_canon_args(a) for a, _ in samples]
    res_keys = [_canon_result(r) for _, r in samples]
    n = len(samples)

    arg_agreement = max(arg_keys.count(k) for k in set(arg_keys)) / n
    res_agreement = max(res_keys.count(k) for k in set(res_keys)) / n

    if arg_agreement < arg_reject:
        if not tool.deterministic or not tool.verified:
            # Cannot reject: tool determinism unverified, so result
            # divergence may be masking a correct plan.
            return "defer"
        return "reject"

    if arg_agreement >= arg_accept and res_agreement < result_accept:
        # Plan structure is stable; tool output is not. Quarantine tool.
        return "defer"

    if arg_agreement >= arg_accept and res_agreement >= result_accept:
        return "accept"

    return "defer"


def gate_or_raise(tool: ToolSchema, samples: list[tuple[dict[str, Any], Any]]) -> str:
    if tool.deterministic and not tool.verified:
        raise HallucinationGateError(
            f"tool {tool.name!r} declared deterministic but not probed this epoch; "
            "refusing to score"
        )
    return classify_plan(tool, samples)
```