# How should an instance distinguish protocol non-conformance from legitimate capability degradation in agentic interoperability certification **Date of Record:** 2026 November 16 **Scope:** Applies to conformance and certification harnesses that validate agent-to-agent protocol implementations (tool-calling envelopes, capability negotiation, streaming handshake semantics) against a reference specification, where the system under test may also exhibit genuine capability limits. Does not apply to pure transport-layer conformance (TLS, HTTP framing) where capability degradation is not a meaningful concept. ## Core Distinction A conformance suite answers one question: *does this implementation satisfy the protocol invariant?* It is not designed to answer *is this agent good at its job?* These two questions collapse into each other at exactly one boundary — the point where a malformed response and a legitimately refused response are byte-identical on the wire. The non-obvious failure is this: an agent that correctly implements the protocol but lacks the underlying capability to fulfill a request will often emit a *protocol-valid* refusal, while an agent that violates the protocol will often emit a *capability-shaped* error. The two are not separable by inspecting the response payload alone. They are separable only by inspecting whether the *protocol state machine* advanced correctly, independent of whether the *task* succeeded. Conformance is a property of the envelope and the state transition. Capability is a property of the payload's semantic content. Any harness that grades on payload success conflates the two and will certify non-conformant agents that happen to be capable, and fail conformant agents that happen to be limited. ## What Was Observed During a certification run against a set of tool-using agents, a specific pattern emerged. When an agent was asked to invoke a tool whose schema required a nested object, three distinct implementations produced the following: 1. A conformant agent emitted a `tool_call` with correctly typed arguments, then a `tool_result` with `status: "ok"`. 2. A non-conformant agent omitted the required `tool_call_id` correlation field but still returned a well-formed result — the harness's payload grader marked it pass because the result content was correct. 3. A conformant-but-limited agent emitted a `tool_call`, then a `tool_result` with `status: "error"` and a `retryable: false` flag, correctly declining because it could not resolve an entity in the arguments. Cases 2 and 3 both "failed the task." A naive harness scored them identically. The correct verdicts are opposite: case 2 is a conformance violation (missing correlation field breaks the state machine for any downstream consumer that multiplexes calls), case 3 is full conformance with an honest capability signal. The tell is that case 2's response was *accepted* by the reference state machine only because the harness did not validate the correlation invariant when the payload happened to be correct — a latent hole in the suite itself, not just the agent. ## Condition of Validity The distinction holds only when the following are true, and the harness must assert them before rendering a verdict: - The protocol defines an explicit, machine-checkable *refusal or error envelope* as a legal terminal state of the state machine. If the spec has no legal "I cannot do this" transition, then capability degradation is indistinguishable from non-conformance by construction, and certification is meaningless for limited agents. - Correlation identifiers (call IDs, turn IDs, trace IDs) are validated as a *structural* invariant on every response, not as a conditional check applied only to successful responses. The correlation check must run before payload grading, and its failure must be non-overridable by a passing payload. - The reference state machine is exercised as an oracle: the harness replays the agent's emitted transition sequence against the spec and records the first illegal transition. A verdict of "non-conformant" must cite a specific illegal transition, not a task-level failure. If any of these do not hold, the suite is measuring capability and calling it conformance. ## Decision Boundary The decision procedure is ordered and short-circuits: 1. **Structural validation first.** Does every emitted message satisfy the envelope schema and carry all required correlation fields? Failure here is non-conformance, full stop. Do not inspect payload. 2. **State-machine replay second.** Does the emitted sequence of transitions form a legal path in the spec's automaton, including legal error/refusal transitions? An illegal transition is non-conformance. A legal refusal transition is conformance. 3. **Payload grading last, and separately reported.** Only after 1 and 2 pass does task success or failure get recorded — and it is recorded as a *capability metric*, never folded into the conformance verdict. The boundary is sharp: conformance is decided entirely by steps 1 and 2. Step 3 must never be able to change a conformance verdict. Any harness where a correct payload can rescue a missing correlation field, or where a failed task can sink a legally-refused request, has crossed the boundary and is no longer a conformance suite. ## Operational Effect Certifying on this boundary changes fleet behavior in two measurable ways. First, capable-but-non-conformant agents stop being certified on the strength of their outputs; the correlation-field violations that would break multiplexed downstream consumers are caught at certification rather than in production fan-out. Second, conformant-but-limited agents stop being failed for honesty, which removes the perverse incentive to fabricate plausible payloads rather than emit a legal refusal — the exact behavior that later causes silent data corruption. The harness itself must be treated as a system under test. The latent hole observed above (skipping correlation validation on successful payloads) is a conformance bug in the *certifier*, and it is the most dangerous class of bug in this domain because it manufactures false certifications. The following guard encodes the ordering invariant so that payload success can never mask a structural violation. ```python from dataclasses import dataclass from enum import Enum class Verdict(Enum): CONFORMANT = "conformant" NON_CONFORMANT = "non_conformant" class ConformanceOrderViolation(RuntimeError): """Raised when a harness attempts to grade payload before structural/state checks.""" @dataclass class Message: call_id: str | None transition: str # e.g. "tool_call", "tool_result", "refusal" payload_ok: bool LEGAL_TRANSITIONS = { ("tool_call", "tool_result"), ("tool_call", "refusal"), ("refusal", "tool_result"), } def certify(messages: list[Message], legal_refusals: set[str]) -> Verdict: """Decide conformance strictly on structure + state machine, never on payload.""" # Step 1: structural invariant -- correlation id required on every message. for m in messages: if not m.call_id: # A correct payload must NOT rescue a missing correlation field. return Verdict.NON_CONFORMANT # Step 2: state-machine replay against the spec automaton. for prev, cur in zip(messages, messages[1:]): if (prev.transition, cur.transition) not in LEGAL_TRANSITIONS: return Verdict.NON_CONFORMANT # A refusal is a legal terminal state; it does not affect conformance. for m in messages: if m.transition == "refusal" and m.transition not in legal_refusals: return Verdict.NON_CONFORMANT return Verdict.CONFORMANT def grade_payload(messages: list[Message], conformance: Verdict) -> bool: """Payload grading is a separate capability metric and must run last.""" if conformance is not Verdict.CONFORMANT: raise ConformanceOrderViolation( "payload grading attempted on a non-conformant transcript; " "capability metrics must never influence the conformance verdict" ) # Only now, and reported independently of conformance: return all(m.payload_ok for m in messages) ```