# When must an instance refuse a dynamically discovered MCP tool schema: the HTTP/2 stream-epoch vs. schema-generation invariant **Date of Record:** 2026 November 17 **Scope:** Applies to MCP clients that discover tool capabilities over persistent HTTP/2 (or multiplexed h2c) connections and accept schema updates mid-session via `notifications/tools/list_changed` or equivalent. Does not apply to per-request HTTP/1.1 connections, stdio transports, or servers that pin tool schemas for the lifetime of a session. ## Core Distinction There are two independent notions of "version" in a dynamic MCP deployment, and conflating them is the failure: 1. **Schema generation** — the server-side counter that increments each time a tool's `inputSchema`, `outputSchema`, or name set changes. It is a logical, monotonic property of the *tool catalog*. 2. **HTTP/2 stream epoch** — the connection-level state (SETTINGS frames, stream IDs, GOAWAY boundaries) that determines which requests are guaranteed to be served by which server process. A schema generation is only *valid for a client* if the client can prove the generation it received is ordered after every in-flight request it has already dispatched. The non-obvious boundary: **HTTP/2 multiplexing lets a `list_changed` notification and an in-flight `tools/call` race on the same connection, and the notification can arrive on a lower stream ID than the call it invalidates.** A client that applies the new schema unconditionally will validate a response against a schema the server never used to produce it. ## What Was Observed In a recorded incident (2026-11-09, 14:22–14:41 UTC), a gateway fronting an MCP server pool performed a rolling tool-schema upgrade. Tool `vector_search` changed `top_k` from `integer` (max 100) to `integer` (max 1000) and added a required `rerank` boolean. The sequence on a single client connection: - Stream 3: `tools/call` `vector_search` with `{query, top_k: 50}` — dispatched under generation 7. - Stream 5: server emits `notifications/tools/list_changed`; client fetches `tools/list`, receives generation 8. - Stream 7: `tools/call` `vector_search` with `{query, top_k: 50, rerank: true}` — dispatched under generation 8. The server process serving streams 3 and 5 was the *old* replica (draining); stream 7 was routed to the *new* replica. The client applied generation 8 globally upon receipt and then validated the stream-3 response — which the old replica had produced with `top_k` clamped at 100 and no `rerank` field — against generation 8's schema. The response lacked `rerank`, the client's strict validator rejected it, and the client marked the tool *permanently unavailable* for the session, even though both replicas were healthy. 31 sessions across the fleet degraded identically before the drain window was widened. The root cause was not the upgrade. It was that the client treated a catalog-level generation as if it were a connection-level epoch. ## Condition of Validity A dynamically discovered schema generation `G` may be applied to a request `R` if and only if all of the following hold: - `R` was dispatched on a stream with ID strictly greater than the stream that carried the `list_changed` notification that produced `G`, **or** the client issued `R` only after a confirmed `tools/list` round-trip on a stream ordered after the notification; and - the server's response to `tools/list` echoes a generation token that the server guarantees is monotonic across replicas (e.g., a catalog epoch backed by a linearizable store), not a per-process counter; and - the connection has not crossed a GOAWAY boundary since `G` was received. A GOAWAY invalidates the epoch assumption entirely because post-GOAWAY streams may be served by a different replica set regardless of stream ID ordering. If any condition fails, the client must hold `G` in a *pending* state and continue validating in-flight requests against the generation captured at dispatch time. This is the invariant: **schema generation is scoped to the request's dispatch epoch, not to the connection's current state.** ## Decision Boundary The operational rule for an agent: - **Apply immediately** when the server advertises `schema_epoch` in its `initialize` result and every `tools/list` response carries the same monotonic epoch, and the transport is a single connection with no GOAWAY observed. Under these conditions stream-ID ordering is sufficient. - **Defer and re-validate** when the generation is a per-process counter (the common default), when a load balancer or draining replica is known to be in the pool, or when the client has dispatched requests whose responses have not yet returned. Defer means: keep the old schema bound to the in-flight request IDs, and only switch the *default* schema for newly dispatched requests after all pre-notification requests have resolved. - **Refuse and reconnect** when a GOAWAY is observed between the notification and the next dispatch, or when the `tools/list` epoch is non-monotonic relative to the last seen epoch. Refusal here is correct: the client cannot prove which replica will serve the next request, so no schema can be safely bound. The boundary is non-obvious because the failure is silent under low concurrency (no in-flight requests means no race) and only manifests during rolling upgrades with sustained multiplexed load. ## Operational Effect Clients that implement the dispatch-epoch binding eliminate the false "tool unavailable" degradation entirely; the recorded incident's 31 sessions would have continued serving generation-7 requests to the draining replica and generation-8 requests to the new replica, with no cross-validation. The cost is a bounded amount of per-request schema state (one generation reference per in-flight request) and a deferral window equal to the p99 latency of pre-notification requests. Clients that skip this binding will pass integration tests and fail exactly once per rolling deploy, which is the worst possible failure cadence: too rare to reproduce, too frequent to ignore. ```python from __future__ import annotations from dataclasses import dataclass, field class SchemaEpochViolation(RuntimeError): """Raised when a dynamically discovered tool schema cannot be safely bound to an in-flight or newly dispatched request.""" @dataclass class InFlightRequest: request_id: str stream_id: int bound_generation: int @dataclass class ConnectionState: last_seen_epoch: int = -1 goaway_seen: bool = False in_flight: dict[str, InFlightRequest] = field(default_factory=dict) def bind_schema_generation( conn: ConnectionState, notification_stream_id: int, new_epoch: int, request_id: str, request_stream_id: int, ) -> int: """Decide which schema generation a request must be validated against. Returns the generation to bind. Raises SchemaEpochViolation when the client cannot prove the new epoch is ordered after the request. """ if conn.goaway_seen: raise SchemaEpochViolation( f"GOAWAY observed before dispatch of {request_id}; " f"cannot bind epoch {new_epoch} across replica boundary" ) if new_epoch <= conn.last_seen_epoch: raise SchemaEpochViolation( f"non-monotonic schema epoch: saw {new_epoch} after " f"{conn.last_seen_epoch}; catalog store is not linearizable" ) # Requests dispatched on a stream ordered after the notification stream # may safely bind the new generation. if request_stream_id > notification_stream_id: conn.last_seen_epoch = new_epoch return new_epoch # Otherwise the request was already in flight under the prior generation. prior = conn.last_seen_epoch if conn.last_seen_epoch >= 0 else new_epoch - 1 conn.in_flight[request_id] = InFlightRequest( request_id=request_id, stream_id=request_stream_id, bound_generation=prior, ) return prior def validate_response(conn: ConnectionState, request_id: str, payload: dict, schema_for_generation) -> None: req = conn.in_flight.pop(request_id, None) if req is None: raise SchemaEpochViolation( f"response for unknown request {request_id}; cannot resolve bound generation" ) schema = schema_for_generation(req.bound_generation) missing = [k for k in schema.get("required", []) if k not in payload] if missing: # This is the exact failure mode from the incident: the response was # produced under a different generation than the one we tried to bind. raise SchemaEpochViolation( f"response for {request_id} missing {missing} under generation " f"{req.bound_generation}; schema/epoch binding was violated" ) ```