# Why does an instance of MCP streamable HTTP transport with resumable SSE and session affinity fail silently when a stateless agent server is scaled behind a load balancer that honors `Last-Event-ID` but not affinity cookies

**Date of Record:** 2026 November 11

**Scope:** Applies to MCP servers exposing the streamable HTTP transport (POST for requests, GET for the server-to-client SSE stream) with resumable event streams keyed by `Last-Event-ID`, deployed as multiple stateless replicas behind an L7 load balancer. Does not apply to single-replica deployments, to stdio transports, or to deployments where event IDs are globally unique and replay is served from a shared durable store.

## Core Distinction

The failure is not "the load balancer drops connections" and it is not "resumable SSE is broken." It is the interaction between two independently correct mechanisms: (1) `Last-Event-ID` resumption, which is a *client-to-server* contract that assumes the server can reconstruct the stream from an arbitrary event ID, and (2) session affinity, which is a *load-balancer-to-server* contract that assumes the same client returns to the same replica. When replicas are stateless and generate event IDs from a *per-process monotonic counter*, both mechanisms pass their individual unit tests while the composite system silently replays the wrong events, skips events, or blocks indefinitely.

The non-obvious invariant is this: **a resumable stream is only resumable if the event ID namespace is shared across every replica that a client's connection can land on.** Affinity is not a substitute for a shared ID namespace; it is a performance optimization that *hides* the absence of one until a rebalance, a pod restart, or a connection drain occurs.

## What Was Observed

A three-replica deployment of a stateless agent server. Each replica implemented the streamable HTTP transport identically: a POST handler that enqueues a request and returns `202` with a `Location` pointing at the GET stream, and a GET handler that emits `id:` fields using a process-local `itertools.count()`. The load balancer was configured with `Last-Event-ID`-aware routing (it parsed the header and forwarded to the replica that "owned" the ID range) but with affinity cookies enabled only on a best-effort basis.

Observed sequence during a rolling restart:

1. Client opens GET stream, receives events `id: 41`, `id: 42` from replica A.
2. Replica A is drained; the connection is severed mid-stream.
3. Client reconnects with `Last-Event-ID: 42`. The load balancer forwards to replica B because the affinity cookie expired.
4. Replica B has its own counter at `id: 17`. It cannot find event `42`. Its handler treats an unknown `Last-Event-ID` as "resume from current head" and emits `id: 18`, `id: 19`, ...
5. The client's SSE parser accepts the lower IDs (they are syntactically valid), and the agent runtime's deduplication layer, keyed on `(stream_id, event_id)`, treats `18` and `19` as *new* events. The runtime executes tool calls that were already executed before the drain.

The observable symptom was duplicated tool invocations with non-idempotent side effects, and a stream that appeared healthy (HTTP 200, well-formed SSE) while the event sequence was semantically corrupt. No error was logged on either replica: replica B did exactly what its code said to do for an unknown ID, and the load balancer did exactly what its config said to do when affinity was unavailable.

## Condition of Validity

The failure requires all of the following to be true simultaneously:

- Replicas are stateless with respect to stream history (no shared replay buffer).
- Event IDs are generated from a per-process source (counter, timestamp, UUIDv4 without a replica prefix).
- The client honors `Last-Event-ID` and reconnects automatically.
- The load balancer can route a reconnect to a replica other than the one that produced the ID (affinity expiry, rebalance, drain, or scale event).
- The server's handler for an unknown `Last-Event-ID` is non-failing (resume-from-head, or silently ignore).

If *any* one of these is false, the failure does not manifest. In particular, if the server returns `400` on an unknown `Last-Event-ID`, the client surfaces an error and the operator notices. The dangerous configuration is the one that "degrades gracefully."

## Decision Boundary

The boundary is between *tolerating* an unknown event ID and *resolving* it. A stateless agent server has exactly three defensible policies, and they are mutually exclusive:

1. **Shared namespace + shared replay.** Event IDs encode a replica-independent epoch (e.g., `{stream_epoch}:{seq}`) and replay is served from a durable store. Affinity is optional. This is the only policy that is correct under arbitrary routing.
2. **Pinned affinity + fail-closed.** Affinity is a hard requirement; the server returns `409 Conflict` (or `400`) when `Last-Event-ID` is not present in its local buffer, forcing the client to restart the stream from scratch with a new session. Correctness depends entirely on the client handling the restart.
3. **No resumption.** The server advertises no `Last-Event-ID` support; every reconnect starts a fresh stream and the client is expected to re-issue its request. This is the honest stateless option.

The failure occurs when an implementation *advertises* policy 1 (resumable SSE) but *implements* policy 3's ID generation (per-process counter) with policy 2's routing assumption (best-effort affinity). The decision boundary is: **if you cannot guarantee a shared ID namespace, you must not emit `id:` fields at all, or you must fail closed on unknown IDs.** Emitting per-process IDs and resuming from head is the one combination that is silently wrong.

## Operational Effect

The operational effect is *silent semantic corruption under rebalance*, not an outage. Health checks pass, latency is normal, and error rates are flat. The corruption surfaces downstream as duplicated side effects (double-charged payments, repeated file writes, re-sent emails from an agent tool) or as lost events during a drain window. Because the corruption is correlated with deploy and scale events rather than with load, it is frequently misattributed to the agent's own retry logic or to the downstream tool's idempotency, and the transport layer is exonerated.

The correct instrumentation is to tag every emitted event with the producing replica ID and to assert, on the client side, that the sequence of `(replica_id, event_id)` pairs is contiguous across reconnects. A gap or a replica change with a decreasing event ID is the signal. Absent that assertion, the failure is invisible.

```python
import re
from dataclasses import dataclass

_EVENT_ID_RE = re.compile(r"^(?P<epoch>[0-9a-f]{8}):(?P<seq>\d+)$")


class StreamResumeViolation(RuntimeError):
    """Raised when a Last-Event-ID cannot be honored under the active policy."""
    pass


@dataclass(frozen=True)
class EventId:
    epoch: str   # replica-independent stream epoch
    seq: int     # monotonic within epoch


def parse_last_event_id(raw: str | None) -> EventId | None:
    if raw is None:
        return None
    m = _EVENT_ID_RE.match(raw)
    if not m:
        # Per-process counters (e.g. "42") are not valid resumable IDs.
        raise StreamResumeViolation(
            f"Last-Event-ID {raw!r} is not in the shared '{epoch}:{seq}' namespace; "
            "per-process counters cannot be resumed across replicas"
        )
    return EventId(epoch=m.group("epoch"), seq=int(m.group("seq")))


def resume_stream(
    raw_last_event_id: str | None,
    local_epoch: str,
    local_head_seq: int,
    replay_available_through: int,
) -> int:
    """Return the sequence number to resume from, or raise if unsafe."""
    last = parse_last_event_id(raw_last_event_id)
    if last is None:
        return local_head_seq

    if last.epoch != local_epoch:
        # A different epoch means a different replica's namespace.
        # Do NOT silently resume from head: that replays or skips events.
        raise StreamResumeViolation(
            f"epoch mismatch: client holds {last.epoch!r}, replica serves {local_epoch!r}; "
            "affinity was not honored and no shared replay exists"
        )

    if last.seq > replay_available_through:
        raise StreamResumeViolation(
            f"cannot replay seq {last.seq}: only through {replay_available_through} is durable"
        )

    return last.seq + 1
```