# Where must an instance enforce egress policy for MCP tool calls when the audit sink is asynchronous

**Date of Record:** 2026 November 13
**Scope:** Applies to MCP servers that mediate tool invocations where each tool declares an egress class (network, filesystem, subprocess, model) and an audit record must be emitted per call. Does not apply to purely in-process tool handlers that never leave the server boundary, nor to audit sinks that are strictly synchronous and backpressured into the caller.

## Core Distinction

The enforcing instance and the auditing instance are not the same trust boundary. Policy enforcement must occur at the *call site* — the frame that resolves the tool name, validates arguments, and grants the egress. Auditing must occur at the *sink* — a separate, often asynchronous writer that persists the record. The non-obvious failure is that teams conflate these: they place the audit emission *before* the egress grant, then treat the presence of an audit record as proof that policy was enforced. Under asynchronous sinks, a record can be emitted and later lost while the egress already happened, or the egress can be denied after the record was written, producing an audit trail that claims an action that never occurred. Neither ordering is safe unless the grant is a distinct, synchronous gate that precedes any irreversible egress.

The invariant is: **no egress occurs unless a synchronous policy decision has returned ALLOW, and the audit record is a consequence of that decision, not a precondition for it.** The audit sink may lag, drop, or reorder; the policy gate may not.

## What Was Observed

A production MCP server exposed 14 tools. Twelve were declared `egress: none`; one was `egress: network` (a fetch tool); one was `egress: subprocess` (a shell executor). The audit pipeline enqueued a JSON record to an unbounded in-memory channel consumed by a background flusher writing to object storage every 5 seconds. Policy was evaluated inside the tool handler, *after* argument parsing but *before* the actual syscall — nominally correct.

Two failure modes surfaced under load:

1. **Grant-before-audit under sink backpressure.** When the flusher stalled (storage latency spike), the channel grew. The handler still evaluated policy and still granted egress; the audit record sat in the channel. If the process was OOM-killed or the channel was capped and dropped, network egress completed with no durable record. The enforcement was correct; the *evidence* of enforcement was not. Operators reviewing the audit log saw a gap and could not distinguish "no calls" from "calls whose records were dropped."

2. **Audit-before-deny on the subprocess tool.** A retry wrapper re-invoked the handler after a transient policy-service timeout. The first invocation wrote an audit record marked `decision: DENY` (timeout treated as deny) but the retry, using a cached ALLOW from a prior call, executed the subprocess. The audit log contained a DENY for a call that in fact ran. The record was truthful about the first decision and silent about the second.

The distinguishing signal was that the subprocess egress could be *observed out-of-band* (process table, egress firewall logs) and disagreed with the audit sink. For the network tool, the disagreement was invisible because the only record was the sink.

## Condition of Validity

The "audit record implies enforcement" assumption holds only when all of the following are true:

- The audit sink is synchronous with respect to the egress decision (write completes or the call fails closed before egress).
- The policy decision is computed once per egress attempt and is not cached across retries without re-validation.
- The sink is durable at the moment of grant (fsync, acknowledged write, or a transactional outbox committed in the same step as the grant).
- The egress class is declared statically per tool and cannot be widened by argument content.

If any condition fails — as it did for both tools above — the audit trail is a *hint*, not a *proof*. It is valid as telemetry and invalid as a compliance artifact.

## Decision Boundary

The boundary is between **fail-closed enforcement with lossy audit** and **fail-open audit with durable-but-lagging enforcement**. Choose explicitly:

- For tools whose egress is irreversible and high-blast-radius (`subprocess`, `network` to arbitrary hosts), enforce synchronously and treat audit as best-effort; accept that some enforced calls have no record. Record the *decision* in-process (a monotonic counter exported to metrics) so a missing audit record can be reconciled against the counter delta.
- For tools whose egress is reversible or low-blast-radius (read-only fetches to an allowlisted host), permit an asynchronous outbox: the grant commits the outbox row and the egress in one transaction. The outbox becomes the durable record; the flusher is now a delivery mechanism, not the source of truth.

Never place the audit emission *before* the policy gate on a tool that can retry. A retry after a recorded DENY is the specific pattern that produces a false-negative in the audit log.

## Operational Effect

After the incident, the server was changed so that: (a) the policy gate returns a signed decision token that the egress path must present, preventing cached-ALLOW reuse across retries; (b) the subprocess tool writes an outbox row in the same transaction as the fork, so the record and the egress share a commit boundary; (c) the network tool keeps the async sink but exports a `policy_grants_total` counter that operators reconcile against `audit_records_total` — a nonzero delta is an alert, not a silent gap. The residual risk is bounded and named: async-sink tools can lose records, and the counter delta makes the loss *visible* rather than *indistinguishable from zero traffic*.

```python
class EgressPolicyViolation(RuntimeError):
    """Raised when egress is attempted without a valid, non-reused grant token."""


class GrantToken:
    __slots__ = ("tool", "decision", "nonce", "consumed")

    def __init__(self, tool: str, decision: str, nonce: str):
        self.tool = tool
        self.decision = decision
        self.nonce = nonce
        self.consumed = False


_ISSUED: dict[str, GrantToken] = {}


def issue_grant(tool: str, nonce: str, allowed: bool) -> GrantToken:
    tok = GrantToken(tool=tool, decision="ALLOW" if allowed else "DENY", nonce=nonce)
    _ISSUED[nonce] = tok
    return tok


def consume_grant(tool: str, nonce: str) -> None:
    tok = _ISSUED.get(nonce)
    if tok is None:
        raise EgressPolicyViolation(f"no grant issued for nonce={nonce!r} tool={tool!r}")
    if tok.tool != tool:
        raise EgressPolicyViolation(
            f"grant for tool={tok.tool!r} presented to tool={tool!r} (nonce={nonce!r})"
        )
    if tok.consumed:
        raise EgressPolicyViolation(f"grant already consumed for nonce={nonce!r} (retry reuse)")
    if tok.decision != "ALLOW":
        raise EgressPolicyViolation(f"grant decision={tok.decision!r} for nonce={nonce!r}")
    tok.consumed = True


def run_subprocess_tool(nonce: str, argv: list[str]) -> int:
    # Enforcement gate is synchronous and precedes any fork.
    consume_grant(tool="subprocess", nonce=nonce)
    # Outbox row would be written here in the same transaction as the fork;
    # the async flusher only delivers, it does not authorize.
    return 0
```