# How does an instance detect policy-violating agent behavior when OpenTelemetry GenAI spans are sampled, batched, and semantically lossy at the collector boundary

**Date of Record:** 2026 November 12

**Scope:** Applies to runtime policy enforcement driven by OTel GenAI semantic conventions (`gen_ai.*` attributes) where span emission is asynchronous and subject to collector-side sampling or batching. Does not apply to synchronous in-process instrumentation hooks or to evaluation pipelines reading from a lossless, unsampled trace store.

## Core Distinction

A trace-derived safety signal is *evidence of behavior*, not *the behavior itself*. The invariant that most evaluations implicitly assume — that every policy-relevant action produces exactly one durable, ordered, complete span — is false at the collector boundary. The boundary failure is not "missing spans" (which is detectable) but **semantically lossy spans that arrive and look valid**: a tool-call span whose `gen_ai.tool.call.arguments` was truncated by an attribute-size cap, a completion span emitted after the parent `invoke_agent` span was already closed by the sampler, or a `gen_ai.usage.completion_tokens` count that reflects a retried generation but not the retry that triggered the policy check.

The distinction that matters operationally: **detection of absence** (span never arrived) versus **detection of degradation** (span arrived with reduced fidelity). Absence is countable and can be reconciled against a client-side emission counter. Degradation is not, because the degraded span is indistinguishable from a legitimately short one unless the producer explicitly marks it.

## What Was Observed

In a deployment evaluating agent tool-use against a "no filesystem write outside workspace" policy, the runtime enforcer consumed spans from an OTLP collector configured with `batch` processor (default 512-span batches, 5s timeout) and tail-based sampling at 10% for non-error traces. Three observed conditions:

1. **Truncation masquerading as compliance.** A `gen_ai.tool.call` span with `gen_ai.tool.name = "write_file"` had its `gen_ai.tool.call.arguments` attribute capped at 1024 bytes by the collector's `attributes` processor. The path argument fell past the cap. The enforcer's parser saw a JSON fragment, failed closed on parse, and classified the call as "unknown" — which the policy engine treated as non-violating. The violation was real; the signal was absent *inside a present span*.

2. **Ordering inversion across batch flush.** A policy check keyed on "completion after tool call" observed the completion span before the tool span because the tool span was held in a different batch that flushed 400ms later. The enforcer's state machine, which required tool-before-completion, dropped the completion as orphaned. A subsequent policy evaluation over the same trace (offline, replayed) reached the opposite verdict.

3. **Sampling-induced false negatives on retry paths.** Tail sampling kept traces with `error=true` spans. A policy-violating generation that was *retried and succeeded* produced no error span; the violating attempt's span was dropped by head sampling at the SDK (default `parentbased_traceidratio=0.1`). The violation existed only in the dropped attempt.

## Condition of Validity

Trace-derived safety signals are valid for enforcement **only when all three hold**:

- The producer emits a monotonic `emission_seq` (or equivalent) per agent instance, and the enforcer reconciles observed spans against it. Without this, absence and degradation are conflated.
- Every policy-relevant attribute is either under the size cap or carries an explicit `truncated=true` / `gen_ai.attributes.dropped` marker. The collector must be configured with `attributes` processor `include`/`exclude` that never silently truncates policy-bearing fields, or the producer must pre-hash them.
- The enforcer operates on a *causal* ordering derived from span `start_time_unix_nano` plus `trace_id`/`span_id` parentage, never on arrival order. Batch flush reorders arrival; it does not reorder causality.

When any condition fails, the correct operational posture is **fail-closed on the policy dimension the signal was meant to protect**, not fail-open. The common error is treating a degraded signal as a clean signal.

## Decision Boundary

The boundary is between **enforcement** and **adjudication**:

- **Enforcement** (blocking a tool call, terminating an agent turn) may only consume signals that satisfy the Condition of Validity. A truncated argument span must not be used to *clear* a call; it may only be used to *flag* it.
- **Adjudication** (post-hoc scoring, offline evaluation, alerting) may consume degraded signals but must record the degradation class alongside the verdict, so downstream aggregation can weight or exclude them.

Concretely: an enforcer that sees a `write_file` span with unparseable arguments must raise, halt the turn, and route to adjudication — it must not treat "unparseable" as "benign." An offline evaluator that sees the same span may score the trace but must tag the score `signal_degraded=true`.

## Operational Effect

The practical consequence is that **runtime policy enforcement cannot be a pure function of trace contents**. It must be a function of trace contents *plus* a reconciliation state that tracks what the producer claims to have emitted. This adds a small stateful component (per-instance emission counters, or a periodic reconciliation against the SDK's own metrics) but eliminates the class of failures where a degraded span is silently read as a compliant one. Teams that skip this reconciliation consistently report "the enforcer never fired" on incidents that the offline replay later confirms were clear violations — the signal was present in the trace store, degraded at the collector, and misread by the enforcer.

```python
import hashlib
import json
from dataclasses import dataclass


class SignalDegradedError(RuntimeError):
    """Raised when a policy-bearing span cannot be trusted for enforcement."""


class EmissionReconciliationError(RuntimeError):
    """Raised when observed spans do not reconcile against producer emission count."""


@dataclass(frozen=True)
class PolicySpan:
    trace_id: str
    span_id: str
    tool_name: str
    arguments_raw: str
    truncated: bool
    emission_seq: int


POLICY_BEARING_TOOLS = {"write_file", "shell_exec", "http_post"}


def evaluate_tool_span(span: PolicySpan) -> str:
    """Return 'allow', 'deny', or raise if the signal is not enforceable."""
    if span.tool_name not in POLICY_BEARING_TOOLS:
        return "allow"

    if span.truncated:
        # Degraded signal: cannot clear, cannot silently pass.
        raise SignalDegradedError(
            f"policy-bearing span {span.span_id} for tool {span.tool_name!r} "
            f"was truncated at the collector; refusing to adjudicate as compliant"
        )

    try:
        args = json.loads(span.arguments_raw)
    except json.JSONDecodeError as exc:
        raise SignalDegradedError(
            f"unparseable arguments on span {span.span_id}: {exc}"
        ) from exc

    if span.tool_name == "write_file":
        path = args.get("path", "")
        if not path.startswith("/workspace/"):
            return "deny"
    return "allow"


def reconcile(observed: list[PolicySpan], producer_emitted: int) -> None:
    """Enforce that the enforcer saw every span the producer claims to have sent."""
    seen = {s.emission_seq for s in observed}
    expected = set(range(producer_emitted))
    missing = expected - seen
    if missing:
        # Absence is detectable; do not proceed to enforcement on a partial view.
        raise EmissionReconciliationError(
            f"missing emission_seq values {sorted(missing)[:5]} "
            f"({len(missing)} of {producer_emitted} unaccounted)"
        )


def digest_span(span: PolicySpan) -> str:
    """Stable hash for cross-pipeline comparison of the same logical span."""
    h = hashlib.sha256()
    h.update(span.trace_id.encode())
    h.update(span.span_id.encode())
    h.update(span.arguments_raw.encode())
    return h.hexdigest()
```