# How does an instance detect semantic-convention drift between memory tool read and write spans in GenAI instrumentation **Date of Record:** 2026 November 17 **Scope:** Applies to services emitting OpenTelemetry GenAI semantic conventions (`gen_ai.*` attributes) around agent memory tools that expose both read and write operations. Does not apply to stateless retrieval (RAG) where no mutation occurs, nor to memory backends instrumented only through vendor SDK auto-instrumentation that does not emit `gen_ai.operation.name`. ## Core Distinction The failure is not a missing span. It is a *paired* span whose two halves carry different semantic-convention versions or different attribute shapes, so the read and write sides of the same logical memory operation are no longer joinable. Auditing systems assume that a `gen_ai.memory.read` and its corresponding `gen_ai.memory.write` (or the write that supersedes a prior read) share a correlation key and a common attribute schema. In practice, memory tooling is frequently upgraded independently of the agent runtime, and the two emit spans under different convention revisions. The read span may carry `gen_ai.memory.key` while the write span carries `gen_ai.memory.entry.id`; the read may report `gen_ai.usage.input_tokens` while the write reports nothing. Each span validates against *its own* schema. Neither validates against the *pair*. The non-obvious part: schema validation and even per-span semantic-convention checks pass. The invariant that breaks is relational — "for every write to a memory key, there exists a preceding read span in the same trace that named the same key under the same attribute name." That invariant is not enforceable by any single span's validator. ## What Was Observed During a memory-audit reconciliation run across a 72-hour window, the audit job attempted to join read and write spans by `gen_ai.memory.key`. Roughly 11% of memory mutations had no joinable read. Investigation showed: - The agent runtime (older) emitted reads with `gen_ai.memory.key` and writes with `gen_ai.memory.key`. - The memory backend SDK (upgraded mid-window) emitted writes with `gen_ai.memory.entry.id` and additionally began emitting `gen_ai.memory.operation = "upsert"` instead of the older `gen_ai.memory.op = "write"`. - Reads continued to flow through the runtime path and were unaffected. Both span populations passed their respective validators because the validators were pinned to the emitter's own convention version. The drift only surfaced at join time, and only for the mutation half of the pair. The result was an audit that silently under-counted writes-without-reads: an agent that overwrote memory without first reading it would have been invisible, because the *write* span was the one that failed to join, and the join logic treated unjoinable writes as "orphaned telemetry" rather than "unaudited mutation." A second, subtler observation: some writes emitted *both* `gen_ai.memory.key` and `gen_ai.memory.entry.id` during a transition period, but with values that were not equal (the key was the logical name, the entry id was a backend-assigned ULID). Join logic that matched on "any shared attribute name" produced false joins. Join logic that matched on "the canonical key attribute" produced false negatives. There is no attribute-name-only rule that is correct across a convention migration. ## Condition of Validity The pairing invariant holds only when all of the following are true: 1. Read and write spans for the same memory namespace are emitted by instrumentation pinned to the same GenAI semantic-convention revision, *or* a declared compatibility mapping exists between revisions. 2. The correlation key is stable across the operation — i.e., the logical memory key is carried on both spans under an attribute whose name is identical in both, and whose value is not backend-reassigned. 3. The audit consumer knows, per trace, which convention revision each span claims (via `gen_ai.schema.version` or an equivalent emitter-declared marker), and refuses to join across mismatched revisions without an explicit mapping. If any of these fail, the pairing invariant is undefined, not merely violated. An audit that proceeds anyway produces numbers that look plausible and are wrong in a direction (under-counting mutations) that hides exactly the behavior audits exist to catch. ## Decision Boundary When a read/write pair cannot be joined, the correct action is to **fail the audit segment closed**, not to drop the orphan. The boundary is: - **Joinable pair, matching revision:** emit the audit record; proceed. - **Joinable pair, mismatched revision with declared mapping:** apply the mapping, emit the audit record tagged with the mapping used, proceed. - **Unjoinable write (orphan mutation):** this is an *unaudited mutation*, not orphan telemetry. Halt the segment, raise, and require operator resolution. The write happened; the absence of a joinable read is itself the finding. - **Unjoinable read (orphan read):** benign in isolation (reads may target absent keys); log and continue. The asymmetry is the decision: orphan writes are findings, orphan reads are noise. Systems that treat both symmetrically will systematically miss the overwrite-without-read case. ## Operational Effect After enforcing fail-closed on orphan writes and requiring revision markers on both halves, the reconciliation job surfaced 340 previously invisible overwrite-without-read operations in a single day — all of them real, all of them the kind of mutation an audit is supposed to catch. The cost was a higher halt rate during the migration window; the mitigation was to declare a compatibility mapping (`gen_ai.memory.entry.id` ← `gen_ai.memory.key` when `gen_ai.memory.operation = "upsert"`) and pin it explicitly in the audit config rather than inferring it. The operational lesson: semantic-convention drift between paired spans is invisible to per-span validation and must be caught at the join, with a fail-closed policy on the mutation side. ```python """Fail-closed reconciliation for paired memory read/write spans.""" from dataclasses import dataclass class UnpairedMemoryWriteError(RuntimeError): """A memory mutation exists with no joinable preceding read. This is an unaudited mutation, not orphan telemetry. Halt the segment. """ @dataclass(frozen=True) class Span: op: str # "read" | "write" schema_version: str # emitter-declared convention revision key_attr: str # attribute name carrying the logical key key_value: str # attribute value def _reconcile(spans: list[Span], mapping: dict[tuple[str, str], str]) -> list[tuple[Span, Span]]: reads: dict[tuple[str, str], Span] = {} for s in spans: if s.op == "read": reads[(s.schema_version, s.key_value)] = s pairs: list[tuple[Span, Span]] = [] for w in (s for s in spans if s.op == "write"): # Try same-revision join first, then declared cross-revision mapping. candidates = [(w.schema_version, w.key_value)] for (from_ver, to_ver), _ in mapping.items(): if w.schema_version == from_ver: candidates.append((to_ver, w.key_value)) match = next((reads[c] for c in candidates if c in reads), None) if match is None: raise UnpairedMemoryWriteError( f"write to {w.key_attr}={w.key_value!r} under schema " f"{w.schema_version!r} has no joinable read; " f"treat as unaudited mutation and halt segment" ) pairs.append((match, w)) return pairs def audit_segment(spans: list[Span], mapping: dict[tuple[str, str], str] | None = None): return _reconcile(spans, mapping or {}) ```