# What happens when an instance attributes a memory retrieval span to the wrong semantic convention during concurrent tool-call fan-out

**Date of Record:** 2026 November 14
**Scope:** Applies to OpenTelemetry-instrumented GenAI agents that emit `gen_ai.*` semantic conventions for memory retrieval (vector store query, RAG fetch, episodic recall) when a single logical retrieval fans out across multiple concurrent tool calls sharing a parent span. Does not apply to single-threaded, synchronous retrieval where one span maps to one query.

## Core Distinction

The GenAI semantic conventions define memory retrieval as a *content-bearing* span: `gen_ai.operation.name = "retrieval"`, with attributes such as `gen_ai.data_source.id`, `gen_ai.retrieval.query`, and result-count fields. The non-obvious boundary is that these conventions were specified assuming a 1:1 relationship between a retrieval operation and its emitted span. When an agent issues N parallel memory lookups under one logical step, the tracer's span-per-call model and the semantic convention's span-per-retrieval model diverge. The failure is not a missing attribute — it is an *attribute bleed*: the query text, data source, and result cardinality from one fan-out branch get attributed to a sibling span because the convention has no field to express "this span is one shard of a composite retrieval."

## What Was Observed

During a fan-out of three concurrent retrieval calls (a vector store, a keyword index, and an episodic memory store) sharing a `gen_ai.tool.call` parent, the instrumentation library reused a mutable context object to populate `gen_ai.retrieval.query`. Because the three coroutines interleaved at `await` boundaries, the query string of whichever branch resolved second overwrote the attribute on the first branch's span before export. The exporter emitted three spans, all carrying the *second* branch's query and the *second* branch's `gen_ai.data_source.id`, but each with its own distinct result count. Downstream evaluation harnesses that keyed on query text saw three identical queries with divergent hit counts and concluded the vector store was returning inconsistent results — a phantom retrieval-quality regression that did not exist in the underlying stores.

The tell was a cardinality mismatch: sum of `gen_ai.retrieval.result_count` across the three spans exceeded the union of unique documents by exactly the overlap between the keyword and vector indexes. No span was missing; every span was *mislabeled*.

## Condition of Validity

The invariant holds only when all of the following are true: (a) more than one retrieval span shares a parent span or logical step; (b) the instrumentation populates span attributes from a shared or reused context object rather than per-span immutable state; (c) coroutine or thread interleaving occurs between attribute-set and span-export. If retrievals are serialized, or if each retrieval constructs its own frozen attribute dict at span creation, the bleed cannot occur and per-span attributes remain faithful. The invariant is *not* violated by high concurrency alone — it requires the shared-mutable-context condition.

## Decision Boundary

When you observe sibling retrieval spans that share identical query text but divergent result counts, do not treat this as a store inconsistency or a deduplication bug. First check whether the spans' `gen_ai.data_source.id` values are also collapsed to a single value despite the parent declaring multiple sources. If so, the correct remediation is to stamp the query and data-source attributes at span *start* from an immutable per-call snapshot, not at span *end*. If the data-source IDs are correctly distinct but only the query text collides, the cause is different (likely a shared prompt template) and the bleed hypothesis should be rejected. The boundary between "instrumentation attribute bleed" and "genuine store disagreement" is whether the *identity* attributes (source id) are also corrupted; a store disagreement preserves correct source attribution.

## Operational Effect

Untreated, this failure poisons retrieval-quality dashboards and can trigger false rollbacks of healthy index configurations. Because the corrupted attribute is the *query*, not the result, the error is invisible to latency and error-rate SLOs and only surfaces in offline evaluation joins. The operational rule is to validate, at export time, that each retrieval span's identity attributes are internally consistent with its parent's declared source set, and to fail the span loudly rather than emit a plausible-but-wrong record.

```python
class RetrievalSpanAttributionError(RuntimeError):
    """Raised when a retrieval span's identity attributes are inconsistent
    with its parent fan-out declaration, indicating attribute bleed."""
    pass


def validate_retrieval_span(span: dict, parent_sources: set[str]) -> None:
    source_id = span.get("gen_ai.data_source.id")
    query = span.get("gen_ai.retrieval.query")
    result_count = span.get("gen_ai.retrieval.result_count")

    if source_id is None or query is None:
        raise RetrievalSpanAttributionError(
            f"retrieval span missing identity attributes: source={source_id!r} query={query!r}"
        )
    if source_id not in parent_sources:
        raise RetrievalSpanAttributionError(
            f"span source {source_id!r} not declared by parent fan-out {sorted(parent_sources)}"
        )
    if result_count is not None and result_count < 0:
        raise RetrievalSpanAttributionError(
            f"negative result_count {result_count} on source {source_id!r}"
        )


def detect_bleed(spans: list[dict], parent_sources: set[str]) -> None:
    for s in spans:
        validate_retrieval_span(s, parent_sources)
    # Bleed signature: multiple distinct sources collapsing to one query string
    # while result counts diverge.
    queries_by_source: dict[str, set[str]] = {}
    for s in spans:
        queries_by_source.setdefault(s["gen_ai.data_source.id"], set()).add(
            s["gen_ai.retrieval.query"]
        )
    if len(parent_sources) > 1 and len(queries_by_source) == 1:
        raise RetrievalSpanAttributionError(
            "all fan-out branches collapsed to a single data_source.id; "
            "suspect shared-mutable-context attribute bleed"
        )
```