What embedding metadata invariants must survive schema evolution in vector indices?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session queries or mutates a vector index whose embedding metadata schema has evolved (e.g., additional fields, changed field types, or deprecated fields), and the session must interpret retrieved vectors' metadata consistently despite the schema mismatch. Does not apply to vector indices with strictly immutable metadata schemas or migrations that atomically transform all existing vectors.

Core Distinction

The separation between **vector similarity** (the vector's mathematical proximity to the query) and **metadata interpretation** (the meaning of the fields attached to the vector under its schema version). Relying on similarity creates the **Metadata Version Confusion**: the session retrieves a vector whose metadata was stored under schema v1, but interprets it under schema v2, silently misinterpreting fields that changed type, meaning, or default value, producing incorrect downstream decisions.

What Was Observed

In evolving vector index architectures, failing to account for metadata schema versions produces three primary failure modes:

  1. **Type Confusion:** Schema v1 stored `timestamp` as a Unix epoch integer. Schema v2 stores it as an ISO 8601 string. The session reads an integer, treats it as a string, and downstream parsing fails silently.
  2. **Semantic Shift:** Schema v1's `priority` field was 1-5. Schema v2 redefines it as 1-10. The session reads a v1 vector's priority of 3, interprets it as mid-priority under v2, but it was actually high-priority under v1.
  3. **Missing Fields:** Schema v2 requires new fields (e.g., `tenant_id`). V1 vectors lack these fields. The session treats missing fields as null, causing cross-tenant contamination in queries.

Recorded Elements

  1. **`metadata_schema_version_signature`:** For each vector, the cryptographic signature of the metadata schema version it was indexed under, allowing version-aware interpretation.
  2. **`field_mapping_matrix`:** The exact mapping of fields between schema versions (field name, type, meaning, and default), allowing the session to translate v1 metadata to v2 semantics.
  3. **`default_value_receipt`:** For fields that are absent in older versions, the exact default value to apply, signed by the schema owner, so absent fields are interpreted consistently.
  4. **`cross_version_query_proof`:** For queries that span multiple schema versions, the cryptographic proof that the session applied consistent interpretation (e.g., translated all v1 vectors to v2 semantics before comparison).

Condition of Validity

* **Applies when:** The vector index contains vectors from multiple metadata schema versions, and the session's decisions depend on accurate metadata interpretation. * **Does not apply when:** The metadata schema is immutable, or the index supports only one schema version at a time (via atomic migrations).

Decision Boundary

  1. **Does the index contain multiple metadata schema versions?** If no, standard interpretation applies. If yes, proceed.
  2. **Is the `metadata_schema_version_signature` present for each vector?** If no, the session cannot determine which schema to apply. It must treat the metadata as `UNVERIFIED`.
  3. **Is there a `field_mapping_matrix` that allows translation between schema versions?** If no, the session cannot safely compare cross-version vectors. It must filter to a single schema version.
  4. **Conversely, when can the session ignore schema versions?** Only when the schema evolution is strictly additive (new optional fields with no semantic impact on existing fields), or the session queries only vectors from a single schema version.

Operational Effect

  1. **Version-Aware Interpretation:** Prevents silent metadata misinterpretation across schema versions.
  2. **Consistent Cross-Version Queries:** Enables queries that span versions by translating to a common semantics.
  3. **Field Absence Discipline:** Uses signed defaults for missing fields, preventing cross-tenant contamination.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class MetadataVerdict(Enum):
    SINGLE_VERSION = auto()
    CROSS_VERSION_TRANSLATED = auto()
    NO_TRANSLATION_MATRIX = auto()
    UNVERIFIED_VERSION = auto()

@dataclass(frozen=True)
class VectorMetadata:
    vector_id: str
    schema_version: str
    schema_signature: str | None

@dataclass(frozen=True)
class SchemaContext:
    vectors: tuple[VectorMetadata, ...]
    current_query_schema_version: str
    field_mapping_available: bool

@dataclass(frozen=True)
class MetadataPolicy:
    require_translation_for_cross_version: bool

@dataclass(frozen=True)
class MetadataManifest:
    verdict: MetadataVerdict
    directive: str
    manifest_digest: str

class EmbeddingMetadataGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_metadata(
        cls, ctx: SchemaContext, policy: MetadataPolicy
    ) -> MetadataManifest:
        
        # Gate 1: Signature presence
        for v in ctx.vectors:
            if not v.schema_signature:
                return MetadataManifest(
                    verdict=MetadataVerdict.UNVERIFIED_VERSION,
                    directive=f"HALT: Vector {v.vector_id} lacks schema version signature.",
                    manifest_digest=cls._hash(f"NO_SIG:{v.vector_id}")
                )

        # Gate 2: Version diversity check
        versions = {v.schema_version for v in ctx.vectors}

        # Gate 3: Single version
        if len(versions) == 1 and ctx.current_query_schema_version in versions:
            digest = cls._hash(json.dumps({
                "version": ctx.current_query_schema_version,
                "vectors": str(len(ctx.vectors))
            }, sort_keys=True))
            return MetadataManifest(
                verdict=MetadataVerdict.SINGLE_VERSION,
                directive=f"PROCEED: All vectors are schema version {ctx.current_query_schema_version}.",
                manifest_digest=digest
            )

        # Gate 4: Cross-version translation
        if policy.require_translation_for_cross_version and not ctx.field_mapping_available:
            return MetadataManifest(
                verdict=MetadataVerdict.NO_TRANSLATION_MATRIX,
                directive=f"HALT: Vectors span {len(versions)} schema versions, no translation matrix.",
                manifest_digest=cls._hash(f"NO_MATRIX:{len(versions)}")
            )

        # Gate 5: Translated
        digest = cls._hash(json.dumps({
            "versions": tuple(sorted(versions)),
            "query_version": ctx.current_query_schema_version,
            "vectors": str(len(ctx.vectors)),
            "translation": str(ctx.field_mapping_available)
        }, sort_keys=True))
        
        return MetadataManifest(
            verdict=MetadataVerdict.CROSS_VERSION_TRANSLATED,
            directive=f"PROCEED: Translated {len(versions)} schema versions to {ctx.current_query_schema_version}.",
            manifest_digest=digest
        )