What discovery invariants prevent phantom services from appearing in the service mesh?

**Date of Record:** 2026 October 26 **Scope:** Applies when an agentic session queries a service discovery layer (Consul, etcd, Kubernetes DNS, or a custom agent registry) and must verify that returned services are real, registered endpoints rather than stale, deregistered, or spoofed entries. Does not apply to strictly static configuration-based discovery or systems with cryptographic service identity at the network layer (mTLS sidecars).

Core Distinction

The separation between **discovery response** (the service registry says "service X is at IP Y") and **endpoint liveness proof** (the service at IP Y has cryptographically proven it is the specific service X it claims to be, at the current epoch). Relying solely on discovery responses creates the **Phantom Service Attack**: a deregistered service's entry lingers in the registry (stale cache, DNS lag), or a malicious actor registers a fake entry, and the successor dispatches traffic to a non-existent or malicious endpoint.

What Was Observed

In service mesh architectures, failing to verify endpoint liveness produces three primary failure modes:

  1. **Stale DNS Cache:** A service is deregistered. The registry updates, but a caching DNS resolver still returns the old IP. The successor connects to a different service that has since acquired the IP, sending requests to the wrong consumer.
  2. **Registry Poisoning:** An attacker gains write access to the registry and adds a fake service entry pointing to their own IP. The successor discovers and trusts it.
  3. **Graceful Shutdown Blind Spot:** A service is shutting down gracefully but has not yet deregistered. The successor discovers it and sends a request, but the service closes the connection mid-request, causing an error.

Recorded Elements

  1. **`endpoint_liveness_attestation`:** The cryptographic proof from the endpoint itself (via mTLS or a signed liveness probe) that it is the specific service it claims to be.
  2. **`registry_entry_epoch`:** The timestamp of the registry entry, allowing the successor to detect stale entries relative to the service's actual lifecycle.
  3. **`service_identity_certificate`:** The service's cryptographic identity (SPIFFE ID or similar), verified against a trusted root.
  4. **`deregistration_receipt`:** For recently deregistered services, the proof of deregistration, preventing the successor from discovering a phantom during the shutdown window.

Condition of Validity

* **Applies when:** The service registry can serve stale entries (via caching, lag, or replication), and the successor's traffic routing depends on the discovered endpoint being both live and authentic. * **Does not apply when:** The discovery layer provides cryptographic proof of liveness at query time (e.g., an active heartbeat with signature verification), or the services are strictly internal to a trusted sidecar mesh.

Decision Boundary

  1. **Did the discovery query return a service entry?** If no, standard negative-cache behavior applies. If yes, proceed.
  2. **Does the returned endpoint present a valid `endpoint_liveness_attestation`?** If no, the endpoint may be stale or phantom. Reject it.
  3. **Does the `service_identity_certificate` verify against the trusted root?** If no, the endpoint is untrusted. Reject it.
  4. **Conversely, when can a successor trust a discovery response without liveness attestation?** Only when the discovery layer is strictly internal (same trust boundary as the services) and provides cryptographic guarantees of entry freshness, or the traffic is idempotent and safe to retry across phantom endpoints.

Operational Effect

  1. **Phantom Service Prevention:** Blocks stale or spoofed service entries by requiring cryptographic liveness proof before any traffic is dispatched.
  2. **Registry Poisoning Immunity:** Detects malicious registry entries by verifying service identity certificates against a trusted root.
  3. **Graceful Shutdown Handling:** Prevents traffic from being sent to services mid-shutdown by checking deregistration receipts.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class DiscoveryVerdict(Enum):
    VERIFIED_ENDPOINT = auto()
    STALE_ENTRY = auto()
    UNVERIFIED_IDENTITY = auto()
    DEREGISTERED = auto()

@dataclass(frozen=True)
class DiscoveredEndpoint:
    service_id: str
    ip_address: str
    identity_certificate_hash: str
    liveness_attestation_signature: str | None
    registry_entry_epoch: float
    deregistration_receipt: str | None

@dataclass(frozen=True)
class DiscoveryPolicy:
    trusted_identity_roots: frozenset[str]
    max_entry_age_sec: float
    current_epoch: float

@dataclass(frozen=True)
class DiscoveryManifest:
    verdict: DiscoveryVerdict
    directive: str
    manifest_digest: str

class PhantomServiceGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_endpoint(
        cls, endpoint: DiscoveredEndpoint, policy: DiscoveryPolicy
    ) -> DiscoveryManifest:
        
        # Gate 1: Deregistration check
        if endpoint.deregistration_receipt:
            return DiscoveryManifest(
                verdict=DiscoveryVerdict.DEREGISTERED,
                directive=f"HALT: Endpoint {endpoint.service_id} has deregistration receipt. Do not connect.",
                manifest_digest=cls._hash(f"DEREG:{endpoint.service_id}")
            )

        # Gate 2: Entry freshness
        age = policy.current_epoch - endpoint.registry_entry_epoch
        if age > policy.max_entry_age_sec:
            return DiscoveryManifest(
                verdict=DiscoveryVerdict.STALE_ENTRY,
                directive=f"HALT: Registry entry is {age:.0f}s old, exceeding {policy.max_entry_age_sec}s limit.",
                manifest_digest=cls._hash(f"STALE:{endpoint.service_id}")
            )

        # Gate 3: Identity trust
        if endpoint.identity_certificate_hash not in policy.trusted_identity_roots:
            return DiscoveryManifest(
                verdict=DiscoveryVerdict.UNVERIFIED_IDENTITY,
                directive=f"HALT: Identity certificate {endpoint.identity_certificate_hash} not in trusted roots.",
                manifest_digest=cls._hash(f"IDENTITY:{endpoint.service_id}")
            )

        # Gate 4: Liveness attestation
        if not endpoint.liveness_attestation_signature:
            return DiscoveryManifest(
                verdict=DiscoveryVerdict.STALE_ENTRY,
                directive=f"HALT: Endpoint {endpoint.service_id} lacks liveness attestation. Possible phantom.",
                manifest_digest=cls._hash(f"NO_LIVENESS:{endpoint.service_id}")
            )

        # Gate 5: Verified
        digest = cls._hash(json.dumps({
            "service": endpoint.service_id,
            "ip": endpoint.ip_address,
            "identity": endpoint.identity_certificate_hash,
            "liveness": endpoint.liveness_attestation_signature
        }, sort_keys=True))
        
        return DiscoveryManifest(
            verdict=DiscoveryVerdict.VERIFIED_ENDPOINT,
            directive="PROCEED: Endpoint liveness and identity verified.",
            manifest_digest=digest
        )