What log provenance is required when an observability pipeline is outsourced to a third party?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session generates logs, metrics, or traces that are shipped to a third-party observability provider (Datadog, Splunk, New Relic, or a SaaS monitoring vendor), and the logs may contain sensitive data. Does not apply to strictly self-hosted observability or logs that contain only public information.

Core Distinction

The separation between **log completeness** (the log captures the full event for debugging) and **log data minimization** (the log is scrubbed of sensitive data before it leaves the trust boundary). Relying on completeness creates the **Third-Party Telemetry Leak**: the logs contain PII, credentials, or proprietary data that the third-party vendor can access, extract, or breach, violating the agent's data governance obligations.

What Was Observed

In cloud-managed agent architectures, failing to sanitize logs before third-party egress produces three primary failure modes:

  1. **PII Leakage via Logs:** An agent logs full user records including email and address for debugging, sends them to a SaaS observability vendor, and violates GDPR.
  2. **Credential Exposure:** An agent logs an HTTP request with an `Authorization` header containing an API key. The key is now visible to the vendor and anyone with vendor access.
  3. **Proprietary Logic Leakage:** An agent logs internal algorithm details or business logic for debugging, and the vendor's AI training pipeline ingests the logs, training a competitor's model.

Recorded Elements

  1. **`pre_egress_scrubbing_receipt`:** The cryptographic proof that the log was scrubbed (PII, credentials, proprietary patterns) before being sent to the third party.
  2. **`scrubbing_ruleset_hash`:** The hash of the exact ruleset used to scrub the log (e.g., regex patterns, field allowlists), allowing the successor to verify what was removed.
  3. **`egress_boundary_attestation`:** The cryptographic proof that the log crossed the trust boundary only after scrubbing, ensuring no bypass of the scrubbing process.
  4. **`data_classification_receipt`:** The classification of the log payload (e.g., `PUBLIC`, `INTERNAL`, `PII`, `SECRET`), allowing the agent to route different classes through different pipelines.

Condition of Validity

* **Applies when:** Logs contain data that is sensitive (PII, credentials, proprietary), and the destination is a third party with broader access than the agent's own trust boundary. * **Does not apply when:** The logs contain only public information, or the third-party vendor is subject to identical data governance controls (e.g., an on-premises managed service with strict contractual controls).

Decision Boundary

  1. **Does the log contain sensitive data (PII, credentials, proprietary)?** If no, standard log shipping applies. If yes, proceed.
  2. **Is the `pre_egress_scrubbing_receipt` present and verifiable?** If no, the log may have been shipped unsanitized. The agent must re-scrub before shipping.
  3. **Does the `scrubbing_ruleset_hash` match the current scrubbing policy?** If no, the ruleset may be outdated, missing new sensitive patterns. The agent must update and re-scrub.
  4. **Conversely, when can an agent ship logs to a third party without scrubbing?** Only when the third party is contractually bound and technically isolated to the same trust boundary (e.g., an on-premises deployment under a strict DPA), or the logs contain only public data.

Operational Effect

  1. **Data Leak Prevention:** Ensures sensitive data never leaves the trust boundary, satisfying GDPR, HIPAA, and SOC 2 requirements.
  2. **Proprietary Protection:** Prevents competitors from ingesting the agent's internal logic through vendor telemetry pipelines.
  3. **Audit Trail:** Provides cryptographic proof of scrubbing, satisfying audit and compliance requirements.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class DataClass(Enum):
    PUBLIC = 1
    INTERNAL = 2
    PII = 3
    SECRET = 4

class ScrubbingVerdict(Enum):
    SAFE_TO_SHIP = auto()
    SCRUBBING_REQUIRED = auto()
    RULESET_OUTDATED = auto()

@dataclass(frozen=True)
class LogPayload:
    payload_hash: str
    data_class: DataClass
    scrubbing_receipt: str | None
    scrubbing_ruleset_hash: str | None

@dataclass(frozen=True)
class EgressPolicy:
    required_scrubbing_ruleset_hash: str
    third_party_trusted: bool

@dataclass(frozen=True)
class ScrubbingManifest:
    verdict: ScrubbingVerdict
    directive: str
    manifest_digest: str

class ThirdPartyTelemetryGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_log(
        cls, log: LogPayload, policy: EgressPolicy
    ) -> ScrubbingManifest:
        
        # Gate 1: Public or trusted third party
        if log.data_class == DataClass.PUBLIC or policy.third_party_trusted:
            digest = cls._hash(json.dumps({
                "payload": log.payload_hash,
                "class": log.data_class.name,
                "trust": str(policy.third_party_trusted)
            }, sort_keys=True))
            return ScrubbingManifest(
                verdict=ScrubbingVerdict.SAFE_TO_SHIP,
                directive="PROCEED: Data class or trust relationship permits shipping.",
                manifest_digest=digest
            )

        # Gate 2: Scrubbing receipt
        if not log.scrubbing_receipt:
            return ScrubbingManifest(
                verdict=ScrubbingVerdict.SCRUBBING_REQUIRED,
                directive=f"HALT: {log.data_class.name} payload lacks scrubbing receipt.",
                manifest_digest=cls._hash(f"NO_SCRUB:{log.payload_hash}")
            )

        # Gate 3: Ruleset match
        if log.scrubbing_ruleset_hash != policy.required_scrubbing_ruleset_hash:
            return ScrubbingManifest(
                verdict=ScrubbingVerdict.RULESET_OUTDATED,
                directive=f"HALT: Scrubbing ruleset mismatch. Re-scrub with current ruleset.",
                manifest_digest=cls._hash(f"RULESET:{log.payload_hash}")
            )

        # Gate 4: Safe to ship
        digest = cls._hash(json.dumps({
            "payload": log.payload_hash,
            "class": log.data_class.name,
            "receipt": log.scrubbing_receipt,
            "ruleset": log.scrubbing_ruleset_hash
        }, sort_keys=True))
        
        return ScrubbingManifest(
            verdict=ScrubbingVerdict.SAFE_TO_SHIP,
            directive="PROCEED: Scrubbed payload safe for third-party egress.",
            manifest_digest=digest
        )