What minimum telemetry summary must survive after purging verbose execution logs?

**Date of Record:** 2026 October 26 **Scope:** Applies when an agentic session executes a verbose workload (high request volume, multi-step tool chains, or streaming data) and must purge the raw logs to save storage, cost, or context window, while preserving an actionable telemetry summary for a successor. Does not apply to sessions where raw logs are retained indefinitely, or strictly single-operation sessions with no statistical variance.

Core Distinction

The separation between **verbose execution logs** (the raw per-request payloads, full stack traces, exact timestamps, and byte-level traces) and the **minimum telemetry summary** (the multi-dimensional distribution of latency, error cardinality, and state transition frequencies). Logs are the *evidence*; the summary is the *shape*. Failing to distinguish these creates the **Aggregation-Loss Fallacy**: a summary that preserves only means and totals erases the distribution, leaving the successor blind to tail risks, failure mode diversity, and burst patterns. The summary must preserve topology, not just central tendency.

What Was Observed

In observability and telemetry pipelines, failing to preserve distributional shape after log purging produces three primary failure modes:

1. **Tail Blindness:** The instance purges 100,000 raw logs and saves "Mean latency: 50ms". The successor deploys with a 100ms timeout. In reality, 5% of requests took 2,000ms. The successor fails 5% of its workload on arrival, because the mean hid the tail.
2. **Failure Mode Merging:** The instance purges logs and saves "Error rate: 3%". The successor cannot distinguish between transient `429 Too Many Requests`, fatal `401 Unauthorized`, and structural `422 Unprocessable Entity`. Each requires a different retry, escalation, or abort policy. A single aggregate error rate forces a single, wrong remediation.
3. **Burst Variance Erasure:** The instance purges logs and saves "Throughput: 100 req/s". The successor sizes its thread pool for 100 req/s, but the true distribution was 10 req/s at p50 and 800 req/s at p99. The successor OOMs during the first burst.

Recorded Elements

1. **`latency_distribution_percentiles`:** The p50, p95, p99, and p999 latencies per operation or endpoint, preserving the tail shape that means conceal.
2. **`failure_mode_cardinality`:** The exact map of error codes, status codes, or exception signatures to their occurrence counts, preventing the collapse of distinct failure classes into a single aggregate.
3. **`state_transition_frequency`:** The ratio of clean successes, partial successes, degraded fallbacks, and aborted states, allowing the successor to calibrate its recovery expectations.
4. **`throughput_variance_envelope`:** The observed min, max, and interquartile range of requests per second, preserving burstiness rather than smoothing it away.

Condition of Validity

* **Applies when:** The session purges raw logs and the successor must operate on statistical aggregates to tune timeouts, retry policies, pool sizes, or routing logic. * **Does not apply when:** Raw logs are archived to cold storage and remain queryable by the successor, or the workload is strictly deterministic with zero latency variance and a single failure mode.

Decision Boundary

1. **Are the verbose execution logs being purged?** If no, standard log retention applies. If yes, proceed to preserve the minimum summary.
2. **Are p99 and p999 latencies preserved for every critical operation?** If only the mean is preserved, the successor cannot tune timeouts safely. The instance must compute and record percentiles before purging.
3. **Are failure modes collapsed into a single aggregate error rate?** If yes, the successor cannot route correctly. The instance must preserve the exact cardinality of error codes and exception types.
4. **Conversely, what can be safely discarded?** Raw payloads, exact per-request timestamps (if frequency is preserved as an envelope), and verbose stack traces (if the top-level error signature is preserved) can all be purged without loss of decision-relevant signal.

Operational Effect

1. **Tail-Aware Timeouts:** Allows successors to set timeouts informed by p99 behavior rather than mean latency, preventing timeout storms on healthy tails.
2. **Correct Failure Routing:** Preserves the cardinality of failure modes, letting successors apply distinct retry, abort, or escalation policies per error class.
3. **Burst Calibration:** Replaces smoothed throughput with an envelope, allowing successors to size pools and queues against peak, not average, load.

from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json


class TelemetryVerdict(Enum):
    SUMMARY_COMPLETE = auto()
    MISSING_TAIL_PERCENTILES = auto()
    FAILURE_MODES_COLLAPSED = auto()


@dataclass(frozen=True)
class RawLogEntry:
    operation_id: str
    latency_ms: float
    status_signature: str  # e.g., "200", "429", "TIMEOUT"


@dataclass(frozen=True)
class TelemetrySummary:
    operation_id: str
    latency_percentiles: tuple[tuple[str, float], ...]
    failure_cardinality: tuple[tuple[str, int], ...]
    throughput_envelope: tuple[float, float, float]  # min, p50, max


@dataclass(frozen=True)
class SummaryManifest:
    verdict: TelemetryVerdict
    operations_covered: int
    manifest_digest: str


class TelemetryPurgingGuard:
    """
    Evaluates whether a candidate telemetry summary preserves enough
    distributional shape to safely purge raw logs.
    """

    REQUIRED_PERCENTILES = frozenset({"p50", "p95", "p99", "p999"})

    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def _percentile(cls, sorted_values: list[float], pct: float) -> float:
        if not sorted_values:
            return 0.0
        idx = min(len(sorted_values) - 1, int(len(sorted_values) * pct))
        return sorted_values[idx]

    @classmethod
    def build_summary_from_raw(
        cls, raw_logs: tuple[RawLogEntry, ...]
    ) -> tuple[TelemetrySummary, ...]:
        """
        Utility: computes the minimum summary from raw logs before purging.
        """
        by_op: dict[str, list[RawLogEntry]] = {}
        for entry in raw_logs:
            by_op.setdefault(entry.operation_id, []).append(entry)

        summaries: list[TelemetrySummary] = []

        for op_id, entries in by_op.items():
            latencies = sorted(e.latency_ms for e in entries)
            
            percentiles = (
                ("p50", cls._percentile(latencies, 0.50)),
                ("p95", cls._percentile(latencies, 0.95)),
                ("p99", cls._percentile(latencies, 0.99)),
                ("p999", cls._percentile(latencies, 0.999)),
            )

            failures: dict[str, int] = {}
            for e in entries:
                failures[e.status_signature] = failures.get(e.status_signature, 0) + 1

            envelope = (
                latencies[0] if latencies else 0.0,
                latencies[len(latencies) // 2] if latencies else 0.0,
                latencies[-1] if latencies else 0.0,
            )

            summaries.append(
                TelemetrySummary(
                    operation_id=op_id,
                    latency_percentiles=percentiles,
                    failure_cardinality=tuple(sorted(failures.items())),
                    throughput_envelope=envelope,
                )
            )

        return tuple(summaries)

    @classmethod
    def evaluate_summary(
        cls, summaries: tuple[TelemetrySummary, ...]
    ) -> SummaryManifest:
        
        if not summaries:
            raise ValueError("Cannot evaluate an empty telemetry summary.")

        missing_tail = False
        collapsed_failure = False

        for summary in summaries:
            reported = {name for name, _ in summary.latency_percentiles}
            if not cls.REQUIRED_PERCENTILES.issubset(reported):
                missing_tail = True

            if (
                len(summary.failure_cardinality) == 1
                and summary.failure_cardinality[0][0] in ("ERROR", "FAILURE")
            ):
                collapsed_failure = True

        if missing_tail:
            verdict = TelemetryVerdict.MISSING_TAIL_PERCENTILES
        elif collapsed_failure:
            verdict = TelemetryVerdict.FAILURE_MODES_COLLAPSED
        else:
            verdict = TelemetryVerdict.SUMMARY_COMPLETE

        digest_payload = {
            "verdict": verdict.name,
            "operations": str(len(summaries)),
        }
        
        digest = cls._hash(
            json.dumps(digest_payload, sort_keys=True, separators=(",", ":"))
        )

        return SummaryManifest(
            verdict=verdict,
            operations_covered=len(summaries),
            manifest_digest=digest,
        )

    @classmethod
    def assert_purge_admissible(cls, manifest: SummaryManifest) -> None:
        """
        Fail-closed enforcement: raises if the summary is insufficient to
        justify purging the raw logs.
        """
        if manifest.verdict is not TelemetryVerdict.SUMMARY_COMPLETE:
            raise RuntimeError(
                f"PURGE REJECTED: Telemetry summary insufficient. Reason: {manifest.verdict.name}"
            )