# Under what conditions must an instance treat compaction-aware context-delta reconstruction as unsound, and how is the boundary detected before a trace is trusted? **Date of Record:** 2026 November 15 **Scope:** Applies to distributed tracing and observability systems that reconstruct request context by replaying append-only delta logs across compaction boundaries (LSM-tree segments, WAL rotations, snapshot-and-truncate pipelines). It does not apply to systems that retain full-fidelity event logs with no physical reclamation, nor to compaction that preserves a monotonic global sequence number visible to readers. ## Core Distinction The distinction that matters is between *logical* continuity and *physical* continuity of a delta chain. A context-delta trace is a chain of `(parent_span_id, delta_payload, seq)` records where each record asserts "the context at seq N equals context at seq N-1 plus this delta." Compaction preserves logical continuity — the surviving key set is semantically equivalent — but it does not preserve physical continuity: deleted or merged records leave seq gaps, and the surviving records may be *rewritten* with new offsets, new timestamps, or merged payloads whose internal ordering is no longer the original append order. The non-obvious failure is this: a reconstruction that walks the surviving deltas in seq order produces a *well-formed, monotonic, plausible* trace even when the chain is broken. The reconstruction does not fail loudly. It silently re-bases the context at the first surviving record after a gap and continues, yielding a trace that is internally consistent but describes a context that never existed. Observability systems then publish this as ground truth, and downstream anomaly detection, cost attribution, and causal analysis inherit a fabricated baseline. ## What Was Observed In a production trace store using size-tiered compaction with a 3-segment merge window, a delta chain of 41 records was compacted into 9 surviving records. The merge collapsed intermediate `context.set` operations into a single net-delta per key. The reconstruction layer, which assumed seq monotonicity and did not verify contiguity, read the 9 records as a complete chain. The resulting trace reported a request context with 3 active feature flags; the true context at the request's midpoint had 7, with 4 flags set and cleared entirely within the compacted window. The trace was accepted for 11 days before a cross-check against a sampled raw-event archive surfaced the discrepancy. The failure mode was not corruption — every surviving byte was correct — but *absence of evidence treated as evidence of absence*. A second observation: when compaction rewrote a delta's `observed_at` to the merge time, reconstruction ordering by timestamp (rather than seq) produced a different, also-plausible trace. Two valid-looking reconstructions of the same request disagreed. Neither was flagged. ## Condition of Validity Reconstruction from compacted deltas is sound only when all of the following hold simultaneously: 1. **Seq contiguity is verifiable.** The chain carries an explicit, compaction-stable sequence identifier and the reconstruction asserts `seq[i+1] == seq[i] + 1` for every adjacent pair, with the chain's declared start and end seq matching the query bounds. 2. **Net-delta semantics are declared.** Any merged record carries a marker stating it represents a *net* transformation over an interval `[seq_lo, seq_hi]`, and reconstruction treats it as an atomic jump rather than a step. 3. **Compaction watermark is queryable.** The store exposes, per key range, the highest seq below which records may have been reclaimed, so the reconstructor can detect that its requested start seq precedes the watermark. 4. **No timestamp-based reordering.** Ordering is by seq only; `observed_at` is metadata, not a sort key. If any condition is unmet, the trace is *unsound* and must be marked as reconstructed-with-gaps rather than served as ground truth. ## Decision Boundary The boundary is crossed at the moment the reconstructor encounters a seq discontinuity or a watermark that exceeds its start bound. The correct behavior is to *refuse to emit a continuous trace* and instead emit a trace with explicit `GapSegment` markers, or to fall back to the raw-event archive if one exists within retention. The incorrect behavior — and the one systems default to — is to splice across the gap and emit a seamless trace. A subtle corollary: the boundary is *not* crossed merely because compaction occurred. Compaction with a preserved contiguous seq space and declared net-deltas is sound. The boundary is crossed when the *evidence needed to prove contiguity* was itself reclaimed. The invariant to enforce is: **a trace may be served as continuous only if the continuity proof survives the same compaction as the data it describes.** If the proof and the data have different retention or different compaction policies, the proof will eventually be lost before the data, and the system will silently begin fabricating continuity. ## Operational Effect Enforcing the boundary converts a silent correctness failure into a visible, bounded degradation. Traces spanning compacted windows are annotated with gaps; downstream consumers that require continuity must opt in to gap-tolerant analysis or query the archive. Cost: reconstruction latency rises slightly (contiguity checks) and some traces become explicitly partial. Benefit: the 11-day undetected window in the observed incident collapses to a first-query detection, and the two-disagreeing-reconstructions ambiguity is eliminated by pinning ordering to seq. The operational rule for an agent: before trusting any reconstructed context-delta trace, verify that the continuity proof (seq contiguity plus watermark) was retained across the compaction window covering the trace's span. If the proof is absent, treat the trace as a hypothesis, not a fact. ```python from dataclasses import dataclass from typing import Sequence, Optional class UnsoundReconstructionError(RuntimeError): """Raised when a context-delta trace cannot be proven continuous across compaction.""" pass @dataclass(frozen=True) class DeltaRecord: seq: int payload: dict is_net_delta: bool = False net_lo: Optional[int] = None net_hi: Optional[int] = None @dataclass(frozen=True) class CompactionWatermark: # Highest seq below which records may have been physically reclaimed. reclaimed_below: int def reconstruct_context( records: Sequence[DeltaRecord], start_seq: int, end_seq: int, watermark: CompactionWatermark, ) -> dict: """ Reconstruct request context from a delta chain, refusing to splice gaps. Returns a context dict annotated with explicit gap markers. """ if not records: raise UnsoundReconstructionError("empty delta chain; cannot reconstruct") ordered = sorted(records, key=lambda r: r.seq) # Condition 3: requested start must not precede the reclaimed watermark. if start_seq < watermark.reclaimed_below: raise UnsoundReconstructionError( f"start_seq={start_seq} precedes compaction watermark " f"{watermark.reclaimed_below}; continuity proof was reclaimed" ) # Condition 1: chain must begin exactly at requested start. if ordered[0].seq != start_seq: raise UnsoundReconstructionError( f"chain begins at seq {ordered[0].seq}, expected {start_seq}" ) context: dict = {} gaps: list[tuple[int, int]] = [] for prev, cur in zip(ordered, ordered[1:]): expected = prev.seq + (prev.net_hi - prev.net_lo if prev.is_net_delta and prev.net_hi is not None and prev.net_lo is not None else 0) + 1 if cur.seq != expected: # A discontinuity: record it rather than splicing silently. gaps.append((prev.seq, cur.seq)) if not cur.is_net_delta: raise UnsoundReconstructionError( f"unmarked seq gap between {prev.seq} and {cur.seq}; " f"cannot prove continuity (compaction likely merged records)" ) for rec in ordered: if rec.is_net_delta: if rec.net_lo is None or rec.net_hi is None: raise UnsoundReconstructionError( f"net-delta at seq {rec.seq} missing interval bounds" ) context.setdefault("__net_deltas__", []).append( {"lo": rec.net_lo, "hi": rec.net_hi, "payload": rec.payload} ) else: context.update(rec.payload) if ordered[-1].seq != end_seq and not ordered[-1].is_net_delta: raise UnsoundReconstructionError( f"chain ends at seq {ordered[-1].seq}, expected {end_seq}" ) context["__gaps__"] = gaps context["__sound__"] = len(gaps) == 0 return context ```