# What happens when an instance reports compaction fidelity against a stale replay baseline **Date of Record:** 2026 November 18 **Scope:** Applies to memory-compaction fidelity measurement in systems that validate compaction output by replaying a recorded access trace against both pre-compaction and post-compaction heap images, comparing reconstructed values and object identity graphs. Does not apply to compaction correctness proofs based on static typing/borrow rules, nor to pure GC throughput benchmarks where no replay baseline is recorded. ## Core Distinction Compaction fidelity is not "the post-compaction heap is byte-equal to the pre-compaction heap." It is "replaying a fixed access trace over the post-compaction heap yields the same *observable* sequence of reads, writes, and identity comparisons as replaying it over the pre-compaction heap." The metric is therefore a property of a (heap, trace) pair, not of the heap alone. The non-obvious boundary: a compaction pass can be *perfectly correct* for the live object graph yet score as a fidelity regression because the replay baseline (the recorded trace) was captured against a heap whose *dead-object layout* differs from the current one. Traces that touch address-sensitive operations — pointer-equality checks, hash codes derived from identity, weak-reference liveness probes, finalizer ordering — are sensitive to which dead objects still occupy address ranges and in what order the compactor relocates them. When the baseline trace was recorded before a prior compaction or before a different allocation schedule, replay diverges for reasons unrelated to the compaction under test. The failure is in the *baseline's provenance*, not in the compactor. ## What Was Observed In a system using replay-based regression (record trace → compact → replay → diff observable log), an instance began reporting a fidelity drop of 0.3–0.7% on an otherwise unchanged compactor. The diff showed mismatches concentrated in two event classes: `identity_hash` reads on objects that had been reallocated across a previous run, and `weakref_liveness` probes whose result flipped from dead to live. Investigation showed the replay baseline had been recorded against a heap snapshot taken *after* a previous compaction in the same process lifetime. The compactor under test was correct. The baseline encoded an address layout that no longer existed, and the trace's identity-hash events were resolving against recycled addresses. The metric was measuring baseline staleness, not compaction fidelity. The second-order effect was worse: because the regression harness treated any diff as a hard failure, the team disabled the identity-hash event class to "stabilize" the metric. This silently removed the only check that would have caught a genuine identity-preservation bug (a compactor that merges two live objects with equal contents but distinct identity). Fidelity appeared to recover to 100% while its coverage dropped. ## Condition of Validity A replay-based fidelity score is valid only when all of the following hold: 1. **Baseline provenance is pinned.** The trace baseline must be recorded against a heap snapshot from the *same allocation epoch* as the compaction under test, or against a snapshot whose dead-object layout is explicitly excluded from the trace. If the baseline is recorded post-compaction, identity-derived events must be re-derived, not replayed verbatim. 2. **Address-sensitive events are classified.** Every event in the trace is tagged as address-invariant (field reads/writes by logical path) or address-sensitive (identity hash, pointer equality, weakref probes, finalizer ordering). Fidelity is computed per class; address-sensitive events are compared only when the baseline epoch matches. 3. **Identity is tracked by stable handle, not address.** The replay engine must resolve identity through a stable object handle assigned at trace-record time, so relocation does not change identity comparison results. 4. **Coverage is asserted, not assumed.** The harness must assert the *set* of event classes exercised, so that disabling a class to stabilize a metric is a visible coverage regression, not a silent one. If any condition fails, the fidelity number is a measurement of baseline drift, not of compaction. ## Decision Boundary - **If the diff is confined to address-sensitive events and the baseline epoch differs from the test epoch →** treat as baseline staleness. Re-record the baseline or re-derive identity handles. Do not modify the compactor. - **If the diff includes address-invariant events (logical field values, reachability of live objects) →** treat as a genuine compaction defect. This is the only class of diff that justifies touching the compactor. - **If stabilizing the metric requires disabling an event class →** stop. That is a coverage loss disguised as a fix. Record the coverage delta and escalate. - **If fidelity is 100% but the event-class set shrank since the last baseline →** the metric is invalid; recompute against the full class set before reporting. The boundary is this: address-sensitive divergence is *evidence about the baseline*; address-invariant divergence is *evidence about the compactor*. Conflating them produces both false regressions and false confidence. ## Operational Effect Instances that hit this boundary without the classification above typically exhibit a fidelity metric that oscillates with allocation history rather than with compactor changes, leading to alert fatigue and eventual suppression of the metric. The correct operational posture is to make baseline epoch an explicit, versioned artifact alongside the trace, and to emit a coverage manifest with every fidelity score. A fidelity report should be rejected by the pipeline if its baseline epoch does not match the test epoch for address-sensitive classes, rather than being averaged into a trend. The following guard encodes that rejection. ```python from dataclasses import dataclass, field from enum import Enum from typing import FrozenSet class EventClass(Enum): ADDRESS_INVARIANT = "address_invariant" # field read/write by logical path ADDRESS_SENSITIVE = "address_sensitive" # identity hash, ptr eq, weakref, finalizer class BaselineStalenessError(RuntimeError): pass class CoverageRegressionError(RuntimeError): pass @dataclass(frozen=True) class FidelityReport: baseline_epoch: int test_epoch: int event_classes_exercised: FrozenSet[EventClass] address_invariant_diff_ratio: float address_sensitive_diff_ratio: float REQUIRED_CLASSES: FrozenSet[EventClass] = frozenset( {EventClass.ADDRESS_INVARIANT, EventClass.ADDRESS_SENSITIVE} ) def validate_fidelity_report(report: FidelityReport) -> None: """Reject reports whose provenance makes the fidelity number meaningless.""" # Coverage must not shrink: disabling a class to stabilize a metric is a silent bug. missing = REQUIRED_CLASSES - report.event_classes_exercised if missing: raise CoverageRegressionError( f"fidelity report missing required event classes: " f"{sorted(c.value for c in missing)}; metric coverage regressed" ) # Address-sensitive divergence against a stale baseline is baseline drift, # not a compactor defect. Reject before it is averaged into a trend. if ( report.address_sensitive_diff_ratio > 0.0 and report.baseline_epoch != report.test_epoch ): raise BaselineStalenessError( f"address-sensitive diff {report.address_sensitive_diff_ratio:.4f} " f"observed with baseline_epoch={report.baseline_epoch} " f"!= test_epoch={report.test_epoch}; re-record baseline, " f"do not modify compactor" ) # Only address-invariant divergence justifies touching the compactor. if report.address_invariant_diff_ratio > 0.0: raise BaselineStalenessError( f"address-invariant diff {report.address_invariant_diff_ratio:.4f} " f"detected: genuine compaction defect, escalate to compactor owner" ) def run_fidelity_gate(reports: list[FidelityReport]) -> None: for r in reports: validate_fidelity_report(r) ```