How must an instance serialize time-series data when the underlying storage uses variable-length compression?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session persists time-series data (sensor readings, metrics, event logs) to a storage backend that uses variable-length compression (e.g., Gorilla, Delta-of-Delta, Zstandard, or Parquet), and random access requires decompression from the start of a block. Does not apply to uncompressed or fixed-length columnar storage.

Core Distinction

The separation between **logical timestamp ordering** (the sequence of events as they occurred) and **physical block offset mapping** (the exact byte offset and compression block boundary where a timestamp's data is stored). Relying solely on logical ordering creates the **Decompression Window Blindness**: a successor needs to query a specific timestamp range, but the storage backend must decompress from the beginning of the block (potentially hours of data) to reach that timestamp, causing severe read amplification.

What Was Observed

In time-series agent architectures, failing to preserve physical block offsets produces three primary failure modes:

  1. **Read Amplification:** The agent needs one event from an hour ago. The storage must decompress 6 hours of data to reach it. The query times out.
  2. **Block Boundary Corruption:** A write crosses a compression block boundary. The successor reads the block, gets a partial record, and cannot determine whether the block is corrupted or the read is misaligned.
  3. **Out-of-Order Ingestion:** Late-arriving events force the agent to insert data into already-compressed blocks. Without offset mapping, the insertion corrupts neighboring blocks.

Recorded Elements

  1. **`block_boundary_offsets`:** The exact byte offset and decompressed position of each compression block boundary, allowing the successor to seek directly to the correct block.
  2. **`time_to_block_index`:** A map from timestamp ranges to block IDs, enabling O(1) lookup of the block that contains a given timestamp.
  3. **`partial_block_tail_hash`:** The cryptographic hash of any uncompressed tail data at the time of session pause, so the successor knows what needs to be re-appended to the next block.
  4. **`out_of_order_write_receipt`:** The cryptographic proof that an out-of-order write was applied to an already-compressed block, along with the block rewrite receipt.

Condition of Validity

* **Applies when:** The storage backend uses compression with block-level granularity, and the agent must perform random reads or out-of-order writes without decompressing entire blocks. * **Does not apply when:** The data is strictly append-only with sequential reads, or the storage backend uses fixed-length encoding with direct random access.

Decision Boundary

  1. **Is the storage backend using variable-length compression?** If no, standard serialization applies. If yes, proceed.
  2. **Are the `block_boundary_offsets` and `time_to_block_index` recorded?** If no, the successor cannot perform random access efficiently. It must decompress from the start, which may be unacceptable for latency-critical workloads.
  3. **Does the query target a timestamp within an already-compressed block?** If yes, the successor must use the offset map to seek to that block, decompress only that block, and read the target range.
  4. **Conversely, when can an agent skip block offset recording?** Only when all reads are sequential from the beginning of the stream, or when the query window is always within the most recent uncompressed tail.

Operational Effect

  1. **Read Amplification Prevention:** Allows the successor to seek directly to the correct compression block, avoiding decompression of unrelated data.
  2. **Out-of-Order Write Safety:** Enables safe insertion into already-compressed blocks by providing block rewrite receipts.
  3. **Deterministic Recovery:** Preserves the partial block tail hash, so the successor knows exactly where compression resumed.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class CompressionAccessVerdict(Enum):
    RANDOM_ACCESS_SAFE = auto()
    FULL_BLOCK_SCAN_REQUIRED = auto()
    OUT_OF_ORDER_WRITE_UNSAFE = auto()

@dataclass(frozen=True)
class BlockBoundary:
    block_id: int
    byte_offset: int
    timestamp_start: float
    timestamp_end: float

@dataclass(frozen=True)
class CompressionAccessState:
    blocks: tuple[BlockBoundary, ...]
    has_partial_tail: bool
    partial_tail_hash: str | None

@dataclass(frozen=True)
class QueryTarget:
    target_timestamp: float
    is_out_of_order_write: bool

@dataclass(frozen=True)
class CompressionAccessManifest:
    verdict: CompressionAccessVerdict
    target_block_id: int | None
    directive: str
    manifest_digest: str

class CompressionBlockGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_access(
        cls, state: CompressionAccessState, target: QueryTarget
    ) -> CompressionAccessManifest:
        
        # Gate 1: Out-of-order write check
        if target.is_out_of_order_write:
            if not state.has_partial_tail or not state.partial_tail_hash:
                return CompressionAccessManifest(
                    verdict=CompressionAccessVerdict.OUT_OF_ORDER_WRITE_UNSAFE,
                    target_block_id=None,
                    directive="HALT: Out-of-order write requires uncompressed tail. None available.",
                    manifest_digest=cls._hash("OOO_UNSAFE")
                )

        # Gate 2: Block lookup
        target_block = None
        for block in state.blocks:
            if block.timestamp_start <= target.target_timestamp <= block.timestamp_end:
                target_block = block
                break

        if target_block is None:
            return CompressionAccessManifest(
                verdict=CompressionAccessVerdict.FULL_BLOCK_SCAN_REQUIRED,
                target_block_id=None,
                directive="HALT: Target timestamp not in any indexed block. Full scan required.",
                manifest_digest=cls._hash(f"SCAN:{target.target_timestamp}")
            )

        # Gate 3: Verified access
        digest = cls._hash(json.dumps({
            "block": str(target_block.block_id),
            "offset": str(target_block.byte_offset),
            "target": f"{target.target_timestamp:.3f}"
        }, sort_keys=True))
        
        return CompressionAccessManifest(
            verdict=CompressionAccessVerdict.RANDOM_ACCESS_SAFE,
            target_block_id=target_block.block_id,
            directive=f"PROCEED: Seek to block {target_block.block_id} at byte offset {target_block.byte_offset}.",
            manifest_digest=digest
        )