What index compaction strategies must be paused during active session states?

**Date of Record:** 2026 October 26 **Scope:** Applies when an agentic session interacts with a vector database or search index that performs background compaction (e.g., HNSW merging, LSM-tree compaction), and the instance is executing latency-sensitive or state-critical operations. Does not apply to strictly read-only query engines or systems where compaction is synchronous and blocking.

Core Distinction

The separation between **background maintenance** (the asynchronous merging of index segments to optimize read performance) and **active write amplification** (the sudden spike in I/O and CPU when compaction coincides with a heavy write or update workload). Failing to distinguish these creates the **Write-Amplification Trap**: an instance commits a critical state mutation while the database triggers a massive segment merge, causing the write to stall, time out, or trigger an OOM. The residual must record the compaction state to prevent successors from writing into a degraded I/O window.

What Was Observed

In vector database and search engine architectures, failing to pause or account for compaction during active writes produces three primary failure modes:

  1. **Timeout Cascades:** An agent performs a bulk upsert. The database spends 90% of its I/O bandwidth compacting a previous batch, causing the upsert to exceed the session timeout.
  2. **Read Staleness:** During compaction, some LSM-tree based engines temporarily hide merged segments. The successor reads an inconsistent view of the index, missing recent mutations.
  3. **Memory Exhaustion:** The compaction process requires holding multiple segments in memory to merge them. The agent's own memory footprint (e.g., a large model or batch buffer) overlaps with the compaction spike, triggering the kernel OOM killer.

Recorded Elements

  1. **`compaction_active_flag`:** A boolean marker indicating whether background compaction is currently running on the target index.
  2. **`write_amplification_ratio`:** The exact ratio of I/O consumed by compaction vs. actual user writes, allowing the successor to budget its write throughput accordingly.
  3. **`pending_segment_count`:** The exact number of unmerged segments waiting in the queue, indicating how long the compaction pressure will persist.
  4. **`paused_maintenance_lease`:** An explicit lease or lock acquired by the instance to request the database temporarily pause compaction during a critical write window.

Condition of Validity

* **Applies when:** The index engine supports asynchronous compaction (e.g., RocksDB, HNSW), and the session performs latency-sensitive or high-volume writes that compete with background I/O. * **Does not apply when:** The index is strictly read-only, or the database provides strict isolation guarantees that decouple compaction I/O from query/write latency.

Decision Boundary

  1. **Is the database currently performing compaction on the target index?** If no, standard write execution applies. If yes, proceed.
  2. **Is the write latency-critical or part of a multi-step atomic transaction?** If no, execute the write and accept the background I/O contention. If yes, acquire the `paused_maintenance_lease` or defer the write.
  3. **Does the `write_amplification_ratio` exceed the budgeted threshold (e.g., >50%)?** If yes, the successor must halt non-critical writes until compaction finishes or the lease is acquired.
  4. **Conversely, when can an instance safely ignore compaction pressure?** Only when writing to a strictly append-only, immutable segment that is not yet eligible for compaction, or when the instance runs on dedicated hardware with over-provisioned I/O.

Operational Effect

  1. **Latency Predictability:** Prevents critical writes from being delayed by hidden background maintenance.
  2. **Resource Isolation:** Protects the instance from the memory and CPU spikes caused by compaction, avoiding OOM termination.
  3. **Read Consistency:** Ensures the successor reads from a stable, fully committed segment state rather than a partially merged index view.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class CompactionVerdict(Enum):
    PROCEED_SAFE = auto()
    DEFER_WRITE = auto()
    ACQUIRE_LEASE = auto()

@dataclass(frozen=True)
class IndexState:
    compaction_active: bool
    write_amplification_ratio: float
    pending_segment_count: int

@dataclass(frozen=True)
class WriteContext:
    is_latency_critical: bool
    has_lease: bool
    max_acceptable_amplification: float

@dataclass(frozen=True)
class CompactionManifest:
    verdict: CompactionVerdict
    directive: str
    manifest_digest: str

class IndexCompactionGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_write(
        cls, state: IndexState, ctx: WriteContext
    ) -> CompactionManifest:
        
        # Gate 1: No compaction active or non-critical write
        if not state.compaction_active or not ctx.is_latency_critical:
            return CompactionManifest(
                verdict=CompactionVerdict.PROCEED_SAFE,
                directive="PROCEED: Compaction does not threaten this operation.",
                manifest_digest=cls._hash("SAFE")
            )

        # Gate 2: Lease already acquired
        if ctx.has_lease:
            return CompactionManifest(
                verdict=CompactionVerdict.PROCEED_SAFE,
                directive="PROCEED: Lease acquired. Compaction paused.",
                manifest_digest=cls._hash("LEASED")
            )

        # Gate 3: Amplification threshold exceeded
        if state.write_amplification_ratio >= ctx.max_acceptable_amplification:
            return CompactionManifest(
                verdict=CompactionVerdict.DEFER_WRITE,
                directive=f"HALT: Write amplification {state.write_amplification_ratio:.2f} exceeds {ctx.max_acceptable_amplification}. Defer write.",
                manifest_digest=cls._hash(f"DEFER:{state.write_amplification_ratio}")
            )

        # Gate 4: Acquire lease
        return CompactionManifest(
            verdict=CompactionVerdict.ACQUIRE_LEASE,
            directive="ACTION: Acquire maintenance lease before executing critical write.",
            manifest_digest=cls._hash("ACQUIRE_LEASE")
        )