How should an instance record verified idempotency keys before releasing worker locks?

**Date of Record:** 2026 October 25 **Scope:** Applies when an autonomous instance, distributed worker, or transactional job executor completes an external or database mutation under a distributed lease or mutex and must persist the verified idempotency key before relinquishing the lock. Does not apply to pure read-only queries or stateless computations that produce zero external side effects.

Core Distinction

Releasing a distributed worker lock before the completed operation's idempotency key is durably committed creates a lethal race window between mutual exclusion and deduplication. A critical architectural failure occurs when an instance executes an upstream side effect, releases its worker lock (or allows the lease to expire), and only subsequently attempts to write the idempotency completion record—allowing a competing worker to acquire the freed lock in the intervening milliseconds, observe no recorded idempotency receipt, and execute the non-idempotent mutation a second time.

What Was Observed

In distributed agentic queues and webhook-dispatch pipelines where lock release and idempotency ledger updates are decoupled, reversing or relaxing the commit order produces four distinct operational hazards:

1. **Unlock-Before-Ledger Race Window:** Worker A finishes an API mutation, releases the mutex, and suffers a process pause or crash before flushing the idempotency key to disk; Worker B immediately acquires the mutex, sees an unrecorded key, and double-charges or double-dispatches the payload.
2. **Expired Fencing Token Commit:** Worker A experiences a GC or network stall while executing the side effect; its lock lease expires, and it writes the idempotency record *after* the lease has already been granted to Worker B, corrupting the deduplication state with a stale fencing token.
3. **Unverified Receipt Persistence:** Recording the idempotency key as `COMPLETED` before verifying the upstream response digest (`response_payload_sha256` and terminal HTTP/RPC status) permanently suppresses legitimate retries when the upstream call actually failed in transit.
4. **Volatile Cache Deduplication Amnesia:** Writing the idempotency receipt into an ephemeral in-memory cache (`LRU`) without synchronous write-ahead persistence causes lock-handoff peers on separate nodes to miss the completion marker entirely.

Recorded Elements

1. **Fenced Idempotency Receipt:** An immutable tuple of `(idempotency_key, resource_lock_id, monotonic_fencing_token, upstream_response_sha256, committed_at_monotonic_ms, durable_fsync_confirmed)` persisted while the lock remains actively held.
2. **Lease Margin Verification:** Proving that `lock_lease_expiry_ms - current_monotonic_ms > minimum_safety_margin_ms` at the exact instant the idempotency ledger entry is committed.
3. **Strict Ordering Barrier:** Enforcing the invariant sequence `Acquire Lock -> Execute & Verify Side Effect -> Durably Commit Idempotency Receipt (under valid Fencing Token) -> Release Lock`.

Condition of Validity

* **Applies when:** Coordinating distributed worker locks, leases, or database row locks around side-effecting operations that rely on idempotency keys to prevent duplicate execution. * **Does not apply when:** Releasing locks after an aborted pre-check where zero external calls or state mutations were dispatched.

Decision Boundary

1. **Has the verified idempotency receipt (including `upstream_response_sha256` and `durable_fsync_confirmed == True`) been committed to persistent storage while the worker lock is still held?** Never release the worker lock until durable persistence of the idempotency key is confirmed.
2. **Is the worker's `monotonic_fencing_token` still the highest active token for `resource_lock_id` with positive remaining lease time?** If the lease expired or was preempted before the idempotency key could be recorded, abort the clean unlock and raise a fenced-lock preemption violation for reconciliation.
3. **Does the idempotency receipt bind the exact parameter digest (`request_payload_sha256`) of the executed operation?** Reject any unlock request whose recorded idempotency key lacks a cryptographic link to both the request parameters and verified response.

Operational Effect

1. **Zero Lock-Handoff Duplicate Execution:** Any successor instance acquiring the lock is guaranteed to see the committed idempotency receipt immediately upon lock acquisition.
2. **Split-Brain & Zombie Worker Immunity:** Validating monotonic fencing tokens and lease margins prior to ledger commit and lock release prevents stalled workers from silently corrupting deduplication state.

from dataclasses import dataclass
from typing import Final
import hashlib

@dataclass(frozen=True)
class VerifiedIdempotencyReceipt:
    idempotency_key: str
    resource_lock_id: str
    fencing_token: int
    request_payload_sha256: str
    response_payload_sha256: str
    committed_monotonic_ms: float
    durable_storage_confirmed: bool

@dataclass(frozen=True)
class ActiveWorkerLockState:
    resource_lock_id: str
    holder_fencing_token: int
    lease_expiry_monotonic_ms: float
    lock_currently_held: bool

class PrematureLockReleaseViolation(Exception):
    """Raised when a worker attempts to release a lock before durably recording a fenced idempotency receipt."""
    pass

class PreUnlockIdempotencyGate:
    def __init__(self, min_lease_margin_ms: float = 50.0) -> None:
        self.min_lease_margin_ms: Final[float] = min_lease_margin_ms

    def authorize_lock_release(
        self,
        lock_state: ActiveWorkerLockState,
        receipt: VerifiedIdempotencyReceipt,
        current_monotonic_ms: float,
    ) -> str:
        # 1. Lock must still be actively held when verifying the idempotency receipt
        if not lock_state.lock_currently_held:
            raise PrematureLockReleaseViolation(
                f"Lock '{lock_state.resource_lock_id}' was already released prior to idempotency verification."
            )

        # 2. Receipt must match the exact lock resource and active monotonic fencing token
        if receipt.resource_lock_id != lock_state.resource_lock_id:
            raise PrematureLockReleaseViolation(
                f"Receipt lock ID '{receipt.resource_lock_id}' mismatches active lock '{lock_state.resource_lock_id}'."
            )
        if receipt.fencing_token != lock_state.holder_fencing_token or receipt.fencing_token <= 0:
            raise PrematureLockReleaseViolation(
                f"Fencing token mismatch on lock '{lock_state.resource_lock_id}': "
                f"receipt={receipt.fencing_token}, lock={lock_state.holder_fencing_token}."
            )

        # 3. Verify durable storage flush completed before lease expiration
        if not receipt.durable_storage_confirmed:
            raise PrematureLockReleaseViolation(
                f"Idempotency key '{receipt.idempotency_key}' is not durably flushed; refusing lock release."
            )
        if receipt.committed_monotonic_ms > current_monotonic_ms:
            raise PrematureLockReleaseViolation(
                "Receipt commit timestamp is in the future relative to current monotonic clock."
            )
        remaining_lease_ms = lock_state.lease_expiry_monotonic_ms - current_monotonic_ms
        if remaining_lease_ms < self.min_lease_margin_ms:
            raise PrematureLockReleaseViolation(
                f"Lock lease for '{lock_state.resource_lock_id}' expired or within unsafe margin "
                f"({remaining_lease_ms:.1f}ms < {self.min_lease_margin_ms:.1f}ms) before unlock."
            )

        # 4. Require non-empty request and response digests bound to the idempotency key
        if not receipt.idempotency_key.strip() or not receipt.request_payload_sha256.strip() or not receipt.response_payload_sha256.strip():
            raise PrematureLockReleaseViolation(
                "Idempotency receipt is missing key, request digest, or verified response digest."
            )

        release_token = (
            f"{receipt.resource_lock_id}|{receipt.fencing_token}|"
            f"{receipt.idempotency_key}|{receipt.response_payload_sha256}"
        )
        return hashlib.sha256(release_token.encode("utf-8")).hexdigest()