# How must an instance scope correlation identifiers so asynchronous replies are not orphaned?

**Date of Record:** 2026 November 2

**Scope:** Applies to asynchronous request-reply protocols where a reply arrives after the request, correlated by an identifier the requester generates. Does not apply to synchronous transports where reply and request are coincident in a single exchange.

## Core Distinction

A correlation identifier is a key into the requester's pending-reply table. If its scope (per process, per host, per cluster, per tenant) is narrower than the window in which replies can arrive, the reply is valid but the table cannot find the request it answers. The reply is orphaned — dropped, discarded, or delivered to a different pending request that reuses the identifier.

**Invariant.** Correlation is sound if and only if `correlation_scope ⊇ reply_window`, where reply_window is the set of endpoints, processes, and cluster instances that may receive a reply before the pending-reply TTL expires.

**Violation.** The **Orphaned Reply**. The requester deploys, the process restarts, the cluster rebalances; the reply arrives at a new instance whose pending table never saw the request. The reply is dropped as "unknown correlation id". The requester retries, the service processes again, and side effects duplicate — but the duplication is attributed to the retry, not to the orphan.

**Asymmetry.** The reply is well-formed and correctly correlated. Only the correlation table lacks the entry, and the drop is logged as a warning rather than a failure, so the requester does not learn the reply was lost until a timeout fires — if it fires.

**Recorded elements.** The `correlation-scope-receipt` (the scope in which correlation ids are unique), the `reply-window-receipt` (endpoints and duration over which a reply is valid), and the `correlation-reuse-signature` (whether ids are reused across restart) together make the subsumption checkable.

## What Was Observed

**Signature.** Reply timeouts on the requester while the responder reports successful processing. The responder's log shows "replied", the requester's shows "timeout". The gap correlates with requester restarts, deploys, or cluster rebalances, not with network conditions.

**Trace.** Correlation ids scoped per process, restart on deploy every hour. Reply window 5 minutes. A request issued at t=59m, deploy at t=60m, reply arrives t=61m at a fresh process whose pending table is empty. Reply dropped. Request timeout at t=64m, retry at t=65m, second processing at the responder. First processing is orphaned; second is processed; side effects duplicate.

**Why monitoring stays green.** Requester sees a timeout (normal). Responder sees a success (normal). Both metrics are within expected range. Orphaned replies are logged as warnings below the alert threshold.

## Condition of Validity

The instance must ensure at least one holds before it relies on correlation:

1. **Must scope to reply window.** Correlation ids are unique across every process, host, and cluster instance that may receive a reply within the pending-reply TTL.
2. **Must persist across restart.** The pending-reply table survives restart, so a reply arriving after a restart is matched rather than orphaned.
3. **Must fail-closed on unknown correlation.** A reply with unknown correlation id is surfaced as an error rather than dropped as a warning, so the requester learns the reply was lost and does not retry blindly.

## Decision Boundary

- Scope covers reply window, table persists, unknown-correlation is an error: sound.
- Scope covers reply window, table is in-memory: sound in normal operation, orphaned on restart.
- Scope narrower than reply window (per process, per host): orphan is guaranteed on any failover.
- Reused ids across restart: an orphan can be delivered to a *different* pending request, which is worse than orphan — a wrong reply.

## Operational Effect

The invariant couples a key generated at the requester to a table that lives on one or more receivers. The instance's correct posture is to scope ids globally within the reply window and treat unknown-correlation as an error.

**Trap.** Reset the pending table on restart to avoid stale entries. That cures the stale-entry concern and creates the orphan — the correct fix is a persisted table with TTL, not reset-on-restart.

```python
from dataclasses import dataclass


class OrphanedReplyError(RuntimeError):
    """Raised when a correlation-id scope cannot cover the reply window it must span."""


@dataclass(frozen=True)
class CorrelationGate:
    correlation_scope: str  # "process", "host", "cluster", "tenant-global"
    persists_across_restart: bool
    unknown_correlation_is_error: bool

    def verify(self) -> bool:
        if self.correlation_scope == "process":
            raise OrphanedReplyError(
                "process-scoped correlation: replies to a restarted process are orphaned"
            )
        if not self.persists_across_restart:
            raise OrphanedReplyError(
                "pending-reply table does not survive restart: failover orphans replies"
            )
        if not self.unknown_correlation_is_error:
            raise OrphanedReplyError(
                "unknown correlation dropped as warning: reply loss is invisible to requester"
            )
        return True


def dispatch_correlation_mode(gate: CorrelationGate) -> str:
    try:
        gate.verify()
    except OrphanedReplyError:
        return "reconcile-before-retry"
    return "async-correlate"
```