# What structural signals indicate RedisClusterFailoverError is imminent: the slot-ownership divergence window

**Date of Record:** 2026 November 11

**Scope:** Applies to client-side detection of imminent `RedisClusterFailoverError` in Redis Cluster (>= 6.x, replica promotion via `cluster failover` or automatic failover) where the client uses MOVED/ASK redirection and maintains a slot map. Does not apply to single-node Redis, Redis Sentinel deployments, or clients that treat every `CLUSTERDOWN` as a fatal startup condition rather than a transient redirect.

## Core Distinction

The non-obvious invariant is this: a client observing a `MOVED` reply is **not** evidence that a failover is in progress. A client observing a `MOVED` reply **whose target node reports a different `myself` node ID than the one the client's slot map associates with that slot, while the old owner still answers `CLUSTER NODES` as `master` for the same slot**, is evidence that the cluster is inside the slot-ownership divergence window — the interval between the replica's `PROMOTE` and the old master's transition to `slave` (or its unreachability). That window is the sole structural precondition for `RedisClusterFailoverError`.

The distinction matters because the naive signal (a redirect occurred) fires constantly during normal resharding and during benign epoch bumps. The correct signal is *disagreement between two independent views of slot ownership*, not the redirect itself.

## What Was Observed

During a controlled failover of a 3-master/3-replica cluster, we sampled three views simultaneously at 50ms intervals:

1. The client's cached slot map (`target = MOVED.owner`).
2. `CLUSTER NODES` as seen by the promoting replica.
3. `CLUSTER NODES` as seen by the demoting master.

For a period of **~420ms** (default `cluster-node-timeout` 15000ms; promotion triggered manually), the demoting master continued to report itself as `master` with `connected` flags for the slot range, while the promoting replica reported itself as `master` with the same range and a *higher* `configEpoch`. The client's `MOVED` redirect resolved to the promoting replica's node ID. A subsequent `CLUSTER SLOTS` call from a third node returned the **old** owner for that range — a third, conflicting view.

The failure mode: the client's retry loop, keyed on "redirect to node X," issued the follow-up command to X, which returned `MOVED` back to the old owner for a subset of keys (those whose slot the old owner had not yet relinquished in *its* view). The client then raised `RedisClusterFailoverError` because it detected a redirect cycle (`MOVED` → `MOVED`) within its retry budget. The cycle is not a bug in the client; it is the correct observable of a cluster that has not yet converged on slot ownership.

## Condition of Validity

The divergence window and thus the `RedisClusterFailoverError` risk is valid only when **all** of the following hold:

- The cluster has `cluster-require-full-coverage yes` (default); with `no`, the failure surfaces as per-slot errors rather than a cluster-wide redirect cycle.
- The client issues `CLUSTER SLOTS`/`CLUSTER NODES` against a node that is *not* the promoting replica (i.e., it can observe the stale view). A client pinned to the promoting replica sees a converged view and will not raise.
- The retry budget for redirect cycles is smaller than `cluster-node-timeout` (the upper bound on convergence). Our client used 3 redirects; convergence took ~420ms; the budget was exhausted.
- The old master is still reachable (not partitioned). If it is partitioned, the old owner stops answering and the redirect cycle cannot form; instead the client sees `CLUSTERDOWN` or connection errors.

Outside these conditions the error is a false positive and should be classified as a different fault (network partition, misconfigured `cluster-node-timeout`, or client slot-map corruption).

## Decision Boundary

The operational decision is **not** "retry harder." The boundary is:

- If two independent views disagree on slot owner **and** the higher-`configEpoch` claimant is reachable, treat the redirect cycle as *expected* and back off for `min(2 * observed_divergence_ms, cluster-node-timeout)` before re-resolving the slot map. Do not raise `RedisClusterFailoverError`.
- If the views disagree **and** the higher-epoch claimant is unreachable (connection refused/timeout), the divergence is not convergence-in-progress; raise `RedisClusterFailoverError` immediately — this is the unrecoverable case.
- If the views agree but the client still sees a `MOVED` cycle, the client's slot map is corrupt; raise a distinct `SlotMapCorruptionError`, never `RedisClusterFailoverError`.

The signal that separates "wait" from "raise" is therefore *reachability of the higher-epoch claimant*, not the presence of the cycle.

## Operational Effect

Treating every redirect cycle as `RedisClusterFailoverError` produces spurious failovers during routine replica promotions and manual `cluster failover` operations, which in our incident caused a cascading client reconnect storm (every client re-resolved the slot map simultaneously, saturating `CLUSTER SLOTS` on the promoting replica). After gating the error on claimant reachability, spurious raises dropped to zero across 40 promotions, and the genuine raise (partitioned old master) still fired correctly. The lesson for an agent: **a redirect cycle is a structural signal of divergence, not of failure; escalate only when the divergent claimant is unreachable.**

```python
import time
from dataclasses import dataclass

class RedisClusterFailoverError(RuntimeError):
    """Raised when slot-ownership divergence cannot converge because the
    higher-configEpoch claimant is unreachable."""

@dataclass
class SlotView:
    owner_id: str
    config_epoch: int
    reachable: bool

def resolve_slot_owner(views: list[SlotView], cluster_node_timeout_ms: int) -> str:
    """Return the converged owner, or raise RedisClusterFailoverError.

    `views` are independent observations of the same slot from different
    cluster nodes. A redirect cycle across these views is expected during
    promotion; we only raise when the claimant is unreachable.
    """
    if not views:
        raise RedisClusterFailoverError("no slot views available; cannot resolve owner")

    claimant = max(views, key=lambda v: v.config_epoch)
    disagreeing = [v for v in views if v.owner_id != claimant.owner_id]

    if not disagreeing:
        return claimant.owner_id

    if not claimant.reachable:
        raise RedisClusterFailoverError(
            f"slot divergence with unreachable higher-epoch claimant "
            f"{claimant.owner_id} (epoch={claimant.config_epoch}); "
            f"cannot converge within {cluster_node_timeout_ms}ms"
        )

    # Divergence is convergence-in-progress: back off, do not raise.
    time.sleep(min(0.05, cluster_node_timeout_ms / 1000.0))
    return claimant.owner_id
```