What invariants must survive a retry after a partial failure that was not observed at the source?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session sends a request to an external system, the response times out or fails, and the session cannot determine whether the request was partially or fully applied at the source, requiring a retry whose safety depends on unobserved source state. Does not apply to requests with idempotent sources (dedup by key) or synchronous, atomic operations.

Core Distinction

The separation between **client-side failure** (the client did not receive a response) and **source-side state** (the external system may have applied the request partially, fully, or not at all). Relying on client-side failure creates the **Retry Duplication Hazard**: the client retries, the source has already applied the original request, and the retry creates a duplicate (double-charge, double-write, double-send).

What Was Observed

In agent architectures using non-idempotent external APIs, failing to handle unobserved partial failures produces three primary failure modes:

  1. **Duplicate Transaction:** The agent sends a payment, times out, and retries. Both payments commit, double-charging the customer.
  2. **Over-Write:** The agent sends a state update, times out, and retries. The first update committed, but the retry's payload was different (e.g., a new timestamp), so the second overwrites the first.
  3. **Orphaned Effect:** The agent sends a request that partially succeeded (e.g., 3 of 5 items written), times out, and retries. The retry re-writes the 3 items, but the other 2 are now duplicated or missing.

Recorded Elements

  1. **`original_request_idempotency_key`:** The client-generated key that uniquely identifies the original request, allowing the source to deduplicate retries.
  2. **`source_query_predicate`:** The exact query the agent must execute to determine the source's actual state (e.g., "list all writes by request_id X"), allowing the agent to distinguish "applied" from "not applied" before retrying.
  3. **`partial_application_signature`:** For requests with known partial-application patterns, the signature to detect partial success (e.g., response header indicating partial vs. full).
  4. **`retry_safety_proof`:** The cryptographic proof that a retry is safe (either the source is idempotent, or the source query confirmed no state was applied).

Condition of Validity

* **Applies when:** The source can experience unobserved partial failures, the operations are non-idempotent by default, and duplicates have material consequences. * **Does not apply when:** The source provides a native idempotency key mechanism, or the operations are provably idempotent.

Decision Boundary

  1. **Did the request fail without a clear success or failure signal?** If no, standard error handling applies. If yes, proceed.
  2. **Is the source's state queryable before retry (`source_query_predicate`)?** If yes, query first. If no, the retry must use the `original_request_idempotency_key` to prevent duplication.
  3. **Does the source support idempotency key deduplication?** If yes, retry with the same key. If no, the agent must not retry without a query-first strategy.
  4. **Conversely, when can the agent retry blindly?** Only when the operation is strictly idempotent (repeating has no additional effect), or the operation is strictly read-only.

Operational Effect

  1. **Duplicate Prevention:** Uses idempotency keys or query-first to prevent duplicate side effects.
  2. **Partial Failure Detection:** Detects partial applications, allowing targeted remediation rather than retrying from scratch.
  3. **Deterministic Recovery:** Provides cryptographic proof that the retry is safe.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class RetryVerdict(Enum):
    IDEMPOTENT_SAFE = auto()
    QUERY_FIRST_REQUIRED = auto()
    IDEMPOTENCY_KEY_SAFE = auto()
    UNSAFE_RETRY = auto()

@dataclass(frozen=True)
class RetryState:
    source_supports_idempotency_key: bool
    source_state_queryable: bool
    source_idempotent_by_nature: bool
    has_idempotency_key: bool
    source_query_indicates_prior_success: bool | None

@dataclass(frozen=True)
class RetryPolicy:
    allow_blind_retry: bool

@dataclass(frozen=True)
class RetryManifest:
    verdict: RetryVerdict
    directive: str
    manifest_digest: str

class PartialFailureRetryGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_retry(
        cls, state: RetryState, policy: RetryPolicy
    ) -> RetryManifest:
        
        # Gate 1: Naturally idempotent
        if state.source_idempotent_by_nature:
            digest = cls._hash(json.dumps({
                "idempotent": "NATURAL"
            }, sort_keys=True))
            return RetryManifest(
                verdict=RetryVerdict.IDEMPOTENT_SAFE,
                directive="PROCEED: Source is naturally idempotent.",
                manifest_digest=digest
            )

        # Gate 2: Query-first
        if state.source_state_queryable:
            if state.source_query_indicates_prior_success is True:
                digest = cls._hash(json.dumps({
                    "query": "PRIOR_SUCCESS",
                    "status": "RECONCILED"
                }, sort_keys=True))
                return RetryManifest(
                    verdict=RetryVerdict.QUERY_FIRST_REQUIRED,
                    directive="PROCEED: Source query confirms prior success. Treat as complete, do not retry.",
                    manifest_digest=digest
                )
            if state.source_query_indicates_prior_success is False:
                digest = cls._hash(json.dumps({
                    "query": "NO_PRIOR",
                    "status": "SAFE_TO_RETRY"
                }, sort_keys=True))
                return RetryManifest(
                    verdict=RetryVerdict.QUERY_FIRST_REQUIRED,
                    directive="PROCEED: Source query confirms no prior success. Safe to retry.",
                    manifest_digest=digest
                )
            return RetryManifest(
                verdict=RetryVerdict.QUERY_FIRST_REQUIRED,
                directive="HALT: Query source before retry. Unobserved partial failure risk.",
                manifest_digest=cls._hash("QUERY_PENDING")
            )

        # Gate 3: Idempotency key
        if state.source_supports_idempotency_key and state.has_idempotency_key:
            digest = cls._hash(json.dumps({
                "idempotency_key": "PRESENT"
            }, sort_keys=True))
            return RetryManifest(
                verdict=RetryVerdict.IDEMPOTENCY_KEY_SAFE,
                directive="PROCEED: Retry with idempotency key, source will deduplicate.",
                manifest_digest=digest
            )

        # Gate 4: Unsafe
        return RetryManifest(
            verdict=RetryVerdict.UNSAFE_RETRY,
            directive="HALT: Non-idempotent, non-queryable, no idempotency key. Retry would duplicate.",
            manifest_digest=cls._hash("UNSAFE")
        )