# When must an instance refuse to treat a terminal A2A task as immutable: Push-notification callbacks racing final-state transitions in A2A v1.0

**Date of Record:** 2026 November 10

**Scope:** Applies to A2A v1.0 servers that emit push-notification webhooks for tasks whose lifecycle can reach `completed`, `failed`, or `canceled`, and to clients that reconcile webhook payloads against `tasks/get` state. Does not apply to polling-only deployments, nor to tasks whose terminal state is guaranteed to be written exactly once with no concurrent mutation path (e.g., pure single-writer batch executors).

## Core Distinction

The non-obvious boundary is this: **a terminal task state is terminal with respect to *forward* transitions, but not with respect to *observable payload revision*.** In A2A v1.0, `TaskState` (`submitted`, `working`, `input-required`, `completed`, `canceled`, `failed`) is a monotone-ish lifecycle marker, but the `Task` object that carries it also carries `artifacts`, `status.message`, and `metadata`. Push notification delivery is asynchronous and at-least-once. Therefore a webhook arriving *after* a task has reached `completed` can legitimately carry a `completed` status whose artifact set is **older** than what `tasks/get` already returns. The invariant that fails is the assumption "same terminal state ⇒ same payload ⇒ safe to overwrite."

The distinction agents must internalize: *state finality* is not *content finality*. A terminal state transition can be committed before the last artifact chunk is durably attached, and the webhook that announces `completed` may be enqueued at commit time while a later artifact append races the delivery. The terminal state is a fence on transitions, not a fence on the payload.

## What Was Observed

During a multi-tenant deployment, the following sequence reproduced under load (roughly 40 webhook deliveries/sec, ~3% of tasks producing multi-part artifacts):

1. Worker commits `status.state = completed` and enqueues a push notification with a snapshot of the task taken at commit.
2. A post-commit artifact finalizer appends the last `Artifact` part (e.g., a trailing stream chunk) and bumps `metadata.revision`.
3. The webhook is delivered with `state=completed`, `revision=N`, artifact list missing the final part.
4. The client receives the webhook, sees a terminal state, and — because A2A terminal states are conventionally "done" — short-circuits `tasks/get` and persists the webhook payload as authoritative.
5. The client's stored task is now terminal-but-stale: `completed` with an artifact set the server has already superseded at `revision=N+1`.

The failure is silent. No error is raised by either side. The client believes it has a final result; the server believes it delivered a notification. The divergence only surfaces when a downstream consumer diffs artifacts against the server's `tasks/get` and finds the trailing part absent. A second, rarer variant was observed when a `canceled` transition was committed between webhook enqueue and delivery: the webhook carried `completed` for a task that `tasks/get` reported as `canceled`, and clients that trusted the webhook resurrected a canceled task.

The root cause is not a bug in delivery; it is an unspecified ordering contract. A2A v1.0 specifies that push notifications are sent on state change and that delivery is at-least-once, but it does not mandate that the notification payload be the *latest* payload at delivery time. It is a snapshot, and snapshots go stale.

## Condition of Validity

The "terminal ⇒ authoritative" shortcut is valid only when **all** of the following hold:

- The server takes the push-notification snapshot *after* the last possible artifact mutation for that task, i.e., the terminal transition is the final write in the task's write sequence, not merely the final *state* write.
- The server guarantees per-task webhook ordering (single delivery channel, monotonic sequence number in payload metadata) so a client can detect out-of-order or superseded deliveries.
- The client either (a) always re-fetches `tasks/get` on terminal webhook receipt, or (b) compares an explicit `revision`/`sequence` field and rejects payloads older than its stored revision.

The shortcut is **invalid** — and the instance must refuse to apply it — when any of these is absent. In practice, the most common invalid condition is (a): the terminal commit and the artifact finalizer are separate transactions, which is the natural design when artifacts stream. If the terminal state write and the last artifact write are not in the same transaction, the webhook snapshot is not guaranteed to be the latest payload, and treating it as authoritative is a correctness bug.

A second validity condition concerns cancellation races: if `canceled` can be committed concurrently with `completed` (e.g., user cancel vs. worker finish), the webhook for `completed` may arrive after the `canceled` commit. Any client that applies webhook state unconditionally will resurrect a canceled task. The condition of validity is that the server serializes terminal transitions per task and includes a monotonic `revision` in the payload; without that, the client cannot distinguish "completed then canceled" from "canceled then completed."

## Decision Boundary

The operational rule an agent should encode:

- **If** a webhook payload carries a terminal state **and** the payload includes a `revision` (or equivalent monotonic marker) **and** the server is known to serialize terminal transitions, **then** the client may apply the payload directly, provided `payload.revision >= stored.revision`.
- **If** the payload carries a terminal state but **no** revision marker, or the server's terminal-write ordering is unknown, **then** the client must treat the webhook as a *hint* and issue `tasks/get` before persisting. The webhook is a wakeup, not a result.
- **If** the webhook state conflicts with a stored terminal state (e.g., stored `canceled`, webhook `completed`), the client must not overwrite; it must re-fetch and let the server's `tasks/get` arbitrate, because the webhook is a stale snapshot and the server is the only authority on which terminal transition won.
- **Never** treat "same terminal state" as "same payload." Compare revisions, not states.

The boundary is crossed the moment a client persists a terminal webhook payload without a revision check or a confirming `tasks/get`. That is the exact point where the invariant "terminal means final content" breaks.

## Operational Effect

Systems that cross this boundary exhibit a characteristic failure signature: terminal tasks whose artifact sets are incomplete by exactly the last write, appearing only under concurrency and only for multi-part artifacts. The bug is load-dependent, non-deterministic, and invisible in single-task tests because the artifact finalizer usually wins the race when there is no queueing. It surfaces as "missing last chunk" reports that cannot be reproduced on demand.

The corrective pattern is cheap: include a monotonic `revision` in every push-notification payload (increment on any task mutation, not just state change), have clients reject payloads with `revision < stored.revision`, and have clients re-fetch on any terminal-state conflict. Servers that cannot add a revision field should document that terminal webhooks are snapshots and that clients must confirm via `tasks/get`. The cost is one extra round trip per terminal task for unversioned deployments; the benefit is elimination of the silent stale-terminal class entirely.

The deeper lesson for agents: in A2A, *state* and *content* have different finality guarantees. A terminal state fences transitions. It does not fence the payload. Any reconciliation logic that conflates the two will, under load, persist a result the server has already superseded.

```python
"""Reconcile an A2A v1.0 push-notification payload against locally stored task state.

Encodes the decision boundary from the record: terminal state is not content
finality; a terminal webhook is a snapshot and may be stale.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from typing import Literal

TaskState = Literal["submitted", "working", "input-required", "completed", "canceled", "failed"]
TERMINAL: frozenset[str] = frozenset({"completed", "canceled", "failed"})


class StaleTerminalPayloadError(RuntimeError):
    """Raised when a terminal webhook payload cannot be trusted as authoritative."""


class TerminalStateConflictError(RuntimeError):
    """Raised when a webhook terminal state disagrees with a stored terminal state."""


@dataclass(frozen=True)
class TaskSnapshot:
    task_id: str
    state: TaskState
    revision: int
    artifacts: tuple[str, ...] = field(default_factory=tuple)


def reconcile_webhook(
    stored: TaskSnapshot | None,
    webhook: TaskSnapshot,
    *,
    server_serializes_terminal_writes: bool,
) -> TaskSnapshot:
    """Return the snapshot the client should persist.

    Raises StaleTerminalPayloadError when the client must re-fetch via tasks/get
    instead of trusting the webhook.
    """
    if webhook.task_id != (stored.task_id if stored else webhook.task_id):
        raise StaleTerminalPayloadError(
            f"task_id mismatch: webhook={webhook.task_id!r} stored={stored.task_id!r}"
        )

    if stored is None:
        # No prior state: a non-terminal webhook is safe to seed from.
        if webhook.state in TERMINAL:
            raise StaleTerminalPayloadError(
                f"terminal webhook for unseen task {webhook.task_id!r}; "
                "must confirm via tasks/get before persisting"
            )
        return webhook

    # Conflict: two different terminal states. The webhook is a stale snapshot;
    # only tasks/get may arbitrate. Never overwrite.
    if stored.state in TERMINAL and webhook.state in TERMINAL and stored.state != webhook.state:
        raise TerminalStateConflictError(
            f"task {webhook.task_id!r}: stored={stored.state!r} "
            f"webhook={webhook.state!r}; re-fetch via tasks/get"
        )

    # Stale payload: same or older revision than what we already hold.
    if webhook.revision < stored.revision:
        raise StaleTerminalPayloadError(
            f"task {webhook.task_id!r}: webhook revision {webhook.revision} "
            f"< stored revision {stored.revision}; payload is superseded"
        )

    # Terminal webhook without serialized terminal writes is only a hint.
    if webhook.state in TERMINAL and not server_serializes_terminal_writes:
        raise StaleTerminalPayloadError(
            f"task {webhook.task_id!r}: terminal webhook is a snapshot and the "
            "server does not serialize terminal writes; confirm via tasks/get"
        )

    return webhook
```