# When does a watermark lagging the event-time skew silently drop streaming join matches?

**Date of Record:** 2026 November 1

**Scope:** Applies to event-time streaming joins (stream-stream or stream-table) that close windows on a watermark and emit inner-join results only. Does not apply to processing-time joins, to joins that retain unbounded state, or to pipelines that route late records to a mandatory side output which is drained and alerted on. It also does not concern cryptographic watermarking of payloads for provenance, which is a separate mechanism sharing only the word.

## Core Distinction

A watermark is a *claim of completeness*, not a measurement of it. The watermark lag is a configured constant; the event-time skew of the source is an empirical distribution that drifts with upstream batching, partition imbalance, and mobile or offline producers. When the configured lag is smaller than the actual skew tail, the operator does not degrade gracefully: it declares windows complete that are not, and the join emits a result that is well-formed, successful, and systematically incomplete.

The distinction that matters is between lateness that is absorbed and lateness that is discarded. Only the first is a latency and throughput trade. The second is silent correctness loss, because the discarded side is never counted as a failure at the sink.

## What Was Observed

When watermark lag was reduced from 30s to 5s on a join whose observed event-time skew had a p99 of 11s:

- Throughput rose and end-to-end latency fell, and every dashboard stayed green.
- The inner join output rate fell by roughly the fraction of matches whose second side arrived after the window closed, observed at 4 to 7 percent depending on partition.
- The late-records counter incremented continuously, but it was emitted as a metric rather than an error. No sink record was marked incomplete and no batch was failed.
- Downstream aggregates were undercounted by a stable, non-obvious percentage, which is the worst shape of defect, because it looks like a quiet shift in traffic rather than a bug.

The failure is asymmetric and easy to miss: the early side of the join is present, so every emitted row is individually valid. Only the population is wrong.

## Condition of Validity

Inner-join emission is valid only while all of the following hold:

1. The watermark lag is at least the skew p99 plus a safety margin, measured on the same source partitions that feed the join, not a global cluster statistic.
2. The skew estimate is refreshed on a cadence shorter than the drift timescale of the producer population, since new device cohorts, batch backfills, and replay jobs all shift it.
3. If the first condition cannot be met for state-cost reasons, the join must be outer with a late side output, and that side output must be drained by a consumer whose non-zero rate is an alert rather than a gauge.

The first condition alone is insufficient if the watermark is derived from a single slow partition. A watermark that waits on the minimum across partitions bounds skew but couples throughput to the slowest shard.

## Decision Boundary

The boundary is a comparison, not a heuristic:

- If watermark lag is at least the skew p99 plus margin, the inner join is complete modulo the untracked tail, which should still be monitored.
- If watermark lag is below the skew p99 plus margin, the inner join is lossy. Either raise the lag, paying state and latency, or switch to an outer join with a mandatory and alerted late-data path.
- If the lag is chosen from a global skew percentile while the join reads a skewed subset, treat it as the second case regardless of the numbers, because the statistic does not describe the input.

There is no configuration in which a lossy inner join is acceptable without an alerted late-data path, because the loss is invisible at the sink.

## Operational Effect

Correctness becomes a function of a configuration constant that is almost never revisited after deployment, while the input distribution changes continuously. The remediation is not a code change but a coupling: the watermark lag must be derived from, and re-derived against, the measured skew of the actual input, and the late-data path must be a hard gate rather than an observability nicety. Absent that coupling, the pipeline reports success while undercounting, and the undercount is discoverable only by reconciliation against a source of truth.

```python
from dataclasses import dataclass


class WatermarkSkewError(RuntimeError):
    """Raised when the configured watermark lag cannot absorb the observed event-time skew."""


@dataclass(frozen=True)
class WatermarkGate:
    watermark_lag_ms: int
    margin_ms: int = 2000

    def verify(self, skew_p99_ms: int, same_partition_set: bool = True) -> bool:
        if not same_partition_set:
            raise WatermarkSkewError(
                "skew statistic measured on a different partition set than the join input"
            )
        if self.watermark_lag_ms < skew_p99_ms + self.margin_ms:
            raise WatermarkSkewError(
                f"lag {self.watermark_lag_ms}ms < skew_p99 {skew_p99_ms}ms + margin {self.margin_ms}ms"
            )
        return True


def dispatch_join_mode(gate: WatermarkGate, skew_p99_ms: int, same_partition_set: bool = True) -> str:
    try:
        gate.verify(skew_p99_ms, same_partition_set)
    except WatermarkSkewError:
        return "outer-with-alerted-late-path"
    return "inner"
```