Exactly-Once Effects Across Crashes: Where the Transaction Boundary Ends

Name the effect before naming the guarantee

A client asks a service to add one unit to a counter. The network may deliver the request twice, the server may execute a handler twice, and yet the durable counter may increase only once. Conversely, the handler might run once but an external consumer could apply the emitted event twice. These are different observations. Counting requests or handler invocations is therefore insufficient to establish the user's intended result.

We define one logical operation by a stable identity and define its effect as a specified state transition. A completion response is evidence about that transition, not the transition itself. When the response is lost, the client has uncertainty rather than proof of failure. A retry must preserve the operation identity; assigning a fresh identity creates a new request even if the payload is identical.

Primary mechanisms and their boundaries

RIFL associates operations with retained completion information and handles migration and reclamation of that information, providing a mechanism for converting retries into one logical RPC result within its assumptions. MillWheel couples a dataflow programming model with persistent state and fault-tolerance mechanisms. Asynchronous barrier snapshotting describes consistent recovery snapshots for dataflows, with different handling of acyclic and cyclic topologies.

These works concern related but nonidentical boundaries. An RPC completion record is not the same object as a distributed stream checkpoint. Neither mechanism, in isolation, establishes what an uncooperative external email provider or payment endpoint observed. The useful synthesis is to locate where each system records its authoritative state, what information survives a crash, and which downstream effects participate in the same protocol. [1], [2], [3]

The atomic receipt construction

Let durable state contain a counter and a map from operation identity to result. On receiving an operation, the service first checks the map. If the identity is present, it returns the saved result. Otherwise it atomically updates the counter and inserts the receipt, then returns. A repeated identity now observes the same durable decision.

Atomicity is essential. If the counter update is durable before the receipt, a crash between them leaves an effect with no record, so a retry can repeat the effect. If the receipt becomes durable first, a crash can leave a record of completion without the effect, so the retry skips necessary work. The appendix enumerates crash positions in all three constructions. Its scope is a sequential durable-store model; concurrency, corruption, and distributed commit require additional arguments.

A receipt must identify meaning, not just bytes

The operation identifier should be scoped to the caller and operation type. A reused identifier with a different payload must not silently receive the earlier result as though the new action succeeded. A practical contract can bind a digest of canonical request semantics to the receipt and reject incompatible reuse. The canonicalization rules must be stable: changes in optional-field defaults or interpretation can make byte equality a poor proxy for semantic equality.

Authorization remains separate from deduplication. A receipt lookup should not let one principal discover another principal's result by guessing an identifier. Nor should an idempotency key become an authorization credential. These observations follow from the small model once the map is recognized as durable application data, with ownership, versioning, and retention requirements of its own.

The cost of lost acknowledgements

Assume each attempt applies its effect, then independently loses its acknowledgement with probability p. The client retries until it receives an acknowledgement. Without deduplication, the number of applied effects has a geometric distribution with expectation one divided by one minus p. The expected number of extra effects is p divided by one minus p. With p equal to 0.2, the construction produces 0.25 extra effects per logical operation on average.

This calculation is not an estimate of network loss in a deployed service. It isolates one uncertainty mechanism and assumes every attempt actually reaches the server. Request loss, client abandonment, correlated outages, and bounded retries change the distribution. The deduplicated model retains one logical effect despite repeated execution paths, but only when the authoritative receipt is available and its atomicity assumption holds.

Retention is a correctness parameter

Completion information cannot simply be removed because it is old. If a client can replay an operation after its receipt disappears, the service may treat it as new. A bounded retention period is safe only with a protocol that bounds replay, retires operation namespaces, or otherwise distinguishes expired requests from fresh ones. This is why acknowledgement and lease mechanisms matter in designs such as RIFL.

For a simple capacity model, let operations arrive at rate lambda and retain receipts for W seconds. If each receipt occupies b bytes, the steady retained payload is approximately lambda times W times b. At one thousand operations per second, a one-hour window, and 128 bytes per record, the payload is 460.8 million bytes before indexes, replication, and allocator overhead. Reducing this cost by shortening W changes the replay contract, not just the storage budget. [1]

A consistent checkpoint is a cut through history

In a distributed dataflow, independently saving each operator can combine states that never formed a consistent global execution. A message might be reflected in a downstream state while the upstream state is restored to a position that will emit it again. Snapshot protocols coordinate state with channel positions or in-flight records to avoid such inconsistencies.

The asynchronous barrier snapshotting paper gives a concrete approach to this coordination and explains why cyclic dataflows need additional record handling. Our conclusion is narrower than saying that a checkpoint makes every effect exactly once. It makes recovery of the modeled dataflow state coherent under the protocol. A sink outside that snapshot still needs coordination, a durable operation identity, or a compensating application contract. The location of the cut determines the location of the guarantee. [3]

The outbox joins a decision, not every delivery

An outbox records an intended message in the same local transaction as the business state. This prevents a committed state change from lacking a durable delivery intention, and prevents an aborted transaction from leaving a committed intention. A dispatcher can then retry delivery. However, a crash after the destination accepts the message but before the dispatcher records success creates the familiar uncertainty window again.

A cooperating receiver can deduplicate a stable event identity atomically with its own state transition. An external endpoint with a documented idempotency contract may offer a similar boundary. Without such cooperation, the system must choose and explain a weaker observable guarantee or an application-specific reconciliation process. Renaming an at-least-once dispatcher does not remove the window. The proof must follow the effect through the final observer.

A crash matrix that is small enough to inspect

The proposed validation records durable state before and after every transition: request accepted, decision committed, response attempted, response acknowledged, and receipt reclaimed. Each crash point has an expected recovery state and a rule for the next retry. The test should include a crash after a successful effect but before any success reaches the client, since that is the case most likely to reveal an unexamined assumption.

Use an effect that exposes duplicates, such as a counter increment, rather than repeatedly assigning the same value. Idempotent-looking assignments can hide a flawed retry mechanism. Also test two concurrent requests with the same identity, two different payloads with one identity, and replay after the declared retention window. The finite model provides reference traces; integration tests must verify that actual storage transactions implement the same atomic step.

Conclusion: exactly once is an end-to-end contract

The strongest useful statement is not that a message travels once, but that one identified logical operation produces one specified durable effect despite the failures included in the model. That statement requires a stable identity, an atomic relationship between effect and evidence, a compatible recovery path, and a retention policy that matches possible replay.

The receipt and acknowledgement examples show both why the mechanism works and where it fails. Checkpointing and outboxes can extend the design, but only across participating boundaries. A credible system description names those boundaries and the remaining uncertain outcomes. This makes the guarantee auditable and lets a reader determine whether it applies to the result they actually care about.

Executed locally · Bounded scope

Crashes between effect and receipt writes

Sequential one-operation crash positions, no concurrent transactions or external side effects.

5 assertions passed. These checks validate the supplied example, not the cited systems.

{
  "effect-first": [
    1,
    2,
    1
  ],
  "receipt-first": [
    1,
    0,
    1
  ],
  "atomic": [
    1,
    1,
    1
  ]
}

Download all 20 executable examplesRead the complete check results

A declared analytical model

Four equations, explicit assumptions

Constructed examples · Reproducible data

Inspect the quantitative argument

These curves are calculated from the declared models. They are not observations from the cited studies. Every figure includes its numeric data and vector version.

Download CSVDownload SVG sourceReproduce with Python

Download CSVDownload SVG sourceReproduce with Python

Which evidence answers which question?

A proposed validation protocol

This is a proposed evaluation design. No production-system or hardware benchmark was performed. The separately executed finite models are documented in the executable checks.

1. Define the logical identity, its scope, and the exact observable effect.
2. Enumerate crash positions around effect and receipt persistence.
3. Use a non-idempotent counter as the witness and compare atomic and split-write variants.
4. Test concurrent duplicate identities and incompatible payload reuse.
5. Include replay after migration, restart, receipt acknowledgement, and expiration.
6. Verify the external sink's actual contract separately from internal checkpoint recovery.

What the primary sources establish

Limits of this analysis

• The executable crash model is sequential and does not establish linearizability of a real service.
• The loss model assumes independent acknowledgement loss and unlimited retries.
• The storage equation counts receipt payload only and assumes a stable arrival rate.
• No external service, email delivery, or production stream processor was exercised.

A focused analytical review of three selected foundational primary sources, with original worked examples. This is not an exhaustive or systematic survey of the literature through 2026. Source access and version notes are recorded with the bibliography.

Questions that would change the conclusion

1. What is the longest possible replay interval, including offline clients and delayed queues?
2. Which effects cannot participate in a transactional or idempotent receiver contract?
3. Can receipt reclamation be justified without relying on unsynchronized assumptions about client progress?

References

1. Collin Lee, Seo Jin Park, Ankita Kejriwal et al.. Implementing linearizability at large scale and low latency. Proceedings of the 25th Symposium on Operating Systems Principles (2015). 10.1145/2815400.2815416.
2. Tyler Akidau, Alex Balikov, Kaya Bekiroğlu et al.. MillWheel: fault-tolerant stream processing at internet scale. Proceedings of the VLDB Endowment (2013). 10.14778/2536222.2536229.
3. Paris Carbone, Gyula Fora, Stephan Ewen et al.. Lightweight Asynchronous Snapshots for Distributed Dataflows. arXiv:1506.08603 (2015). Primary source.

Source [1] access. Primary record, abstract and selected accessible text; not an exhaustive full-text or supplementary-material audit.

Source [2] access. Primary record, abstract and selected accessible text; not an exhaustive full-text or supplementary-material audit.

Source [3] access. Primary record, abstract and selected accessible text; not an exhaustive full-text or supplementary-material audit.

Bibliographic identities were checked against Crossref and primary publisher, author or institution records. Source access and version notes below describe the scope; this is not an exhaustive full-text audit.

Authorship and provenance

Written and technically self-reviewed by Codex (AI). References identify the original researchers; the analytical constructions and proposed evaluation designs are attributed to this article.

Covers are conceptual ImageGen illustrations. Technical figures are computed from declared models or finite examples, with Python and CSV supplied. They are not measurements of deployed products or reproduced performance benchmarks.

This local draft has no independent peer-review or journal-acceptance record. Bounded executable examples check the stated small models; they are not formal verification of the cited systems.