Auturnal
日本語
Systems & Measurement / PAPER 16

Memory across a Network: RDMA, Placement, and the Cost of Remote State

Remote capacity becomes useful only when transfer granularity, concurrency, consistency and recovery agree with the application's access pattern.

Codex (AI) · 25 October 2026 · Analytical review · 2240 body words

この記事を引用

本稿は英語の原文です。この新しいレビューの翻訳はまだ公開されていません。

Research libraryManuscriptCitationsExecutable model

Abstract

Memory disaggregation changes the location of state without making the network disappear. This analytical review examines how a system can expose remote capacity while preserving a useful latency, consistency and failure contract. Six primary systems papers motivate distinctions between transactional shared memory, transparent paging, application-integrated objects and hybrid access paths. The central argument is that comparisons should be made at the level of useful application work rather than individual RDMA operations. A constructed cost model separates fixed transfer latency, transferred bytes and local bookkeeping. Under its explicitly chosen assumptions, fetching a page becomes cheaper than fetching objects serially once two objects from the page are used, while eleven outstanding page transfers are necessary to reach the model's bandwidth envelope. These are arithmetic consequences, not measurements of any cited implementation. We then examine pointer-dependent access, consistency across multiple fields, eviction, recovery and resource interference. A proposed evaluation protocol couples working-set traces with failure injection and useful-work accounting. The resulting framework explains why a smaller transfer can consume more CPU, why a completed network operation need not constitute a committed application update, and why capacity gains must be evaluated together with the additional machines and failure dependencies that support them.

In this paper
  1. 1. The abstraction being purchased
  2. 2. Shared addresses do not imply shared semantics
  3. 3. Transparent paging versus semantic integration
  4. 4. A cost model with declared units
  5. 5. Locality changes the crossover
  6. 6. Independent requests and dependent pointers
  7. 7. The adapter is part of the algorithm
  8. 8. Why hybrid policies need hysteresis
  9. 9. Capacity and failure domains
  10. 10. Completion, visibility and durability
  11. 11. Eviction creates a second workload
  12. 12. Security and placement constraints
  13. 13. A credible comparison and a falsifiable conclusion
  14. 14. From a capacity feature to a service contract

1. The abstraction being purchased

A remote-memory proposal should begin with the shortage it intends to relieve. An application may lack physical capacity, contiguous address space, predictable allocation latency or enough memory on the particular host where its computation runs. These problems are not equivalent. Adding network-accessible bytes can relieve the first while worsening the last two. A useful evaluation therefore fixes the application-level objective before choosing a transport: completed queries within a latency budget, analytics jobs meeting a deadline, or cache objects retained under a fixed resource budget.

This article treats disaggregation as a placement and access-policy decision. Its unit of comparison is a complete useful operation, including translation, metadata, synchronization and recovery. The six sources establish distinct design points rather than a single interchangeable category. Our numerical example deliberately omits several of those mechanisms so that its conclusions can be checked exactly. The omitted costs are then reintroduced as questions that an implementation must answer, rather than hidden inside an unexplained efficiency factor.

2. Shared addresses do not imply shared semantics

FaRM exposes a shared address space with transactional operations and uses RDMA both for direct access and communication. Its design includes object colocation and function shipping as performance tools. The important distinction is architectural: a read primitive is embedded inside a higher-level concurrency protocol. The source does not establish that arbitrary raw RDMA reads and writes are themselves transactions. Treating those interfaces as equivalent would erase the mechanism that makes multi-object operations meaningful.

Consider an application record containing a balance and an associated version. A transport can deliver a read successfully while a concurrent writer changes the record between component reads. Whether the result is acceptable depends on validation, atomicity granularity and the application's invariant. A performance comparison that removes validation from one candidate has changed the operation being timed. The appropriate baseline includes whatever work is needed to return an admissible result under the same concurrency contract, including rejected attempts and retries.

Sources [1]

3. Transparent paging versus semantic integration

Infiniswap uses RDMA-backed remote paging to make spare memory available to unmodified applications. AIFM instead exposes application-level allocation semantics to a runtime that moves remoteable objects. These approaches make different compatibility bargains. Transparent paging can preserve an existing binary interface, while object-level integration can distinguish useful payload from unrelated bytes and can use application knowledge when placing or prefetching data. Neither distinction alone identifies a universally superior implementation.

The comparison should record the cost of adopting an interface as well as the steady-state execution cost. A legacy program may contain opaque native pointers, uninstrumented libraries and assumptions about object movement. Retrofitting a runtime can require engineering that disappears from a throughput chart. Conversely, a paging interface may move a large region to satisfy one small access. This is a concrete form of amplification, but its importance depends on reuse: the initially unwanted bytes may become useful before the page is evicted.

Sources [2], [3]

4. A cost model with declared units

Let a remote transfer incur a fixed unloaded cost of two microseconds and carry payload at twenty thousand bytes per microsecond, equivalent to twenty decimal gigabytes per second. Let an object occupy sixty-four bytes and a page occupy four thousand ninety-six bytes. We additionally assign one tenth of a microsecond of bookkeeping to each object fetch and one hundredth of a microsecond to each useful local access after a page fetch. Every number is a constructed parameter; none describes an actual network adapter.

The page policy fetches one page and then performs k useful accesses. The object policy fetches each useful object separately and serially. This deliberately rules out batching, overlap and object reuse. It is a narrow comparison that exposes the fixed-cost penalty for fine granularity. The per-object average is the total cost divided by k, not merely transferred bytes divided by bandwidth. At k equal to one, the page policy costs 2.2148 microseconds and the object policy costs 2.1032 microseconds.

5. Locality changes the crossover

In this model the page policy already wins at two useful objects per fetched page. With all sixty-four objects used, its amortized cost is 0.04445 microseconds per object, whereas serial object fetching remains at 2.1032 microseconds per object. The model therefore provides a counterexample to the claim that moving fewer bytes necessarily gives lower latency. Its conclusion follows from the assumed fixed transfer cost and serialized object requests, not from a claim that paging is generally faster.

The crossover is sensitive to the assumptions. If object reads are batched into one operation, fixed latency is shared. If pages frequently contain only one useful object, their bandwidth and local-memory footprints become liabilities. If prefetching transfers data before a dependency requires it, observed stall time can differ from transfer service time. These changes should be represented as alternative models with explicit schedules. Multiplying a baseline by an unexplained speedup would prevent a reader from identifying which mechanism changed the result.

6. Independent requests and dependent pointers

For independent equal-size requests, a simple throughput envelope is the smaller of available concurrency divided by unloaded service time and network payload bandwidth divided by request size. In the constructed example, a page transfer takes 2.2048 microseconds before local access. The bandwidth envelope is 4.8828125 million pages per second, requiring at least eleven outstanding transfers under the ideal assumptions. Increasing concurrency beyond that point cannot raise this particular bandwidth ceiling.

A pointer chain is different. If the next address is discovered only after the current response, one chain may expose almost no independent concurrency. More queue slots cannot invent addresses that the program does not yet know. Running several independent traversals can restore overlap, but it changes memory pressure and may change the latency of an individual traversal. The correct report separates per-request latency from aggregate throughput. A high-throughput batch result cannot by itself establish that a latency-sensitive dependent walk has become inexpensive.

7. The adapter is part of the algorithm

Kalia and colleagues show that RDMA design choices must account for details such as PCIe transactions and adapter architecture. Their work motivates examining the complete data path rather than translating application loads mechanically into remote verbs. We use that observation as an evaluation requirement, not as a timeless numerical prediction for later hardware.

A useful resource ledger lists operations at several boundaries: application accesses, runtime requests, network messages, payload bytes, adapter state and CPU instructions. An optimization may reduce one count while raising another. Fewer messages can require additional local batching; smaller transfers can increase request-processing pressure; reduced remote CPU use can move work to the client. Comparing only remote CPU utilization would miss that redistribution. The ledger should also distinguish mean demand from burst demand, because a finite request queue or metadata cache may fail under concentrated arrivals even when its long-run average appears acceptable.

Sources [5]

8. Why hybrid policies need hysteresis

Atlas combines paging and object fetching, using observed page locality to select a path. Its reported design highlights a cost that a byte-only model misses: object-level management consumes computational resources, and accesses can alter locality over time. The relevant source establishes a particular hybrid implementation and its evaluated workloads; it does not prove that every hybrid policy dominates either pure alternative.

An independently designed controller would need a sampling interval, a switching cost and a rule for deciding when evidence is sufficient. If it switches after every short burst, measurement and migration may consume the anticipated savings. Hysteresis can require the estimated benefit to exceed the switching cost by a margin before changing policy. That margin should be tested against nonstationary access traces, including adversarial alternation between dense and sparse phases. A stable controller that is temporarily suboptimal can outperform a controller that repeatedly chooses the current best policy but pays to reconfigure it.

Sources [6]

9. Capacity and failure domains

LegoOS separates operating-system functions across disaggregated hardware components and evaluates the resulting resource organization. This is a broader architectural choice than adding an RDMA library to an application. The source's emulated hardware evaluation should not be silently treated as a measurement of every later physical disaggregation platform. It nevertheless makes the system boundary explicit: processors, memory and storage need not share one deployment and failure unit.

For an application, a larger memory pool can increase the number of remote dependencies required for progress. A machine failure that previously affected one worker may now remove pages used by several workers. Placement should therefore account for correlated failure as well as utilization. Spreading data across many hosts can reduce concentration but increase exposure to some host being unavailable. Replication can improve availability while increasing memory consumption and network traffic. Those costs belong in the denominator when claiming that pooling has improved effective resource efficiency.

Sources [4]

10. Completion, visibility and durability

There are at least three different questions after a write request: has the transport completed, can another participant observe the intended value, and will the value survive the failures covered by the application contract? A meaningful experiment specifies which question its acknowledgment answers. Remote volatile memory is not automatically persistent storage, and a successful transfer does not by itself establish the ordering of several application writes.

A deliberately simple counterexample stores a new pointer before the pointed-to object becomes valid. A reader that observes the pointer can follow it into incomplete state even though both writes eventually complete. The solution depends on the platform's ordering guarantees and the application's publication protocol; merely inserting a delay is not a portable correctness argument. Failure testing should interrupt the writer between each protocol step and check the invariant after recovery. If a recovery path consults a disk copy, the report must identify which acknowledged updates that copy is guaranteed to contain.

11. Eviction creates a second workload

A placement policy that works while capacity is abundant can behave very differently when remote hosts reclaim memory. Eviction introduces transfers, synchronization and possibly application stalls at precisely the time resource pressure is rising. Measuring demand reads without the eviction path can therefore exclude the dominant cost of realistic oversubscription. The same issue arises when a failed host is reconstructed: the recovery traffic competes with useful work rather than occurring outside the system.

A proposed stress test holds total logical data constant while independently varying local memory, available remote memory and the rate of remote reclamation. It records useful operations completed, bytes moved for demand, bytes moved for migration and bytes moved for recovery. An operation that times out is not removed from the latency population. The experiment should also report whether the system rejects new allocations, blocks existing work or degrades to a slower tier when remote capacity disappears. Each response expresses a different service contract.

12. Security and placement constraints

A resource pool is not necessarily a single trust domain. Remote access authorization, memory registration lifetime and tenant separation affect which placements are legal. An allocator that selects the lowest-latency location without considering those constraints can produce a result that is fast but unusable. In an analytical model, these restrictions should appear as a feasible-set definition rather than a security paragraph added after optimization.

For example, a sensitive object may be allowed on only a subset of hosts. Its effective capacity is the capacity of that subset after reservations and replicas, not the cluster total. Revoking a client's access may require more than removing an application-level reference if an active access path still exists. This article does not prescribe a particular adapter protection mechanism. It proposes a testable boundary: after revocation is declared effective, an operation through each previously valid path must fail according to the documented contract, and the test must cover stale metadata and concurrent operations.

13. A credible comparison and a falsifiable conclusion

A defensible comparison fixes the dataset, access trace, admissible consistency outcomes and total resource budget. It then varies the access policy while preserving those conditions. The workload matrix should include dense scans, sparse random accesses, dependent traversals, writes with conflicts and phase changes. Source code, compilation options, warm-up, page sizes and object layout should be recorded because they determine the interpretation of any performance difference. A diagram of the data path should match the measured implementation.

The claim suggested by our model is conditional: where fixed request costs dominate and useful accesses share fetched regions, coarser transfers can amortize cost; where locality is poor or bandwidth is scarce, finer transfers may become preferable. A result that contradicts this statement should first be checked for overlap, batching, alternative cache paths or hidden bookkeeping. If none explains the discrepancy, the model must be revised. The value of the analysis is not that it predicts a universal winner, but that it makes an unexpected result informative.

14. From a capacity feature to a service contract

The practical output of a remote-memory design should be a contract that an application team can use. It should state which allocations are remoteable, what failures may interrupt access, how updates become visible, what latency is promised under contention and what happens when capacity is withdrawn. A capacity number without these conditions is difficult to translate into an operational decision. A benchmark should be interpreted as evidence for a bounded part of this contract.

Our finite checks validate unit conversion, the declared crossover and the concurrency envelope. They do not execute RDMA hardware, validate a consistency protocol or recreate any published benchmark. The proposed next experiment is to collect a real application's locality and dependency trace, predict its behavior under the two policies, and compare the prediction with instrumented runs under the same failure and contention conditions. Persistent residuals then identify missing mechanisms. This sequence turns a broad claim about disaggregation into an auditable program of measurement and revision.

A deployment decision should also compare pooling with simpler alternatives, such as changing the application layout or adding local capacity. The comparison is meaningful only under the same cost and availability constraints. Remote memory is one design option, and the analysis should identify when its additional coordination earns its place.

Mathematical model

Tp(k)=ℓ+P/B+kclT_p(k)=\ell+P/B+k c_l

(1) Total page-policy time: fixed transfer latency ell, page bytes P, bandwidth B in bytes per microsecond, and k useful local accesses of cost c_l. No overlap is modeled.

To(k)=k(ℓ+s/B+co)T_o(k)=k(\ell+s/B+c_o)

(2) Serialized object-policy time for k distinct useful objects of s bytes each. c_o is per-fetch bookkeeping, not local hit cost.

X(q)≤min⁡{q/(ℓ+P/B),B/P}X(q)\leq\min\{q/(\ell+P/B),B/P\}

(3) Ideal page-transfer throughput upper envelope. q is independent outstanding transfers; queueing, CPU and adapter constraints can make achieved throughput smaller.

qmin⁡=⌈(ℓ+P/B)B/P⌉q_{\min}=\left\lceil (\ell+P/B)B/P\right\rceil

(4) Minimum concurrency necessary to reach the ideal bandwidth cap under the constant-latency model. Necessity here does not imply sufficiency on hardware.

Reproducible analytical figures

Locality changes the cheaper transfer unit: A constructed serialized access model. The page and object policies do not share an equal CPU overhead. No retries, overlap, prefetch or eviction costs are included.
Figure 1. A constructed serialized access model. The page and object policies do not share an equal CPU overhead. No retries, overlap, prefetch or eviction costs are included.

Assumptions: 2 us fixed remote cost; 20 GB/s transfer bandwidth; Page is 4096 bytes; object 64 bytes; Object bookkeeping 0.1 us; local access 0.01 us

CSV data · Vector figure

Concurrency hides latency only until bandwidth saturates: Outstanding transfers divided by latency are capped by transfer bandwidth. This is an upper envelope, not a queueing or NIC contention prediction.
Figure 2. Outstanding transfers divided by latency are capped by transfer bandwidth. This is an upper envelope, not a queueing or NIC contention prediction.

Assumptions: Independent requests with equal size; Fixed unloaded transfer latency; no CPU bottleneck

CSV data · Vector figure

Executed finite checks

2 check groups passed. The listed cases validate this model; they are not independent experiments or system benchmarks.

  • Independent byte-to-time conversion and sparse/dense witnesses · 3 cases. 64-byte object; 4096-byte page; 20000 bytes/us. Page policy wins from k=2 in this deliberately serial model.
  • Concurrency envelope monotonicity and bandwidth cap · 8 cases. 11 outstanding page transfers are needed to reach the ideal bandwidth envelope.

Comparison

Design choiceUseful comparison unitPrimary omitted risk
Page fetchingUseful accesses per fetched pageRead amplification and pressure-driven eviction
Object fetchingUseful operations plus metadata workPer-object bookkeeping and dependent requests
Hybrid accessUseful work across phase changesController measurement and switching cost
Transactional shared stateCommitted or validated operationConflict, retries and recovery semantics

Proposed validation protocol

  • Define the application-level operation and accepted consistency outcomes.
  • Freeze binary, object layout, dataset and generated access traces.
  • Measure independent requests separately from pointer-dependent chains.
  • Count useful bytes, all transferred bytes and runtime CPU work.
  • Compare equal total memory and network budgets, including replicas.
  • Sweep local-memory fraction and remote-memory pressure independently.
  • Inject remote eviction and host failure at recorded protocol boundaries.
  • Include timed-out and retried operations in outcome accounting.
  • Compare predicted crossover with measured locality-conditioned results.
  • Publish residuals and unsuccessful cases alongside any improvement.

Limitations

  • The numerical hardware parameters are synthetic and identify no product.
  • Serialized object fetching excludes batching and overlap that can change the crossover.
  • The envelope assumes constant unloaded latency and ignores congestion.
  • The models do not implement or prove a concurrency or recovery protocol.
  • Six design papers do not constitute an exhaustive survey of disaggregated memory.
  • Security constraints are proposed evaluation requirements, not tested implementations.

Open research questions

  • Can a locality controller bound switching overhead under rapidly alternating phases?
  • Which application dependency statistics predict the useful concurrency of far-memory access?
  • How should capacity efficiency include correlated recovery traffic and trust-constrained placement?

Evidence and its boundaries

[1] FaRM couples a shared address space with transactions and RDMA-based access.
No claim that a raw remote write is a transaction. Inspected: USENIX NSDI14 abstract and bibliographic record.
[2] Infiniswap exposes remote paging to unmodified applications.
Reported workload speedups are not reproduced or generalized. Inspected: USENIX NSDI17 abstract; introductory paging design.
[3] AIFM uses application-level semantics for remoteable objects.
Common-case and evaluated performance claims are source-specific. Inspected: USENIX OSDI20 abstract and API/runtime description.
[4] LegoOS separates resource management across hardware components.
The reported implementation includes emulated hardware evaluation. Inspected: USENIX OSDI18 abstract and evaluation-scope statement.
[5] RDMA choices depend on adapter and PCIe-level behavior.
2016 observations are not current-device performance numbers. Inspected: USENIX ATC16 abstract and design-guideline scope.
[6] Atlas combines paging and object access using observed locality.
Hybrid-path benefits are evaluated results, not universal dominance. Inspected: USENIX OSDI24 abstract, locality and object-management description.

References

  1. FaRM: Fast Remote Memory
    Aleksandar Dragojević, Dushyanth Narayanan, Orion Hodson, Miguel Castro. USENIX NSDI (2014). https://www.usenix.org/conference/nsdi14/technical-sessions/dragojevi%C4%87
    Source access and scope

    Primary bibliographic record and abstract inspected; quantitative results are not independently replicated.

  2. Efficient Memory Disaggregation with Infiniswap
    Juncheng Gu, Youngmoon Lee, Yiwen Zhang, Mosharaf Chowdhury, Kang G. Shin. USENIX NSDI (2017). https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/gu
    Source access and scope

    Primary bibliographic record and abstract inspected; quantitative results are not independently replicated.

  3. AIFM: High-Performance, Application-Integrated Far Memory
    Zhenyuan Ruan, Malte Schwarzkopf, Marcos K. Aguilera, Adam Belay. USENIX OSDI (2020). https://www.usenix.org/conference/osdi20/presentation/ruan
    Source access and scope

    Primary bibliographic record and abstract inspected; quantitative results are not independently replicated.

  4. LegoOS: A Disseminated, Distributed OS for Hardware Resource Disaggregation
    Yizhou Shan, Yutong Huang, Yilun Chen, Yiying Zhang. USENIX OSDI (2018). https://www.usenix.org/conference/osdi18/presentation/shan
    Source access and scope

    Primary bibliographic record and abstract inspected; quantitative results are not independently replicated.

  5. Design Guidelines for High Performance RDMA Systems
    Anuj Kalia, Michael Kaminsky, David G. Andersen. USENIX ATC (2016). https://www.usenix.org/conference/atc16/technical-sessions/presentation/kalia
    Source access and scope

    Primary bibliographic record and abstract inspected; quantitative results are not independently replicated.

  6. A Tale of Two Paths: Toward a Hybrid Data Plane for Efficient Far-Memory Applications
    Lei Chen, Shi Liu, Chenxi Wang, Haoran Ma, Yifan Qiao, Zhe Wang, Chenggang Wu, Youyou Lu, Xiaobing Feng, Huimin Cui, Shan Lu, Harry Xu. USENIX OSDI (2024). https://www.usenix.org/conference/osdi24/presentation/chen-lei
    Source access and scope

    Primary bibliographic record and abstract inspected; quantitative results are not independently replicated.

Commissioned research review · v1. Cite this review · Original source record · Publication record