{ "schema": "auturnal.deep-research-review.v2", "slug": "llm-serving-budgets", "title": "Serving Language Models Within a Memory Budget: Attention, Paging, and Scheduling", "category": "AI systems and measurement", "dek": "The capacity of an inference service is a joint constraint on weights, attention state, memory movement and scheduling. Tokens per second alone does not describe that constraint.", "abstract": "Language-model serving combines a large fixed parameter footprint with request-dependent attention state and variable-length generation. This review connects PagedAttention, FlashAttention and Orca to distinguish three mechanisms: allocation of key-value state, reduction of attention memory traffic, and iteration-level scheduling. We derive explicit key-value memory budgets for multi-head and grouped-query configurations, quantify page-tail waste in a declared length distribution, and compare a small online-softmax calculation with dense attention. The accompanying curves are accounting models, not GPU benchmarks or reproduced source-paper speedups. Their purpose is to expose when a serving claim silently changes precision, context length, concurrency, output length or service objective. A proposed evaluation protocol reports time to first token, inter-token latency, completed-request throughput and memory high-water marks together, while checking numerical and request-isolation properties. Efficient kernels and efficient service scheduling are treated as complementary, separately testable claims.", "english": "Language-model serving combines a large fixed parameter footprint with request-dependent attention state and variable-length generation. This review connects PagedAttention, FlashAttention and Orca to distinguish three mechanisms: allocation of key-value state, reduction of attention memory traffic, and iteration-level scheduling. We derive explicit key-value memory budgets for multi-head and grouped-query configurations, quantify page-tail waste in a declared length distribution, and compare a small online-softmax calculation with dense attention. The accompanying curves are accounting models, not GPU benchmarks or reproduced source-paper speedups. Their purpose is to expose when a serving claim silently changes precision, context length, concurrency, output length or service objective. A proposed evaluation protocol reports time to first token, inter-token latency, completed-request throughput and memory high-water marks together, while checking numerical and request-isolation properties. Efficient kernels and efficient service scheduling are treated as complementary, separately testable claims.", "refs": [ "cs-19-1", "cs-19-2", "cs-19-3" ], "sections": [ [ "1. A serving result needs a request distribution", "A model identifier and a GPU name are insufficient to define a serving experiment. Requests have prompt lengths, generated lengths, arrival times and completion criteria. A workload of short prompts with long outputs stresses different resources from a workload dominated by long prefills. Concurrency changes both the amount of live attention state and the opportunity for batching. Stop conditions can also change the measured work: generating a fixed token budget differs from stopping at an application-specific delimiter. The primary experimental object should therefore be a distribution of completed requests under a declared arrival process. Token throughput remains useful, but it must be attached to that distribution and to the latency objective that the scheduler was required to satisfy.", [] ], [ "2. Three mechanisms that should not be conflated", "PagedAttention organizes key-value cache management around paged allocation and sharing mechanisms. FlashAttention reduces attention memory traffic through an IO-aware exact-attention algorithm. Orca introduces iteration-level scheduling for transformer inference serving. These mechanisms address allocation, kernel execution and scheduling respectively. A memory-management improvement does not by itself establish lower kernel arithmetic cost, and a faster attention kernel does not establish fair request scheduling. The source papers provide concrete implementations and evaluations, but this article does not reproduce their numerical performance claims. Its calculations instead make the resource boundaries explicit so that a later benchmark can attribute a gain to a mechanism rather than to an unnoticed change in workload or precision.", [ "cs-19-1", "cs-19-2", "cs-19-3" ] ], [ "3. The key-value budget can be written down", "For a decoder with L layers, H key-value heads, head dimension d and b bytes per stored scalar, the uncompressed key-value state requires two times L times H times d times b bytes per retained token. The factor two accounts for keys and values. Multiply by the sum of retained sequence lengths across live requests. With 32 layers, eight key-value heads, dimension 128 and two bytes per scalar, this is 128 KiB per token. A 4,096-token request therefore uses 0.5 GiB for this state alone. Using 32 key-value heads under otherwise identical assumptions raises it to 2 GiB. These are declared architectural examples, not a specification of a named deployed model.", [] ], [ "4. Allocation efficiency is not the entire memory budget", "Model weights, activations, workspace buffers, communication buffers and allocator metadata also consume memory. Temporary prefill work can produce a high-water mark larger than steady decoding. Quantizing weights does not automatically quantize the key-value cache, and changing key-value precision can change numerical behavior. A capacity estimate should list these terms separately and reserve headroom for implementation-specific overhead. The figure deliberately plots only the key-value term so that it remains exactly reproducible. It should not be read as the number of requests that a GPU can serve. That number requires a measured full-memory profile under the chosen runtime, model, kernel configuration and admission policy.", [] ], [ "5. Paging changes fragmentation, with a residual cost", "If a request of length T uses pages of B token slots, its allocated capacity is B times the ceiling of T divided by B. The unused tail is between zero and B minus one slots for a nonempty request. Under a uniform distribution of integer lengths from one to 512, and a page size dividing 512, average tail waste is exactly (B minus one) divided by two. A scheme that reserves 512 slots for every request wastes 255.5 slots on average under the same constructed distribution. This comparison isolates tail allocation; it excludes metadata, sharing, external fragmentation and cache eviction. It is not a reproduction of vLLM's memory measurements or a claim that smaller pages are always faster.", [] ], [ "6. Exact attention still has a numerical contract", "Dense attention forms scaled query-key scores, applies softmax and multiplies by values. An IO-aware implementation can avoid materializing the complete score matrix by processing blocks and maintaining running normalization statistics. This reduces intermediate storage and memory movement without turning dense attention into a generally subquadratic arithmetic problem. \"Exact\" describes the attention operation rather than a promise of bitwise identity across every floating-point evaluation order. A benchmark should state dtype, accumulation rules, tolerance and tested input range. Extreme logits are especially useful tests because an unstable implementation can overflow even when a moderate random example looks correct. The appendix includes a small double-precision online-softmax comparison as an algebraic check, not a GPU kernel validation.", [] ], [ "7. Scheduling exposes competing latency objectives", "A large batch can improve hardware utilization while making a newly arrived short request wait. Prefill work and decoding steps also have different shapes, so an iteration scheduler must decide how to share resources between requests at different stages. Time to first token measures one experience; inter-token latency measures another; total completion time measures a third. None can be inferred from aggregate tokens per second alone. Report distributions by prompt and output length, together with failed and cancelled requests. A policy that silently rejects long prompts to improve averages has changed the admitted workload. Admission rules belong in the experimental description and should be included when comparing service capacity.", [] ], [ "8. Shared prefixes need ownership and lifetime rules", "Reusing key-value state for identical prefixes can reduce repeated work, but introduces identity, mutability and lifetime questions. A cache key must identify all inputs that affect the state, including model version and relevant execution configuration. Reused pages must not allow one request to observe another request's private continuation or mutate shared state incorrectly. Cancellation and eviction need to release references without freeing pages still in use. These are correctness obligations, independent of a throughput result. A deployment evaluation should therefore include request-isolation tests and concurrent cancellation cases. A mathematically valid attention kernel cannot compensate for an allocator or scheduler that returns the wrong request's state.", [] ], [ "9. What the executable companion demonstrates", "The companion checks the byte accounting and a six-request page-allocation example. It then computes small dense attention arrays and compares them with an online-softmax recurrence using a fixed seed, multiple block sizes and a stated tolerance. Additional large-logit inputs exercise numerical normalization. These tests validate the supplied Python construction only. They do not execute FlashAttention, PagedAttention or Orca, and they provide no hardware speed comparison. The figures show key-value GiB against retained context and mean page-tail waste against page size. Every parameter and array is exported. The purpose is to give readers independently inspectable resource calculations before they interpret a full serving benchmark.", [] ], [ "10. A benchmark that supports a deployment decision", "Fix the model, tokenizer, numerical configuration, arrival process and length distribution. Compare systems at the same completion and latency objectives, measuring memory throughout prefill and decoding. Include warm-up rules, cache state, cancellation, failures and independent repetitions. Verify representative outputs or a declared numerical acceptance criterion before interpreting speed. Then explain which mechanism changed the result: reduced state waste, lower memory traffic, better overlap or a different scheduling tradeoff. A strong conclusion specifies the region in which it holds and the resource that becomes limiting outside that region. The analytical examples in this review support that discipline; they do not certify a production capacity or a universal ranking of serving engines.", [] ] ], "section_kinds": [ "author_analysis_or_verification_proposal", "source_based_review", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal" ], "equations": [ { "tex": "M_{\\mathrm{KV}}=2LH_{\\mathrm{KV}}d_hb\\sum_i T_i", "explanation": "Uncompressed keys and values only; excludes weights, activations, metadata, padding and other workspace." }, { "tex": "W_{\\mathrm{tail}}=\\sum_i\\left(B\\left\\lceil T_i/B\\right\\rceil-T_i\\right)\\leq N(B-1)", "explanation": "N nonempty requests, integer token lengths and page size B; measured in token slots." }, { "tex": "E[W_{\\mathrm{tail},1}]=(B-1)/2", "explanation": "Uniform integer lengths from 1 through a positive multiple of B; this distribution is an explicit construction." }, { "tex": "m^\\prime=\\max(m,\\max_j s_j),\\quad \\ell^\\prime=e^{m-m^\\prime}\\ell+\\sum_j e^{s_j-m^\\prime}", "explanation": "Online softmax normalization for a new score block; the weighted-value accumulator is rescaled by the same factor." } ], "table": { "headers": [ "Mechanism", "Resource it targets", "Required companion check" ], "rows": [ [ "Key-value paging", "Allocation and state reuse", "Ownership, lifetime and total memory" ], [ "IO-aware attention", "Intermediate storage and memory traffic", "Numerical tolerance and workload shape" ], [ "Iteration scheduling", "Batch utilization and request progress", "First-token, inter-token and completion latency" ], [ "Admission control", "Capacity and overload", "Rejected requests and unchanged workload semantics" ] ] }, "protocol": [ "Pin model, tokenizer, precision, kernels and scheduler configuration.", "Publish arrivals, prompt lengths, output lengths and stop conditions.", "Validate numerical behavior and request ownership before timing.", "Measure full memory high-water marks during prefill and decoding.", "Report completed requests, all three latency measures, cancellations and failures.", "Repeat independent runs with explicit cache state and matched service objectives." ], "limitations": [ "No GPU or serving engine was benchmarked.", "Memory figures include only the stated key-value or allocation term.", "The online-softmax implementation is a small Python check, not an implementation or validation of FlashAttention.", "The selected source papers do not cover all later kernels, architectures or serving systems." ], "evidence": [ { "source": "cs-19-1", "result": "Paged allocation and sharing mechanisms address key-value state management.", "boundary": "The constructed waste calculation excludes several real allocator costs.", "locator": "PagedAttention paper, design description." }, { "source": "cs-19-2", "result": "IO-aware exact attention avoids materializing large intermediates through a block algorithm.", "boundary": "Exact operation does not imply universal bitwise equality or subquadratic dense arithmetic.", "locator": "FlashAttention paper, algorithm and IO analysis." }, { "source": "cs-19-3", "result": "Iteration-level scheduling addresses transformer inference execution at the serving layer.", "boundary": "A scheduler throughput result must be evaluated with its latency and workload conditions.", "locator": "Orca, design and scheduling description." } ], "research_questions": [ "How should admission controllers reserve for prefill peaks without excessive idle headroom?", "Can shared-prefix ownership and reclamation be checked independently of numerical kernels?", "Which workload families reveal when paging overhead outweighs reduced tail waste?" ], "kind": "ai_assisted_research_guide", "author": "Codex (AI)", "original_language": "en", "created_date": "2026-10-22", "published_date": null, "publication_status": "local_draft_pending_owner_upload", "version": 1, "previous_version": null, "analytic": "llm-serving-budgets", "peer_reviewed": false, "journal_publication": false, "disclosure": { "authorship": "Written and technically self-reviewed by Codex (AI). References identify the original researchers; the analytical constructions and proposed evaluation designs are attributed to this article.", "figures": "Covers are conceptual ImageGen illustrations. Technical figures are computed from declared models or finite examples, with Python and CSV supplied. They are not measurements of deployed products or reproduced performance benchmarks.", "review": "This local draft has no independent peer-review or journal-acceptance record. Bounded executable examples check the stated small models; they are not formal verification of the cited systems." }, "review_scope": "A focused analytical review of three selected foundational primary sources, with original worked examples. This is not an exhaustive or systematic survey of the literature through 2026. Source access and version notes are recorded with the bibliography.", "revision_note": "Initial local English manuscript; no server upload or publication action." }