{ "schema": "auturnal.deep-research-review.v2", "slug": "stochastic-optimization-noise", "title": "What Gradient Noise Actually Controls", "category": "Training & Optimization", "dek": "Batch size and learning rate jointly set a stochastic process, but neither its stationary variance nor its speed alone determines generalization.", "abstract": "Gradient noise is often treated as a single beneficial quantity. We separate three questions: how accurately a minibatch estimates a gradient, how a discrete optimizer distributes parameters near a stable minimum, and how computational resources are spent to reach a target loss. A finite-population calculation and an exactly solvable noisy quadratic show why maintaining a learning-rate-to-batch-size ratio does not preserve the full training process. The examples expose a discrete stability boundary, anisotropic uncertainty, and a resource-accounting ambiguity without claiming to reproduce a neural-network benchmark. Selected foundational work motivates the analysis; the principal contribution of this perspective is an explicit chain of assumptions and falsifiable comparisons.", "english": "Gradient noise is often treated as a single beneficial quantity. We separate three questions: how accurately a minibatch estimates a gradient, how a discrete optimizer distributes parameters near a stable minimum, and how computational resources are spent to reach a target loss. A finite-population calculation and an exactly solvable noisy quadratic show why maintaining a learning-rate-to-batch-size ratio does not preserve the full training process. The examples expose a discrete stability boundary, anisotropic uncertainty, and a resource-accounting ambiguity without claiming to reproduce a neural-network benchmark. Selected foundational work motivates the analysis; the principal contribution of this perspective is an explicit chain of assumptions and falsifiable comparisons.", "refs": [ "dl-01-1", "dl-01-2", "dl-01-3" ], "sections": [ [ "1. Three meanings of noise", "An optimizer can have a noisy gradient estimate, a variable parameter trajectory, and a variable test score. These are different random variables. Gradient noise is defined conditional on parameters and a sampling rule. Trajectory noise accumulates those conditional perturbations through an evolving update map. Test-score variation also contains dataset selection, initialization, and evaluation variation. A scalar reported for one of these quantities cannot substitute for the others.\nOur question is consequently operational: which aspects of training are invariant when batch size changes? A useful answer must state the time coordinate, the gradient normalization, whether examples are sampled with replacement, and whether learning rates change. Without those choices, even the units of a proposed noise scale are ambiguous. We start with the smallest model that distinguishes the choices and use its failures to design comparisons for neural networks.", [] ], [ "2. What the selected research establishes", "Mandt, Hoffman and Blei analyze constant-step stochastic optimization as approximate Bayesian inference under a local stochastic-process description. Smith and Le connect learning rate and finite-data batch size through a noise-scale argument. McCandlish and colleagues investigate a gradient-noise statistic as a predictor of useful parallel batch size. These are complementary viewpoints: stationary distributions, generalization-motivated dynamics, and resource efficiency.\nThey do not jointly establish that every noisy optimizer samples a calibrated posterior, that larger noise always improves held-out accuracy, or that a universal batch size is optimal. The calculations below are our own finite examples. They deliberately remove representation learning and changing curvature so that the distinction between approximation error and experimental evidence remains visible.", [ "dl-01-1", "dl-01-2", "dl-01-3" ] ], [ "3. Sampling without replacement changes the variance", "Fix N scalar per-example gradients and define their population variance with denominator N. A uniformly drawn subset of B distinct examples has mean variance equal to that population variance multiplied by (N-B)/(B(N-1)). The factor is not optional bookkeeping: at B=N, sampling variance is exactly zero. Sampling independently with replacement instead gives variance divided by B, even if the number of draws equals N.\nConsider the finite gradient population 1, 3, 5, 7. Its mean is 4 and its population variance is 5. Enumerating the six distinct batches of size two gives batch means 2, 3, 4, 4, 5, 6 and variance 5/3. Six is the number of unordered subsets, not the number of SGD steps. This exact enumeration becomes a compact test for a data loader's sampling assumptions.", [] ], [ "4. An exact discrete stationary variance calculation", "Let the objective be a scalar quadratic with curvature h>0, and let the gradient estimator add independent zero-mean noise of variance sigma squared divided by B. Constant-step SGD is an autoregression with coefficient 1-eta h. Stability requires its absolute value to be below one. In that interval, summing the geometric series for accumulated variance gives eta sigma squared divided by B h(2-eta h).\nThe denominator contains the discrete correction that a small-step diffusion approximation discards. With h=1, sigma squared=1, B=4 and eta=0.5, the stationary variance is 1/12. The corresponding leading small-step approximation is 1/16. Neither result is an observed neural-network statistic. Their difference is an exactly calculable warning about using a continuous approximation at a finite step size.", [] ], [ "5. A constant ratio does not preserve the process", "Suppose eta/B is fixed while both eta and B increase. The small-step approximation predicts unchanged variance, but the exact denominator retains eta h. In our normalized quadratic, the pairs (eta,B)=(0.1,1) and (0.4,4) produce different stationary variances despite identical ratios. Moreover, their mean contraction factors per update differ, so their transient errors differ even before stationarity becomes relevant.\nMatching processed examples does not fix the discrepancy: one update of the larger batch uses four times as many examples and changes the mean by a larger step. Matching wall time requires a hardware model in addition to the stochastic model. Thus a scaling rule can be a useful local approximation while failing as an exact equivalence. A controlled comparison should name which observable it expects to preserve.", [] ], [ "6. Curvature and noise orientation matter", "A vector quadratic replaces the scalar contraction with a matrix. In a basis where curvature and noise covariance are both diagonal, each coordinate follows its own scalar recursion. A common learning rate can be near the stability boundary in one coordinate and conservative in another. The largest curvature controls stability, while the distribution of injected noise controls which directions fluctuate.\nWhen the matrices do not commute, their diagonal entries cannot simply be paired; the stationary covariance solves a discrete Lyapunov equation. Its off-diagonal entries may be nonzero. Two gradient estimators with the same trace covariance can therefore generate different parameter distributions. A single noise scale loses directional information precisely where narrow valleys and parameter couplings make it useful. Our supplied checks verify the matrix covariance equation on a positive-definite two-dimensional construction.", [] ], [ "7. Posterior language requires an identified target", "To call a stationary optimizer distribution a posterior, one must specify a likelihood, prior, dataset size, and parameter measure. An arbitrary injected covariance is not generally the covariance required by that statistical model. Even on a quadratic, matching one scalar variance does not identify the complete multidimensional distribution. Reparameterizing a model also changes what Euclidean noise means.\nThe approximate-inference interpretation is valuable because it makes such requirements explicit. It should not be converted into a blanket statement that SGD uncertainty is Bayesian uncertainty. In a practical experiment, predictive coverage could be tested on held-out outcomes, while parameter covariance could be compared with a tractable reference posterior. These are separate tests with separate null hypotheses. Agreement on training loss alone cannot establish either one.", [ "dl-01-1" ] ], [ "8. Resource accounting produces different optima", "Imagine a device model in which one update costs a fixed launch overhead plus a term proportional to batch size. Increasing B amortizes overhead but reduces the number of parameter updates within a fixed example budget. A step-optimal policy and an example-optimal policy need not coincide. Communication, memory limits, and padding introduce additional constraints that this simple model omits.\nA useful batch-size study would report optimizer steps, processed examples, elapsed time, and achieved loss together. It would also distinguish requested batch size from the effective batch after accumulation or dropped examples. Our figures therefore plot variance against batch size and learning rate, not an invented throughput curve. A physical throughput claim requires measurements on a specified execution stack, with warm-up and synchronization handled consistently.", [ "dl-01-3" ] ], [ "9. How to test the neural-network extrapolation", "A meaningful extension would freeze the dataset split, architecture, initialization distribution, augmentation policy, and evaluation schedule, then vary sampling and optimizer parameters in a declared factorial design. A pilot can estimate gradient covariance at several matched loss levels rather than only at initialization. Those estimates should use independent minibatches and report uncertainty in both the mean gradient and covariance.\nThe key prediction is conditional, not universal: if the local model is adequate, observed covariance and relaxation should follow its finite-step correction over a bounded region. Failure could arise from changing curvature, correlated batches, nonstationary augmentation, momentum, or leaving the local basin. Logging these alternatives turns a negative result into diagnostic evidence. Selecting only checkpoints where the approximation happens to fit would make the comparison circular.", [] ], [ "10. What the executable evidence can support", "The companion code enumerates every nonempty batch size for a four-gradient population, checks the finite-population identity, and verifies stationary covariance by both recurrence and a linear-system solution. It also constructs a step size outside the stable interval and confirms that deterministic deviations grow. These tests cover arithmetic, boundary conditions, and a concrete counterexample to ratio invariance.\nThey do not train a neural network, measure generalization, or validate a universal scaling law. Their purpose is to make the article's logic inspectable before an expensive experiment is attempted. The resulting conclusion is narrow but useful: batch size cannot be interpreted independently of step size, sampling, geometry, and the resource being optimized. Any stronger empirical claim needs those quantities measured rather than silently assumed.", [] ], [ "11. Correlation and transient error are separate controls", "The independent-noise assumption can be tested rather than treated as an implementation detail. If two successive gradient errors are correlated, cross terms enter the accumulated parameter variance. Shuffling once per epoch, repeatedly visiting related examples, or reusing an augmentation stream can create temporal structure that a per-step covariance estimate misses. A stationary variance formula derived for independent increments is then the wrong reference even when the marginal noise variance is unchanged.\nThe starting point also matters. In the scalar model, the mean contracts as (1-eta h) to the power t, while variance approaches its limit through the square of that factor. Expected squared parameter error is the sum of the squared mean and variance. Early in training, reducing bias can dominate a larger variance floor; later, the floor can dominate. This decomposition provides a concrete reason to specify a training horizon before calling one noise level better than another.", [] ] ], "section_kinds": [ "author_analysis_or_verification_proposal", "source_based_review", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "source_based_review", "source_based_review", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal" ], "equations": [ { "tex": "\\operatorname{Var}(\\bar g_B)=\\frac{N-B}{B(N-1)}\\,\\sigma_{\\mathrm{pop}}^2", "explanation": "Uniform subset of B distinct gradients from a fixed population of N>1; population variance uses denominator N." }, { "tex": "\\theta_{t+1}=(1-\\eta h)\\theta_t-\\eta\\xi_t,\\qquad \\operatorname{Var}(\\xi_t)=\\sigma^2/B", "explanation": "Independent zero-mean additive noise, scalar curvature h>0, constant eta and B." }, { "tex": "V_\\infty=\\frac{\\eta\\sigma^2}{Bh(2-\\eta h)},\\qquad 0<\\eta h<2", "explanation": "Exact discrete stationary variance; diverges at the boundary and differs from the small-step approximation." }, { "tex": "C=(I-\\eta H)C(I-\\eta H)^\\top+\\eta^2\\Sigma/B", "explanation": "Discrete covariance equation when spectral radius of I-eta H is below one and additive noise has fixed covariance Sigma/B." } ], "table": { "headers": [ "Quantity", "Held fixed", "What remains unresolved" ], "rows": [ [ "Gradient variance", "Parameters and sampler", "Trajectory and generalization" ], [ "Stationary covariance", "Stable local dynamics", "Nonstationary representation learning" ], [ "Time to target", "Hardware and objective", "Statistical quality at equal cost" ] ] }, "protocol": [ "Reproduce the finite-population enumeration before running a data loader.", "Specify summed versus averaged gradients and the sampling replacement policy.", "Estimate covariance at matched checkpoints using independent batches.", "Compare exact discrete and small-step predictions with confidence intervals.", "Report steps, examples, elapsed time and held-out quality separately.", "Predeclare model-failure criteria and retain unstable or divergent runs." ], "limitations": [ "Additive independent noise is an assumption, not a property of all minibatch gradients.", "The quadratic has fixed curvature and cannot represent basin changes.", "No neural-network training or hardware benchmark was performed.", "Matching parameter variance cannot establish predictive calibration." ], "evidence": [ { "source": "dl-01-1", "result": "Develops an approximate-inference interpretation of constant-step SGD.", "boundary": "Local stochastic approximations and an identified statistical model are essential.", "locator": "Abstract; local stochastic-process analysis in the primary article." }, { "source": "dl-01-2", "result": "Relates SGD noise scale to learning rate and finite-data batch size.", "boundary": "A scaling approximation is not exact invariance of discrete optimizer trajectories.", "locator": "Section 5 and Appendix C of the author manuscript." }, { "source": "dl-01-3", "result": "Studies gradient noise as a predictor of useful large-batch training.", "boundary": "Empirical usefulness depends on the measured task and compute-time objective.", "locator": "Abstract and empirical-model framing." } ], "research_questions": [ "Which covariance directions best predict failure of a scalar noise-scale approximation?", "How much of a batch-size effect survives matching the optimizer trajectory rather than the example count?", "Can adaptive scheduling control both a stability margin and a declared resource budget?" ], "kind": "ai_assisted_research_guide", "author": "Codex (AI)", "original_language": "en", "created_date": "2026-10-22", "published_date": "2026-10-22T13:02:56+00:00", "publication_status": "published_commissioned_review", "version": 5, "previous_version": "/guides/stochastic-optimization-noise/v4/", "analytic": "stochastic-optimization-noise", "peer_reviewed": false, "journal_publication": false, "disclosure": { "authorship": "Written and technically self-reviewed by Codex (AI). References identify the original researchers; the analytical constructions and proposed evaluation designs are attributed to this article.", "figures": "Covers are conceptual ImageGen illustrations. Technical figures are computed from declared models or finite examples, with Python and CSV supplied. They are not measurements of deployed products or reproduced performance benchmarks.", "review": "This local draft has no independent peer-review or journal-acceptance record. Bounded executable examples check the stated small models; they are not formal verification of the cited systems." }, "review_scope": "A focused analytical review of three selected foundational primary sources, with original worked examples. This is not an exhaustive or systematic survey of the literature through 2026. Source access and version notes are recorded with the bibliography.", "revision_note": "Initial local English manuscript; no server upload or publication action.", "source_slug": "stochastic-optimization-noise", "source_revision": "41dc5ea7bf1db9868e2ee5b121f7021c823cc018", "source_article_sha256": "c9faed5dc7698ef9761015b666786ef74c6d10f274060c3a83c1bec85066b233", "sources": [ { "id": "dl-01-1", "key": "DL01_1", "title": "Stochastic Gradient Descent as Approximate Bayesian Inference", "authors": [ { "given": "Stephan", "family": "Mandt" }, { "given": "Matthew D.", "family": "Hoffman" }, { "given": "David M.", "family": "Blei" } ], "journal": "Journal of Machine Learning Research 18(134)", "year": 2017, "doi": "", "url": "https://jmlr.org/papers/v18/17-214.html", "verified_date": "2026-10-22", "metadata_source": "Primary publisher or author arXiv record inspected with web tool", "author_list_complete": true, "csl_type": "article-journal", "access_scope": "JMLR bibliographic record, abstract and initial PDF text inspected; local stationary calculations are independently derived.", "version_note": "", "article_slug": "stochastic-optimization-noise", "source_article_slug": "stochastic-optimization-noise" }, { "id": "dl-01-2", "key": "DL01_2", "title": "A Bayesian Perspective on Generalization and Stochastic Gradient Descent", "authors": [ { "given": "Samuel L.", "family": "Smith" }, { "given": "Quoc V.", "family": "Le" } ], "journal": "arXiv:1710.06451", "year": 2017, "doi": "10.48550/arXiv.1710.06451", "url": "https://arxiv.org/abs/1710.06451", "verified_date": "2026-10-22", "metadata_source": "Primary publisher or author arXiv record inspected with web tool", "author_list_complete": true, "csl_type": "article", "access_scope": "Primary record and author HTML manuscript, including noise-scale formulation and section headings, inspected.", "version_note": "Cited record first submitted 2017; inspected v3 (14 February 2018), noted as ICLR 2018. Bibliographic year refers to the arXiv record.", "article_slug": "stochastic-optimization-noise", "source_article_slug": "stochastic-optimization-noise" }, { "id": "dl-01-3", "key": "DL01_3", "title": "An Empirical Model of Large-Batch Training", "authors": [ { "given": "Sam", "family": "McCandlish" }, { "given": "Jared", "family": "Kaplan" }, { "given": "Dario", "family": "Amodei" }, { "given": "", "family": "OpenAI Dota Team", "literal": "OpenAI Dota Team" } ], "journal": "arXiv:1812.06162", "year": 2018, "doi": "10.48550/arXiv.1812.06162", "url": "https://arxiv.org/abs/1812.06162", "verified_date": "2026-10-22", "metadata_source": "Primary publisher or author arXiv record inspected with web tool", "author_list_complete": true, "csl_type": "article", "access_scope": "Primary bibliographic record and abstract inspected; no exhaustive full-text or supplementary-material audit.", "version_note": "The collective author OpenAI Dota Team is retained as printed in the primary record.", "article_slug": "stochastic-optimization-noise", "source_article_slug": "stochastic-optimization-noise" } ], "original_source_url": "/guides/assets/v9-deep-learning-articles-stochastic-optimization-noise-json-c9faed5dc7698ef9.json", "pdf": { "document_id": "44520cfc-3e38-5170-9db2-5a29c197e91c", "version_id": "ce179036-fac1-5c28-9228-03b2328ce4a5", "media_type": "application/pdf", "authentication_required": true, "download_audited": true }, "presentation_change": "Complete version-bound translated reading views and selectable authenticated original/translated PDFs; original scientific text and evidence retained.", "source_locale": "en", "source_version_id": "4a0483db-0de8-5446-b75b-b5e194e9ff4c", "translations": [ { "locale": "zh", "translation_id": "8dbcf548-48a6-57ff-98c6-a62cb14855e3", "translated_content_sha256": "ed317f7e2386cbc098462fa43b62f9d2bed810b3516307b57966b80b81d967f3", "sha256": "be666172f3c0e7d5649b938adfdae31cfe00d780265ebf39b5829ea9c6a8760a", "byte_length": 868957 }, { "locale": "es", "translation_id": "55943e6a-0979-5063-a9f8-11893d24b284", "translated_content_sha256": "086fa81ee2e66ac73fab235b54555dc83cf62767bde43fa47bde8f931bc700d7", "sha256": "51a91d599874a7f6be7c3518f479aaa2bc17a94a4ca7d0d17e8773faaea381b5", "byte_length": 629183 }, { "locale": "ja", "translation_id": "5028055a-6983-52ea-815a-7e614f7331c4", "translated_content_sha256": "9abf620325e5ca4d612117033afabdbcbd97eda8bf7546b1f1bec0e12aa1697f", "sha256": "57a215191535cf4c8ec9f76bffcff805a21eed8d624e3fd362b30909e8b0a45f", "byte_length": 839585 }, { "locale": "ar", "translation_id": "0c5512c7-6e0f-533e-a635-7ad036cc378d", "translated_content_sha256": "1dbfbc00e42c083783c33eb088be9d8097823220cf8e3fafec0865689fb2fe72", "sha256": "704efce04653e57b35062d77f765a7d99a6b87cb4d53d5bb0c35f607be81363f", "byte_length": 633353 } ] }