{ "schema": "auturnal.deep-research-review.v2", "slug": "normalization-geometry", "title": "Normalization Changes the Geometry of Learning", "category": "Training & Optimization", "dek": "Choosing which coordinates to normalize determines what information, invariance, and cross-example dependence a layer introduces.", "abstract": "Normalization is more than a numerical convenience. Its reduction axes define an information transformation, its stabilizing constant limits exact scale invariance, and its training statistics can couple predictions across examples. We compare batch and layer normalization through explicit two-example and finite-dimensional constructions. A Jacobian calculation separates a removed mean direction from a suppressed radial direction, while a counterexample shows how one sample's batch-normalized value changes when its companions change. The discussion uses foundational normalization papers as historical and technical anchors, then develops independently reproducible algebra rather than repeating benchmark claims. The result is a practical framework for diagnosing normalization choices without attributing every training improvement to a single causal explanation.", "english": "Normalization is more than a numerical convenience. Its reduction axes define an information transformation, its stabilizing constant limits exact scale invariance, and its training statistics can couple predictions across examples. We compare batch and layer normalization through explicit two-example and finite-dimensional constructions. A Jacobian calculation separates a removed mean direction from a suppressed radial direction, while a counterexample shows how one sample's batch-normalized value changes when its companions change. The discussion uses foundational normalization papers as historical and technical anchors, then develops independently reproducible algebra rather than repeating benchmark claims. The result is a practical framework for diagnosing normalization choices without attributing every training improvement to a single causal explanation.", "refs": [ "dl-03-1", "dl-03-2", "dl-03-3" ], "sections": [ [ "1. A normalization layer is a specification", "To specify normalization, one must name a tensor shape, reduction axes, variance convention, stabilizing constant, affine parameters, and training-versus-inference behavior. A formula written only as subtract mean and divide by standard deviation omits most of that specification. Two implementations can share the formula but represent different functions because they average over different coordinates.\nThis matters scientifically as well as operationally. An axis across examples introduces dependence on the batch composition; an axis within an example does not introduce that particular dependence. A normalization may erase a common offset that was irrelevant to the task, or erase one that carried the target. The relevant question is therefore not whether normalization is beneficial in general, but which transformations should leave a representation unchanged and which information must remain recoverable.", [] ], [ "2. Historical claims should remain distinguishable", "Ioffe and Szegedy introduced batch normalization and motivated it through changing internal activation distributions. Ba, Kiros and Hinton introduced layer normalization using statistics within a training case. Santurkar and colleagues argued that batch normalization's optimization benefits are better explained through smoother optimization behavior than through distributional stability alone.\nThese papers establish methods and competing explanatory emphasis, not a universal causal account for every architecture. We do not reproduce their reported speed or accuracy results. Instead, we ask what follows exactly from the normalization map. Such identities remain inspectable even when competing empirical explanations are unresolved. They also indicate which interventions an experiment would need before attributing an observed improvement to invariance, conditioning, stochasticity, or a changed effective learning rate.", [ "dl-03-1", "dl-03-2", "dl-03-3" ] ], [ "3. Centering removes a direction", "For a d-dimensional vector, subtracting its coordinate mean applies the projection P=I-11 transpose/d. The vector of all ones lies in its null space. Therefore x and x+b1 have identical centered representations for every scalar b. A later coordinatewise affine gain and bias cannot recover a sample-specific b from an identical normalized input.\nA concrete example is x=(1,2,4) and x+10=(11,12,14). A layer that only receives their normalized forms cannot distinguish their overall levels. This is appropriate if level is a nuisance, but inappropriate if the label is determined by that level and no other route preserves it. Residual connections or additional unnormalized features can change the information story. The conclusion concerns the isolated transformation, not an entire network that may bypass it.", [] ], [ "4. The stabilizer makes scale invariance approximate", "After centering, let v be the mean squared coordinate value and define the denominator as sqrt(v+epsilon). For a positive scale a, the normalized result is a times the centered vector divided by sqrt(a squared v+epsilon). If epsilon is zero and v is positive, a cancels exactly. If epsilon is positive, the cancellation is incomplete.\nAt very small a, the denominator is dominated by epsilon and the output amplitude decreases toward zero. At large a, the normalized representation approaches its scale-invariant limit. Negative scaling also flips the centered vector's sign, so it should not be described as invariance under arbitrary nonzero scaling. Our first figure computes this transition for a fixed variance and two stabilizers. It is a mathematical sensitivity curve, not a comparison of framework defaults.", [] ], [ "5. The Jacobian separates two suppressed directions", "Write c=Px and s=sqrt(c transpose c/d+epsilon). Differentiation gives J=P/s-cc transpose/(d s cubed). The common-offset direction has eigenvalue zero. Along the centered radial direction c, the eigenvalue is epsilon/s cubed. Directions orthogonal to both c and the all-ones vector have eigenvalue 1/s.\nThis decomposition is useful because it distinguishes a structurally removed direction from a direction suppressed by the stabilizer and signal magnitude. With epsilon=0 and nonzero variance, the radial direction also vanishes locally: moving only along positive scale does not change the output. With epsilon>0, it remains weakly observable. The companion check compares this analytic Jacobian with central finite differences away from the singular zero-variance case and verifies its eigen-relations.", [] ], [ "6. A batch can change one example's output", "Consider one scalar feature and a batch with values (0,2). With population variance and a small stabilizer, the normalized value of the first example is near -1. Replace the second value by -2 while keeping the first at zero; the first normalized value becomes near +1. The first sample is unchanged, yet its training-mode representation reverses sign.\nThis is not a bug in the arithmetic. It follows from using statistics shared by the batch. It also means that an experiment that changes batch composition while calling the model unchanged has altered the function evaluated at that sample. Repeated or correlated examples can further alter the statistics. A rigorous evaluation should specify whether statistics are recomputed, frozen, synchronized across devices, or accumulated through another declared rule.", [] ], [ "7. Inference statistics introduce an additional state", "If inference uses stored means and variances, the model includes those stored quantities as state. Saving only learned affine parameters is then insufficient to reproduce predictions. Running statistics estimated on one distribution can produce systematically shifted outputs on another. Recomputing statistics at deployment instead changes the evaluation protocol and may couple predictions across users or requests.\nA simple diagnostic holds the learned weights fixed and evaluates a chosen input under two explicit sets of statistics. The observed difference measures sensitivity to that state, not necessarily a failure of representation learning. Any adaptation of stored statistics should use an authorized dataset with a declared selection process and should be compared against a frozen reference. Test labels must not silently influence that adaptation. This separates legitimate deployment adaptation from evaluation leakage.", [] ], [ "8. Scale symmetry changes the meaning of parameter size", "Suppose an upstream weight vector appears only through an exactly scale-invariant normalized map. Multiplying it by a positive constant can leave predictions unchanged while changing its Euclidean norm and the magnitude of its parameter gradient. The functional model and its coordinate representation therefore have different notions of distance.\nThis observation explains why weight magnitude alone is a poor diagnostic of functional complexity in such a component. It also means that a penalty on parameter norm can influence optimization even where rescaling barely changes predictions. The exact relation depends on the stabilizer, affine parameters, bypass routes, and optimizer. We therefore avoid asserting that a particular weight-decay setting has a universal interpretation. A controlled study should inspect functional changes and parameter changes separately rather than treating either as a complete proxy for the other.", [] ], [ "9. A causal comparison needs more than a swap", "Replacing batch normalization with layer normalization simultaneously changes reduction axes, cross-example coupling, possible invariances, stored state, and sometimes parameter broadcasting. If performance changes, the replacement does not identify which mechanism caused the difference. Matching only the number of trainable parameters is insufficient.\nA more informative comparison would manipulate one property at a time where feasible: freeze statistics, alter batch composition at fixed examples, vary epsilon over a justified range, or preserve an unnormalized level feature through a separate route. These interventions must be paired with task-specific outcomes and optimization diagnostics. If an intervention changes both gradient scale and information availability, both changes should be measured. This design is more work than a component swap but produces a conclusion that can survive alternative explanations.", [] ], [ "10. What the figures and tests actually establish", "The supplied plots show positive-scale sensitivity and the cross-example dependence of a two-sample normalization. The tests verify shift invariance, the epsilon-dependent radial response, the Jacobian against finite differences, and the sign reversal under a changed batch companion. All inputs, variance conventions, stabilizers, and numerical tolerances are explicit.\nThese checks establish properties of the declared finite maps. They do not determine the best normalization for a language model or image classifier. The scientific value lies in separating exact algebra from empirical architecture selection. Once the map is understood, a real training study can ask whether its invariances align with the task and whether its numerical behavior improves a specified resource-quality tradeoff. Without that separation, the word normalization can conceal several incompatible explanations for the same observed result.", [] ], [ "11. A practical diagnostic sequence", "When training becomes unstable after a normalization change, first verify the axes and broadcasting behavior on a tiny hand-computable tensor. Next compare train and evaluation modes using identical inputs and explicitly captured state. Then inspect the distribution of denominators, especially near zero variance, before changing the learning rate.\nThis ordering matters because a numerical stabilizer can hide an axis error by preventing division by zero while leaving the semantic error intact. Likewise, a successful forward pass says little about derivative conditioning. A finite-difference check on a small smooth input is inexpensive evidence about implementation consistency. It should be complemented by tests at constant inputs and very small variance, where the expected behavior must follow the declared epsilon convention rather than an assumption of exact invariance.", [] ], [ "12. Precision belongs in the specification", "A mathematically positive denominator can still be represented inaccurately in finite precision. Subtracting a large mean from nearly constant coordinates can lose relative accuracy, and accumulating squared deviations in a narrow dtype can alter the variance. These are implementation questions beyond our float64 examples. A deployment test should record accumulation precision and compare small-variance inputs with a higher-precision reference before interpreting a discrepancy as a property of normalization itself.", [] ] ], "section_kinds": [ "author_analysis_or_verification_proposal", "source_based_review", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal" ], "equations": [ { "tex": "P=I-\\frac{\\mathbf1\\mathbf1^\\top}{d},\\quad c=Px,\\quad \\mathcal N_\\epsilon(x)=\\frac{c}{\\sqrt{c^\\top c/d+\\epsilon}}", "explanation": "Per-example coordinate normalization before learned gain and bias; d>=2." }, { "tex": "\\mathcal N_\\epsilon(ax+b\\mathbf1)=\\frac{aPx}{\\sqrt{a^2v+\\epsilon}},\\quad v=\\|Px\\|_2^2/d,\\ a>0", "explanation": "Offset invariance is exact; positive-scale invariance requires epsilon=0 and v>0." }, { "tex": "J=\\frac{P}{s}-\\frac{cc^\\top}{d s^3},\\quad J\\mathbf1=0,\\quad Jc=\\frac{\\epsilon}{s^3}c", "explanation": "Jacobian for epsilon>0; s squared equals v+epsilon." }, { "tex": "\\operatorname{BN}_\\epsilon(0;\\{0,b\\})=\\frac{-b}{\\sqrt{b^2+4\\epsilon}}", "explanation": "First example in a two-scalar batch with population variance and no affine transform." } ], "table": { "headers": [ "Choice", "Exact structural effect", "Required diagnostic" ], "rows": [ [ "Within-example centering", "Removes the shared offset", "Task information in overall level" ], [ "Across-example statistics", "Couples batch members", "Batch-composition sensitivity" ], [ "Positive epsilon", "Regularizes low variance", "Scale dependence and derivative magnitude" ] ] }, "protocol": [ "Specify tensor dimensions, axes, variance denominator, epsilon and affine broadcasting.", "Verify the transformation on a hand-computable tensor.", "Compare analytic and finite-difference Jacobians away from singular cases.", "Vary batch companions while holding the target example fixed.", "Capture stored inference statistics and test mode transitions explicitly.", "Predeclare task-specific interventions before attributing accuracy or speed changes." ], "limitations": [ "The algebra isolates normalization and excludes a complete residual network.", "No training-speed or accuracy benchmark was run.", "Finite differences are tolerance-based consistency checks, not symbolic proof.", "Framework-specific default axes and statistics must be checked separately." ], "evidence": [ { "source": "dl-03-1", "result": "Introduces batch normalization with minibatch statistics and learned affine parameters.", "boundary": "Its historical motivation is not a universal causal explanation for every observed benefit.", "locator": "Primary PMLR abstract and normalization formulation." }, { "source": "dl-03-2", "result": "Introduces normalization using statistics within a training case.", "boundary": "Within-example axes differ from across-batch statistics and their dependencies.", "locator": "Author manuscript abstract and layer-normalization definition." }, { "source": "dl-03-3", "result": "Studies optimization smoothness as an explanation for batch normalization's benefits.", "boundary": "The argument does not make every normalization choice or network equally smooth.", "locator": "Author manuscript abstract and optimization framing." } ], "research_questions": [ "Which task-relevant signals are removed by a proposed reduction axis?", "How much deployment error comes from stored statistical state rather than learned weights?", "Can interventions disentangle information removal from improved optimization conditioning?" ], "kind": "ai_assisted_research_guide", "author": "Codex (AI)", "original_language": "en", "created_date": "2026-10-22", "published_date": "2026-10-22T13:02:56+00:00", "publication_status": "published_commissioned_review", "version": 5, "previous_version": "/guides/normalization-geometry/v4/", "analytic": "normalization-geometry", "peer_reviewed": false, "journal_publication": false, "disclosure": { "authorship": "Written and technically self-reviewed by Codex (AI). References identify the original researchers; the analytical constructions and proposed evaluation designs are attributed to this article.", "figures": "Covers are conceptual ImageGen illustrations. Technical figures are computed from declared models or finite examples, with Python and CSV supplied. They are not measurements of deployed products or reproduced performance benchmarks.", "review": "This local draft has no independent peer-review or journal-acceptance record. Bounded executable examples check the stated small models; they are not formal verification of the cited systems." }, "review_scope": "A focused analytical review of three selected foundational primary sources, with original worked examples. This is not an exhaustive or systematic survey of the literature through 2026. Source access and version notes are recorded with the bibliography.", "revision_note": "Initial local English manuscript; no server upload or publication action.", "source_slug": "normalization-geometry", "source_revision": "41dc5ea7bf1db9868e2ee5b121f7021c823cc018", "source_article_sha256": "7405e5d3524661d7dfeaecaae305d5ea09e7c5100cb171bdbfee82fab71ec080", "sources": [ { "id": "dl-03-1", "key": "DL03_1", "title": "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift", "authors": [ { "given": "Sergey", "family": "Ioffe" }, { "given": "Christian", "family": "Szegedy" } ], "journal": "Proceedings of the 32nd International Conference on Machine Learning, PMLR 37", "year": 2015, "doi": "", "url": "https://proceedings.mlr.press/v37/ioffe15.html", "verified_date": "2026-10-22", "metadata_source": "Primary publisher or author arXiv record inspected with web tool", "author_list_complete": true, "csl_type": "paper-conference", "access_scope": "Primary bibliographic record and abstract inspected; no exhaustive full-text or supplementary-material audit.", "version_note": "", "article_slug": "normalization-geometry", "source_article_slug": "normalization-geometry" }, { "id": "dl-03-2", "key": "DL03_2", "title": "Layer Normalization", "authors": [ { "given": "Jimmy Lei", "family": "Ba" }, { "given": "Jamie Ryan", "family": "Kiros" }, { "given": "Geoffrey E.", "family": "Hinton" } ], "journal": "arXiv:1607.06450", "year": 2016, "doi": "10.48550/arXiv.1607.06450", "url": "https://arxiv.org/abs/1607.06450", "verified_date": "2026-10-22", "metadata_source": "Primary publisher or author arXiv record inspected with web tool", "author_list_complete": true, "csl_type": "article", "access_scope": "Primary bibliographic record and abstract inspected; no exhaustive full-text or supplementary-material audit.", "version_note": "", "article_slug": "normalization-geometry", "source_article_slug": "normalization-geometry" }, { "id": "dl-03-3", "key": "DL03_3", "title": "How Does Batch Normalization Help Optimization?", "authors": [ { "given": "Shibani", "family": "Santurkar" }, { "given": "Dimitris", "family": "Tsipras" }, { "given": "Andrew", "family": "Ilyas" }, { "given": "Aleksander", "family": "Madry" } ], "journal": "arXiv:1805.11604", "year": 2018, "doi": "10.48550/arXiv.1805.11604", "url": "https://arxiv.org/abs/1805.11604", "verified_date": "2026-10-22", "metadata_source": "Primary publisher or author arXiv record inspected with web tool", "author_list_complete": true, "csl_type": "article", "access_scope": "Primary bibliographic record and abstract inspected; no exhaustive full-text or supplementary-material audit.", "version_note": "Record first submitted 2018; inspected v5 (15 April 2019). Primary record notes NeurIPS 2018; this bibliography identifies the author manuscript.", "article_slug": "normalization-geometry", "source_article_slug": "normalization-geometry" } ], "original_source_url": "/guides/assets/v9-deep-learning-articles-normalization-geometry-json-7405e5d3524661d7.json", "pdf": { "document_id": "acea9454-4ce6-5d6e-b1d7-0231656e0598", "version_id": "2f986c06-713f-5177-9f58-a0f9e54dfd21", "media_type": "application/pdf", "authentication_required": true, "download_audited": true }, "presentation_change": "Complete version-bound translated reading views and selectable authenticated original/translated PDFs; original scientific text and evidence retained.", "source_locale": "en", "source_version_id": "4252c852-c5de-50cf-b815-878c4371040a", "translations": [ { "locale": "zh", "translation_id": "c7df8055-1e0a-5ea4-939e-8d35965aa3cb", "translated_content_sha256": "1787fce8f0bfb76590e1f4f70e5e35a66c0774ec7a20faae3c23cefd219a1bb6", "sha256": "7542a1fb51958b011f71258fa9b37deed4713ec11a7e6db5959426e52717ea37", "byte_length": 881816 }, { "locale": "es", "translation_id": "4ce9c25c-eec7-581a-81ca-4b7160d30221", "translated_content_sha256": "f5bee3e99c502a00622c958dae556c71c00fd04b42877ed168acfe62e1d64393", "sha256": "8e15782293bce0f9a5f184277556ad62524298162c4043312199f9526e750734", "byte_length": 647792 }, { "locale": "ja", "translation_id": "a6d719e9-2552-50ad-bb69-ab6a2ffeddfc", "translated_content_sha256": "ca760d21965d0f3c103d79a26a6904114b2111df8459ac3c2cc9cc1390dc39a2", "sha256": "9bf06607cf20d778c576dc0d531a684cba382980337eade386f504d7156a7691", "byte_length": 843696 }, { "locale": "ar", "translation_id": "8bb6400b-427e-5caa-943f-6798b632dec2", "translated_content_sha256": "501de2130e88ffeabc863ea24ef5686463c01a11f25d3909574a1d5ee95a2ab9", "sha256": "0df7283cc63a23551accdcce27bf184db710cbd51f16e09d095785a68ae4ae90", "byte_length": 635002 } ] }