{
  "schema": "auturnal.deep-research-review.v2",
  "slug": "low-rank-adaptation-capacity",
  "title": "What Low-Rank Adaptation Can and Cannot Represent",
  "category": "Generation & Adaptation",
  "dek": "An update's rank bounds a matrix approximation, while task quality depends on input geometry, layer placement, and optimization.",
  "abstract": "Low-rank adaptation reduces the number of trainable parameters by factorizing weight updates, but rank alone does not determine functional capacity or downstream quality. This review separates matrix approximation, input-weighted output error, factorization geometry, optimization scaling, and deployment accounting. Three primary sources motivate the discussion of LoRA, intrinsic dimension, and rank-dependent scaling. Independently constructed examples show how singular-value decay controls an unweighted approximation error and how a rank-one update optimal in weight space can be poor under a changed input covariance. Reproducible figures, exact matrix checks, and a proposed evaluation protocol define what these deductions establish without claiming new language-model benchmarks.",
  "english": "Low-rank adaptation reduces the number of trainable parameters by factorizing weight updates, but rank alone does not determine functional capacity or downstream quality. This review separates matrix approximation, input-weighted output error, factorization geometry, optimization scaling, and deployment accounting. Three primary sources motivate the discussion of LoRA, intrinsic dimension, and rank-dependent scaling. Independently constructed examples show how singular-value decay controls an unweighted approximation error and how a rank-one update optimal in weight space can be poor under a changed input covariance. Reproducible figures, exact matrix checks, and a proposed evaluation protocol define what these deductions establish without claiming new language-model benchmarks.",
  "refs": [
    "dl-14-1",
    "dl-14-2",
    "dl-14-3"
  ],
  "sections": [
    [
      "1. Specify the object whose rank is constrained",
      "A rank constraint applies to a matrix at a particular layer, not directly to the set of functions a whole network can represent. A low-rank update to one linear operator may interact with nonlinearities, residual paths, normalization, and updates at other layers. The resulting functional change can therefore be richer than the rank of any one matrix suggests, while still being restricted in important directions.\n\nA rigorous analysis separates four quantities: trainable parameter count, the rank of the realized weight update, the change in outputs on a stated input distribution, and the downstream task criterion. These are connected but not interchangeable. This article starts with a single linear layer because that setting permits exact counterexamples. Its matrices are constructed, not extracted from a trained language model. Their purpose is to test the logic of capacity claims before those claims are attached to empirical performance on a complex network.",
      []
    ],
    [
      "2. What the three primary sources motivate",
      "Hu and colleagues introduce LoRA by freezing pretrained weights and learning low-rank update factors, with empirical adaptation results and a mergeable linear update. Aghajanyan, Zettlemoyer, and Gupta study low-dimensional reparameterizations of language-model fine-tuning. Their notion of intrinsic dimension is not identical to the matrix rank of a layer update. Kalajdzievski studies rank-dependent scaling in LoRA and proposes a square-root rank scaling under the paper's analysis.\n\nThese contributions motivate three separate questions: whether a restricted parameterization can solve a task, how its effective dimension is measured, and whether optimization remains comparable when that parameterization changes. None provides a universal rule that a specific rank suffices for every domain or model. The examples below do not reproduce the cited performance tables. They derive exact properties of small matrices and use those properties to specify the controls needed in a future fine-tuning comparison.",
      [
        "dl-14-1",
        "dl-14-2",
        "dl-14-3"
      ]
    ],
    [
      "3. Parameter savings do not equal rank-manifold dimension",
      "For a d by k weight matrix, factors B of shape d by r and A of shape r by k contain r times the sum of d and k scalar parameters. Their product has rank at most r. At a generic rank-r point, the matrix manifold has dimension r times d plus k minus r. The difference reflects a factorization redundancy: multiplying B by an invertible r by r matrix and multiplying A by its inverse leaves the product unchanged.\n\nThis gauge freedom matters when interpreting optimizer trajectories or comparing norms of the factors. Two factor pairs can implement the identical update with different individual norms and gradients. Counting trainable scalars remains useful for optimizer-state storage, but it is not the same as counting independent functional directions. The executable example verifies the invariance for a fixed invertible transformation. It does not claim that the resulting optimization problems are numerically equivalent under finite-precision arithmetic or a particular adaptive optimizer.",
      []
    ],
    [
      "4. Singular-value tails give an exact linear benchmark",
      "Suppose a desired update matrix is already known and the criterion is unweighted squared Frobenius error. The best rank-r approximation retains its largest r singular components. The residual squared error is the sum of the discarded squared singular values. This gives an exact benchmark in a setting much simpler than neural-network fine-tuning: there is no nonconvex task loss, changing representation, or unknown target update.\n\nFigure 1 compares the fraction of discarded spectral energy for two constructed eight-dimensional spectra. One is flat; the other decreases geometrically. A small rank captures much more energy in the second case. The statement is about those spectra under a weight-space norm. It does not establish that real adaptation updates possess either spectrum, or that matching spectral energy is sufficient for preserving task behavior. The figure provides a reference against which an empirical claim about low-rank structure can be made precise.",
      []
    ],
    [
      "5. Input covariance changes the relevant approximation problem",
      "For a linear layer and centered inputs, expected squared output error depends on the input covariance. It is the trace of the residual update multiplied by that covariance and by the residual transpose. If inputs are isotropic with unit covariance, this reduces to squared Frobenius error. Otherwise, different input directions receive different weights.\n\nTake a desired update with diagonal entries four and one. An unweighted rank-one approximation keeps the first direction, leaving squared matrix error one. Now let input covariance be diagonal with entries one and rho. The output error of that same approximation is rho. Keeping the second direction instead leaves output error sixteen. At rho = 100, the weight-space choice loses far more on the specified input distribution. Figure 2 varies rho and compares the unweighted choice with the optimal rank-one output criterion. This is an exact distribution-sensitive counterexample, not a measured failure rate of LoRA.",
      []
    ],
    [
      "6. A change of input distribution can reverse the preferred update",
      "The covariance example has a practical implication for domain shift. An update selected under one distribution may allocate its limited rank to directions that become less relevant under another. The issue is not solved by reporting only singular values of the update matrix. One also needs evidence about the inputs reaching that layer and the downstream sensitivity to its output.\n\nFor a single linear layer with a positive-definite input covariance, the weighted problem can be transformed by multiplying the desired update by the covariance square root, taking a low-rank approximation there, and transforming back. Singular covariance restricts the criterion to the observed support and leaves some directions unidentified. In a deep network, later nonlinear transformations introduce another sensitivity weighting, so this linear construction is only a diagnostic. It nevertheless clarifies why a rank recommendation should be accompanied by task and domain information instead of presented as an architecture-independent constant.",
      []
    ],
    [
      "7. Factor initialization and scale affect learning",
      "A common factorized initialization makes the initial update zero by setting one factor to zero while the other is nonzero. The product is then zero, but gradients with respect to the two factors need not have symmetric behavior. If B is zero, the initial gradient of the product-dependent loss with respect to A is zero, whereas the gradient with respect to B can be nonzero. Initial training therefore has a directional structure that is absent from a direct full-matrix parameterization.\n\nIncreasing rank also changes the number of terms in the product. A multiplicative scale can alter activation and gradient magnitudes even when the set of representable nonzero updates remains the same up to rescaling the factors. This is why a rank sweep should record scale, initialization variance, learning rate, and optimizer configuration. The manuscript's finite matrix checks establish representational identities; they do not establish an optimal optimizer schedule or validate any universal scaling law for deep networks.",
      []
    ],
    [
      "8. Layer placement is part of the capacity budget",
      "Two configurations with the same number of trainable parameters can place those parameters in different layers or projections. They may therefore expose very different functional directions. Allocating a small rank across many layers is not generally equivalent to allocating a larger rank in one layer. Even in a linear network, products of modified matrices introduce cross terms; nonlinear networks make the relationship more complicated.\n\nA meaningful comparison should hold total resource budgets constant where possible and disclose the placement rule. It should also distinguish a design chosen before validation from one selected through a large hidden search. Otherwise, an apparent advantage of a rank allocation may reflect additional tuning effort. The small examples here isolate one layer so that the mathematical reference remains exact. A future network study should add layer interventions and activation diagnostics, treating the single-layer results as sanity checks rather than as a theorem about whole-model expressivity.",
      []
    ],
    [
      "9. Deployment savings depend on the chosen execution path",
      "A frozen base model plus many task-specific low-rank factors can reduce storage relative to many independent full checkpoints. A merged update can avoid evaluating the factorized branch separately for a single selected task. Those are different deployment arrangements, and neither by itself determines total serving cost when adapters are switched or batched dynamically.\n\nTraining memory also includes activations, gradients, optimizer states, temporary workspaces, and the frozen base weights. Parameter count is only one term. A benchmark should report these categories and the actual precision, sequence lengths, batch sizes, and memory allocator behavior. It should record whether adapter merging changes quantization or numerical rounding. The present article does not measure GPU memory, latency, or throughput. Its claims are confined to exact parameter counts and matrix identities, leaving implementation-dependent savings to a separately designed experiment with explicit accounting.",
      []
    ],
    [
      "10. A defensible capacity claim is conditional",
      "The central question is not whether low rank is good or bad. It is whether a particular restricted update family can achieve a specified objective under a specified distribution and optimization budget. The spectral example gives a best-case linear reference when the target update is known. The covariance example shows that even this reference changes when the criterion moves from weights to outputs.\n\nAn empirical report should therefore include rank, placement, scaling, search effort, domain composition, and uncertainty across runs. Where a full fine-tuning baseline is feasible, compare functional outcomes rather than only distances between learned weights, since parameter symmetries can make weight distance misleading. Where it is infeasible, state that limitation rather than treating a restricted baseline as unrestricted capacity. The supplied executable examples are intended to make such claims more disciplined: they identify the assumptions needed to move from a low parameter count to a meaningful statement about what an adaptation can represent.",
      []
    ]
  ],
  "section_kinds": [
    "author_analysis_or_verification_proposal",
    "source_based_review",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal",
    "author_analysis_or_verification_proposal"
  ],
  "equations": [
    {
      "tex": "\\Delta W=cBA,\\quad B\\in\\mathbb R^{d\\times r},\\quad A\\in\\mathbb R^{r\\times k},\\quad P=r(d+k)",
      "explanation": "LoRA-style factorization with a stated scale c. Rank is at most r; P counts stored trainable factor entries, not independent rank-manifold coordinates."
    },
    {
      "tex": "(BQ)(Q^{-1}A)=BA,\\qquad \\dim\\{M:\\operatorname{rank}M=r\\}=r(d+k-r)",
      "explanation": "Q is invertible and r <= min(d,k). The dimension statement is for the smooth manifold of exactly rank-r matrices."
    },
    {
      "tex": "\\min_{\\operatorname{rank}(M)\\le r}\\|\\Delta W-M\\|_F^2=\\sum_{i>r}\\sigma_i(\\Delta W)^2",
      "explanation": "Best low-rank approximation under an unweighted Frobenius criterion; singular values are ordered nonincreasingly."
    },
    {
      "tex": "\\mathbb E\\|(\\Delta W-M)x\\|_2^2=\\operatorname{tr}\\!\\left[(\\Delta W-M)\\Sigma_x(\\Delta W-M)^\\top\\right]",
      "explanation": "Centered inputs with finite covariance Sigma_x. Figure 2 uses Delta W = diag(4,1), Sigma_x = diag(1,rho), giving unweighted-choice error rho and optimal rank-one error min(16,rho)."
    }
  ],
  "table": {
    "headers": [
      "Claim",
      "Necessary specification",
      "Counterexample or limitation"
    ],
    "rows": [
      [
        "Small rank captures the update",
        "Target matrix and approximation norm",
        "A flat spectrum retains substantial tail energy"
      ],
      [
        "Small weight error preserves outputs",
        "Input covariance and downstream sensitivity",
        "A low-variance weight direction can dominate shifted inputs"
      ],
      [
        "Few parameters imply cheap deployment",
        "Merging, batching, precision, and memory accounting",
        "Activation and routing costs remain implementation-dependent"
      ]
    ]
  },
  "protocol": [
    "Choose exact target matrices with flat and decaying spectra and verify truncated-SVD residuals independently.",
    "Check factorization gauge invariance and rank bounds using fixed matrices and stated numerical tolerances.",
    "Evaluate input-weighted errors under multiple covariances, including the rank-one reversal example.",
    "For network experiments, prerecord adapter placement, rank, scale, initialization, optimizer, and tuning budget.",
    "Compare held-out task quality and shifted-domain behavior across seeds alongside actual memory and time measurements.",
    "Archive base revisions, adapter factors, merge settings, evaluation data versions, and the limits of any unavailable full-tuning baseline."
  ],
  "limitations": [
    "The executable matrices test algebraic capacity and distribution weighting, not the quality of a trained neural adapter.",
    "The covariance criterion assumes a single linear layer and squared output error; downstream nonlinearities add complexity.",
    "No GPU memory, training throughput, or deployment latency is measured in this local artifact.",
    "Intrinsic parameter dimension, matrix rank, and semantic task complexity are distinct and cannot be substituted without additional evidence."
  ],
  "evidence": [
    {
      "source": "dl-14-1",
      "result": "LoRA factorizes trainable updates while freezing pretrained weights and permits a merged linear update.",
      "boundary": "Reported adaptation quality and resource savings are specific to the paper's configurations.",
      "locator": "Method and empirical rank investigation; arXiv v2."
    },
    {
      "source": "dl-14-2",
      "result": "Low-dimensional reparameterizations provide a lens on language-model fine-tuning.",
      "boundary": "Intrinsic dimension in this setting is not identical to the rank of a layer's update matrix.",
      "locator": "Intrinsic-dimension formulation and fine-tuning experiments; original preprint record."
    },
    {
      "source": "dl-14-3",
      "result": "Rank-dependent scaling can materially affect LoRA optimization; the paper analyzes square-root scaling.",
      "boundary": "The analysis does not remove the need to tune and disclose initialization and optimizer assumptions.",
      "locator": "Scaling-factor analysis; arXiv v1."
    }
  ],
  "research_questions": [
    "Can input and downstream sensitivity measurements guide rank allocation more reliably than update spectra alone?",
    "How stable are useful adaptation subspaces under domain and sequence-length shift?",
    "What comparison protocol separates representational limits from optimization and hyperparameter-search limits?"
  ],
  "kind": "ai_assisted_research_guide",
  "author": "Codex (AI)",
  "original_language": "en",
  "created_date": "2026-10-22",
  "published_date": null,
  "publication_status": "local_draft_pending_owner_upload",
  "version": 1,
  "previous_version": null,
  "analytic": "low-rank-adaptation-capacity",
  "peer_reviewed": false,
  "journal_publication": false,
  "disclosure": {
    "authorship": "Written and technically self-reviewed by Codex (AI). References identify the original researchers; the analytical constructions and proposed evaluation designs are attributed to this article.",
    "figures": "Covers are conceptual ImageGen illustrations. Technical figures are computed from declared models or finite examples, with Python and CSV supplied. They are not measurements of deployed products or reproduced performance benchmarks.",
    "review": "This local draft has no independent peer-review or journal-acceptance record. Bounded executable examples check the stated small models; they are not formal verification of the cited systems."
  },
  "review_scope": "A focused analytical review of three selected foundational primary sources, with original worked examples. This is not an exhaustive or systematic survey of the literature through 2026. Source access and version notes are recorded with the bibliography.",
  "revision_note": "Initial local English manuscript; no server upload or publication action."
}