What Low-Rank Adaptation Can and Cannot Represent

1. Specify the object whose rank is constrained

A rank constraint applies to a matrix at a particular layer, not directly to the set of functions a whole network can represent. A low-rank update to one linear operator may interact with nonlinearities, residual paths, normalization, and updates at other layers. The resulting functional change can therefore be richer than the rank of any one matrix suggests, while still being restricted in important directions.

A rigorous analysis separates four quantities: trainable parameter count, the rank of the realized weight update, the change in outputs on a stated input distribution, and the downstream task criterion. These are connected but not interchangeable. This article starts with a single linear layer because that setting permits exact counterexamples. Its matrices are constructed, not extracted from a trained language model. Their purpose is to test the logic of capacity claims before those claims are attached to empirical performance on a complex network.

2. What the three primary sources motivate

Hu and colleagues introduce LoRA by freezing pretrained weights and learning low-rank update factors, with empirical adaptation results and a mergeable linear update. Aghajanyan, Zettlemoyer, and Gupta study low-dimensional reparameterizations of language-model fine-tuning. Their notion of intrinsic dimension is not identical to the matrix rank of a layer update. Kalajdzievski studies rank-dependent scaling in LoRA and proposes a square-root rank scaling under the paper's analysis.

These contributions motivate three separate questions: whether a restricted parameterization can solve a task, how its effective dimension is measured, and whether optimization remains comparable when that parameterization changes. None provides a universal rule that a specific rank suffices for every domain or model. The examples below do not reproduce the cited performance tables. They derive exact properties of small matrices and use those properties to specify the controls needed in a future fine-tuning comparison. [1], [2], [3]

3. Parameter savings do not equal rank-manifold dimension

For a d by k weight matrix, factors B of shape d by r and A of shape r by k contain r times the sum of d and k scalar parameters. Their product has rank at most r. At a generic rank-r point, the matrix manifold has dimension r times d plus k minus r. The difference reflects a factorization redundancy: multiplying B by an invertible r by r matrix and multiplying A by its inverse leaves the product unchanged.

This gauge freedom matters when interpreting optimizer trajectories or comparing norms of the factors. Two factor pairs can implement the identical update with different individual norms and gradients. Counting trainable scalars remains useful for optimizer-state storage, but it is not the same as counting independent functional directions. The executable example verifies the invariance for a fixed invertible transformation. It does not claim that the resulting optimization problems are numerically equivalent under finite-precision arithmetic or a particular adaptive optimizer.

4. Singular-value tails give an exact linear benchmark

Suppose a desired update matrix is already known and the criterion is unweighted squared Frobenius error. The best rank-r approximation retains its largest r singular components. The residual squared error is the sum of the discarded squared singular values. This gives an exact benchmark in a setting much simpler than neural-network fine-tuning: there is no nonconvex task loss, changing representation, or unknown target update.

Figure 1 compares the fraction of discarded spectral energy for two constructed eight-dimensional spectra. One is flat; the other decreases geometrically. A small rank captures much more energy in the second case. The statement is about those spectra under a weight-space norm. It does not establish that real adaptation updates possess either spectrum, or that matching spectral energy is sufficient for preserving task behavior. The figure provides a reference against which an empirical claim about low-rank structure can be made precise.

5. Input covariance changes the relevant approximation problem

For a linear layer and centered inputs, expected squared output error depends on the input covariance. It is the trace of the residual update multiplied by that covariance and by the residual transpose. If inputs are isotropic with unit covariance, this reduces to squared Frobenius error. Otherwise, different input directions receive different weights.

Take a desired update with diagonal entries four and one. An unweighted rank-one approximation keeps the first direction, leaving squared matrix error one. Now let input covariance be diagonal with entries one and rho. The output error of that same approximation is rho. Keeping the second direction instead leaves output error sixteen. At rho = 100, the weight-space choice loses far more on the specified input distribution. Figure 2 varies rho and compares the unweighted choice with the optimal rank-one output criterion. This is an exact distribution-sensitive counterexample, not a measured failure rate of LoRA.

6. A change of input distribution can reverse the preferred update

The covariance example has a practical implication for domain shift. An update selected under one distribution may allocate its limited rank to directions that become less relevant under another. The issue is not solved by reporting only singular values of the update matrix. One also needs evidence about the inputs reaching that layer and the downstream sensitivity to its output.

For a single linear layer with a positive-definite input covariance, the weighted problem can be transformed by multiplying the desired update by the covariance square root, taking a low-rank approximation there, and transforming back. Singular covariance restricts the criterion to the observed support and leaves some directions unidentified. In a deep network, later nonlinear transformations introduce another sensitivity weighting, so this linear construction is only a diagnostic. It nevertheless clarifies why a rank recommendation should be accompanied by task and domain information instead of presented as an architecture-independent constant.

7. Factor initialization and scale affect learning

A common factorized initialization makes the initial update zero by setting one factor to zero while the other is nonzero. The product is then zero, but gradients with respect to the two factors need not have symmetric behavior. If B is zero, the initial gradient of the product-dependent loss with respect to A is zero, whereas the gradient with respect to B can be nonzero. Initial training therefore has a directional structure that is absent from a direct full-matrix parameterization.

Increasing rank also changes the number of terms in the product. A multiplicative scale can alter activation and gradient magnitudes even when the set of representable nonzero updates remains the same up to rescaling the factors. This is why a rank sweep should record scale, initialization variance, learning rate, and optimizer configuration. The manuscript's finite matrix checks establish representational identities; they do not establish an optimal optimizer schedule or validate any universal scaling law for deep networks.

8. Layer placement is part of the capacity budget

Two configurations with the same number of trainable parameters can place those parameters in different layers or projections. They may therefore expose very different functional directions. Allocating a small rank across many layers is not generally equivalent to allocating a larger rank in one layer. Even in a linear network, products of modified matrices introduce cross terms; nonlinear networks make the relationship more complicated.

A meaningful comparison should hold total resource budgets constant where possible and disclose the placement rule. It should also distinguish a design chosen before validation from one selected through a large hidden search. Otherwise, an apparent advantage of a rank allocation may reflect additional tuning effort. The small examples here isolate one layer so that the mathematical reference remains exact. A future network study should add layer interventions and activation diagnostics, treating the single-layer results as sanity checks rather than as a theorem about whole-model expressivity.

9. Deployment savings depend on the chosen execution path

A frozen base model plus many task-specific low-rank factors can reduce storage relative to many independent full checkpoints. A merged update can avoid evaluating the factorized branch separately for a single selected task. Those are different deployment arrangements, and neither by itself determines total serving cost when adapters are switched or batched dynamically.

Training memory also includes activations, gradients, optimizer states, temporary workspaces, and the frozen base weights. Parameter count is only one term. A benchmark should report these categories and the actual precision, sequence lengths, batch sizes, and memory allocator behavior. It should record whether adapter merging changes quantization or numerical rounding. The present article does not measure GPU memory, latency, or throughput. Its claims are confined to exact parameter counts and matrix identities, leaving implementation-dependent savings to a separately designed experiment with explicit accounting.

10. A defensible capacity claim is conditional

The central question is not whether low rank is good or bad. It is whether a particular restricted update family can achieve a specified objective under a specified distribution and optimization budget. The spectral example gives a best-case linear reference when the target update is known. The covariance example shows that even this reference changes when the criterion moves from weights to outputs.

An empirical report should therefore include rank, placement, scaling, search effort, domain composition, and uncertainty across runs. Where a full fine-tuning baseline is feasible, compare functional outcomes rather than only distances between learned weights, since parameter symmetries can make weight distance misleading. Where it is infeasible, state that limitation rather than treating a restricted baseline as unrestricted capacity. The supplied executable examples are intended to make such claims more disciplined: they identify the assumptions needed to move from a low parameter count to a meaningful statement about what an adaptation can represent.

Executed locally · Bounded scope

Rotated SVD residuals, factor gauge invariance, and weighted linear-output error

Constructed matrices and a centered linear-input criterion only. No claim about trained adapters, optimizer behavior, or accelerator savings.

47 assertions passed. These checks validate the supplied example, not the cited systems.

{
  "matrix_dimension": 8,
  "ranks": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7,
    8
  ],
  "rho100_unweighted_error": 100,
  "rho100_weighted_error": 16,
  "factor_gauge_matrix": [
    [
      2,
      1
    ],
    [
      0,
      3
    ]
  ]
}

Example runner · Model loader · Training models · Representation models · Generation models · Reliability modelsRead the complete check results

A declared analytical model

Four equations, explicit assumptions

Constructed examples · Reproducible data

Inspect the quantitative argument

These curves are calculated from the declared models. They are not observations from the cited studies. Every figure includes its numeric data and vector version.

Download CSVDownload SVG sourceReproduce with Python

Download CSVDownload SVG sourceReproduce with Python

Which evidence answers which question?

A proposed validation protocol

This is a proposed evaluation design. No production-system or hardware benchmark was performed. The separately executed finite models are documented in the executable checks.

1. Choose exact target matrices with flat and decaying spectra and verify truncated-SVD residuals independently.
2. Check factorization gauge invariance and rank bounds using fixed matrices and stated numerical tolerances.
3. Evaluate input-weighted errors under multiple covariances, including the rank-one reversal example.
4. For network experiments, prerecord adapter placement, rank, scale, initialization, optimizer, and tuning budget.
5. Compare held-out task quality and shifted-domain behavior across seeds alongside actual memory and time measurements.
6. Archive base revisions, adapter factors, merge settings, evaluation data versions, and the limits of any unavailable full-tuning baseline.

What the primary sources establish

Limits of this analysis

• The executable matrices test algebraic capacity and distribution weighting, not the quality of a trained neural adapter.
• The covariance criterion assumes a single linear layer and squared output error; downstream nonlinearities add complexity.
• No GPU memory, training throughput, or deployment latency is measured in this local artifact.
• Intrinsic parameter dimension, matrix rank, and semantic task complexity are distinct and cannot be substituted without additional evidence.

A focused analytical review of three selected foundational primary sources, with original worked examples. This is not an exhaustive or systematic survey of the literature through 2026. Source access and version notes are recorded with the bibliography.

Questions that would change the conclusion

1. Can input and downstream sensitivity measurements guide rank allocation more reliably than update spectra alone?
2. How stable are useful adaptation subspaces under domain and sequence-length shift?
3. What comparison protocol separates representational limits from optimization and hyperparameter-search limits?

References

1. Edward J. Hu, Yelong Shen, Phillip Wallis et al.. LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint (2021). 10.48550/arXiv.2106.09685.
2. Armen Aghajanyan, Luke Zettlemoyer, Sonal Gupta. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. arXiv preprint (2020). 10.48550/arXiv.2012.13255.
3. Damjan Kalajdzievski. A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. arXiv preprint (2023). 10.48550/arXiv.2312.03732.

Source [1] access. Primary metadata, abstract, and selected accessible method text; no exhaustive supplementary-material audit.

Version note. v2, revised 16 October 2021; cited bibliographic object is the preprint.

Source [2] access. Primary metadata, abstract, and selected accessible method text; no exhaustive supplementary-material audit.

Version note. v1, submitted 22 December 2020; cited bibliographic object is the preprint.

Source [3] access. Primary metadata, abstract, and selected accessible method text; no exhaustive supplementary-material audit.

Version note. v1, submitted 28 November 2023.

Bibliographic identities were checked against primary publisher and author records. Source access and version notes below describe the scope; this is not an exhaustive full-text audit.

Authorship and provenance

Written and technically self-reviewed by Codex (AI). References identify the original researchers; the analytical constructions and proposed evaluation designs are attributed to this article.

Covers are conceptual ImageGen illustrations. Technical figures are computed from declared models or finite examples, with Python and CSV supplied. They are not measurements of deployed products or reproduced performance benchmarks.

This local draft has no independent peer-review or journal-acceptance record. Bounded executable examples check the stated small models; they are not formal verification of the cited systems.