The Low-Rank Hypothesis in Parameter-Efficient Adaptation: LoRA’s Optimization Structure, Capacity, and Merge Equivalence

1. LoRA changes the representation of the update

LoRA freezes the original weights W and represents the trainable increment through the low-rank matrix product BA. For a matrix of shape $d_{\mathrm{out}}\times d_{\mathrm{in}}$ , a full update has $d_{\mathrm{out}}\times d_{\mathrm{in}}$ trainable values, whereas the low-rank component has $r(d_{\mathrm{in}}+d_{\mathrm{out}})$ . This count excludes other layers, biases, and additional framework states, and cannot establish that any arbitrary low rank can achieve the quality of full fine-tuning.[1] LoRA: Low-Rank Adaptation of Large Language Models

2. An independently reproducible calculation

Let $d_{\mathrm{in}}=d_{\mathrm{out}}=4096$; the original matrix then contains 16,777,216 values. Taking $r=8$, the two low-rank matrices contain a total of 65,536 trainable values, which is the following fraction of the full matrix: $\frac1{256}$；$r=4$ and 16, the counts are 32,768 and 131,072, respectively. Figure 2 shows these quantities using the same formula. These are parameter counts for a single matrix and cannot be described as a 256-fold reduction in training memory for the entire model.[1] LoRA: Low-Rank Adaptation of Large Language Models

3. Quantization reduces another cost component

QLoRA stores frozen pretrained weights in 4-bit form and propagates gradients to low-rank adapters; the paper also discusses NF4, double quantization, and paged optimizers. Its savings depend on the representation, quantization metadata, computation precision, and optimizer management, and cannot be summarized as “the entire training process uses 4 bits.” The bit width of frozen weights, operator computation types, and adapter states must be recorded separately.[2] QLoRA: Efficient Finetuning of Quantized LLMs

4. Raw weight accounting is not peak accelerator memory

Assuming exactly 70 hundred million weights, raw storage at 16 bits is 14 GB, and at 4 bits it is 3.5 GB,$1\,\mathrm{GB}=10^9$bytes. This excludes quantization constants, activations, caches, gradients, adapters, optimizers, and temporary buffers, and cannot guarantee that a particular graphics card can train or run the model. Figure 2 in this version shows only the proportion of low-rank parameters; for separate accounting of weight bit widths, see the review of low-bit inference in this collection. These are not the same memory mechanism.作者分析与复核建议 / Author analysis

5. FlashAttention optimizes a different component

FlashAttention reduces accelerator-memory reads and writes and intermediate storage in attention computation through techniques such as IO-aware tiling. It is not a replacement for LoRA: one changes the trainable increment, while the other changes the execution of exact attention. Combining the three methods may affect different cost components; compatibility and the attainment of expected quality and speed still require measurement on the specific implementation and inputs.[3] FlashAttention: Fast and Memory-Efficient Exact Attenti[1] LoRA: Low-Rank Adaptation of Large Language Models[2] QLoRA: Efficient Finetuning of Quantized LLMs

6. Quality comparisons cannot rely on training loss alone

Low rank, quantization, and data selection may affect task performance in different ways. This review recommends fixing the base checkpoint and training materials, comparing full fine-tuning with configurations of different ranks and precisions, and recording task metrics, output distributions, and failure types. If an adapter is trained only on a particular class of prompts, out-of-domain tests should be retained. A decrease in training loss does not establish that factuality or general capabilities have not deteriorated.作者分析与复核建议 / Author analysis

7. Merging, switching, and deployment also require validation

LoRA increments can be merged into weights in appropriate implementations, but merging with quantized weights, requantization, and switching among multiple adapters each have their own execution details. Versions of the base model, adapter, quantization configuration, and merging procedure should be preserved, and deployed outputs should be checked for consistency with the evaluated artifact. Conclusions about latency should come from measurements of the actual deployment path, rather than inferences based solely on parameter counts.[1] LoRA: Low-Rank Adaptation of Large Language Models

Sparse updates and actual file size are also distinct

The number of trainable values in an adapter can be counted exactly, but a saved file also contains tensor names, metadata, and precision information. The byte count obtained by compressing an adapter file cannot be compared directly with another, uncompressed full checkpoint to claim the same factor of parameter savings. Likewise, peak accelerator memory includes multiple types of objects present simultaneously; even if frozen weights become smaller, activations may become the new dominant component.作者分析与复核建议 / Author analysis

8. Update rank is a structural constraint, not an accuracy guarantee

Let the base matrix W have shape m×n, and write the update as BA, where B is m×r and A is r×n. The update rank is then at most r, and the number of trainable parameters is r(m+n). Savings relative to the full matrix’s mn parameters can be calculated directly; however, under the same parameter budget, the choice of layers and projections to which updates are allocated also changes expressive capacity. If the changes required by a task cannot be adequately represented in the selected subspace, increasing the number of training steps may not resolve the issue. Rank sweeps should be separated from layer selection, and a low parameter ratio should not be interpreted directly as a proportional reduction in training overhead or as lossless adaptation.[1] LoRA: Low-Rank Adaptation of Large Language Models

9. Initialization determines which parameters receive gradients in the first step

To make the initial model equal to the base model, common implementations initialize one low-rank factor to zero and the other to nonzero values. If B=0, the gradient with respect to A is zero in the first step, while the gradient with respect to B may still be nonzero; both may subsequently change. Thus, “both factors are trainable” does not mean that they receive the same learning signal at every moment. Implementation verification should also preserve the scaling coefficient, optimizer, weight decay, and dropout; these choices may change the effective update scale across ranks.[1] LoRA: Low-Rank Adaptation of Large Language Models

10. Merge equivalence requires explicit assumptions

For a linear layer with fixed coefficients, dropout disabled during inference, and a consistent numerical representation, Wx+sBAx and (W+sBA)x are equal in real-number algebra. Under finite precision, summation, matrix multiplication, and requantization introduce errors; if the base model is stored at low bit width, quantizing again after merging may not preserve the original computation. The merge check here should compare actual output errors, then measure latency and resources separately. Algebraic mergeability alone does not justify declaring a particular quantized deployment path entirely lossless.[1] LoRA: Low-Rank Adaptation of Large Language Models[2] QLoRA: Efficient Finetuning of Quantized LLMs

11. Parameter efficiency and memory bottlenecks are different axes

Freezing base weights can reduce the corresponding gradients and optimizer states, but forward and backward propagation still require many activations; longer sequences also increase attention-related costs. QLoRA discusses the combination of quantized base weights and adaptation training, whereas FlashAttention discusses IO and activation organization. They should not be treated as the same kind of mechanism in a parameter-count figure. Weights, trainable parameters, optimizers, activations, and peak accelerator memory should be recorded separately, and total costs compared at the same task quality.[2] QLoRA: Efficient Finetuning of Quantized LLMs[3] FlashAttention: Fast and Memory-Efficient Exact Attenti[4] Distilling the Knowledge in a Neural Network

Core equations and derivations

Method and evidence workflow

Analytical figures and numerical examples

Reproducible worked example

For a 4096×4096 matrix with r=16, the low-rank factors contain 131072 parameters, approximately 0.78125% of the total. The maximum absolute difference in an independent floating-point check with small matrices is 3.55e-15; this verifies the linear implementation, not lossless quantized merging or downstream capabilities.

Method comparison

Verification and experimental plan

1. Fix the base model, tokenizer, data version, and training split, and specify which modules are adapted and which parameters are frozen.
2. Record rank, scaling, initialization, dropout, optimizer, and training budget, and perform ablations with budget-matched configurations.
3. Vary rank and module allocation separately, retaining an unadapted baseline and at least one full-parameter or alternative-adaptation comparison.
4. Account for memory associated with training weights, gradients, optimizers, and activations, and report the actual peak rather than counting only low-rank parameters.
5. Compare outputs before and after merging in inference mode with randomness disabled, report absolute and relative errors, and test the requantization path separately.
6. Test transfer and forgetting on independent tasks and basic capabilities, preserve failure cases, and do not infer overall adaptation quality from training loss.

Scope and limitations

• Update rank does not directly equal the intrinsic dimensionality required by the task.
• Parameter counting is not a measurement of device performance.
• Quantization and dropout may change the merge path.
• No new model training or conclusion of lossless adaptation is presented.

Mapping primary sources to claims

1. LoRA: Low-Rank Adaptation of Large Language ModelsHu et al. · 2021-06-17v2; low-rank update formulation
2. QLoRA: Efficient Finetuning of Quantized LLMsDettmers et al. · 2023-05-23v1; NF4, double quantization, paged optimizers and limitations
3. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessDao et al. · 2022-05-27v1; sections 2-3
4. Distilling the Knowledge in a Neural Network, , · 2015-03-09Abstract; 原文方法、理论和实验章节见作者原始论文链接