{
  "slug": "efficient-adaptation-lora",
  "category": "高效训练",
  "title": "LoRA 与 QLoRA 到底节省了什么：参数、精度和内存的分层账本",
  "dek": "用可复算的参数计数与权重存储示例，分清低秩适配、量化和注意力内核优化各自解决的成本。",
  "english": "Parameter-efficient adaptation, quantized storage, and IO-aware attention target different cost terms. We provide exact illustrative parameter counts and distinguish raw weight memory from total runtime memory.",
  "refs": [
    "lora",
    "qlora",
    "flash"
  ],
  "sections": [
    [
      "摘要",
      "“省显存”可能来自减少可训练参数、降低冻结权重精度，也可能来自减少中间状态搬运。本文以 LoRA、QLoRA 与 FlashAttention 为线索，建立分层成本视角。所有计算示例都给出矩阵维度或位宽假设；它们解释机制，不宣称任何设备上可达到未经测试的训练规模与速度。",
      [
        "lora",
        "qlora",
        "flash"
      ]
    ],
    [
      "1. LoRA 改变的是更新的表示",
      "LoRA 冻结原权重 W，通过低秩矩阵乘积 BA 表示可训练增量。对于 d_out×d_in 的矩阵，完整更新有 d_out×d_in 个可训练数，而低秩部分有 r(d_in+d_out) 个。这个计数不包含其他层、偏置或框架额外状态，也不能据此证明任意低秩都能达到完整微调的质量。",
      [
        "lora"
      ]
    ],
    [
      "2. 一个可独立复算的例子",
      "令 d_in=d_out=4096，则原矩阵有 16,777,216 个数。取 r=8，两个低秩矩阵共有 65,536 个可训练数，是完整矩阵的 1/256；r=4 与 16 时分别为 32,768 与 131,072。图 2 用同一公式展示这些量。它是单矩阵参数计数，不能写成整个模型训练显存缩小了 256 倍。",
      [
        "lora"
      ]
    ],
    [
      "3. 量化压缩另一项成本",
      "QLoRA 把冻结预训练权重以 4 位形式保存，并将梯度传播到低秩适配器；论文还讨论 NF4、双重量化与分页优化器。其节省依赖表示方式、量化元数据、计算精度与优化器管理，不能概括为“整个训练过程都用 4 位”。冻结权重位宽、算子计算类型和适配器状态需要分别记录。",
      [
        "qlora"
      ]
    ],
    [
      "4. 原始权重账本不等于峰值显存",
      "假设恰有 70 亿个权重，16 位原始存储为 14 GB，4 位为 3.5 GB，这里 1 GB=10⁹ 字节。这个理想化计算不含量化常数、激活、缓存、梯度、适配器、优化器、临时缓冲或框架分配，因此不能据此保证某张显卡可训练或运行该模型。图中数字只用于分清核算边界。",
      []
    ],
    [
      "5. FlashAttention 优化不同的项",
      "FlashAttention 通过 IO 感知的分块等方式减少注意力计算中的显存读写与中间保存。它不是 LoRA 的替代品：一个改变可训练增量，一个改变精确注意力的执行。把三种方法组合使用，可能作用于不同成本项；是否兼容、是否达到预期质量与速度，仍需在具体实现与输入上测量。",
      [
        "flash",
        "lora",
        "qlora"
      ]
    ],
    [
      "6. 质量对照不能只看训练损失",
      "低秩、量化和数据选择可能以不同方式影响任务表现。本文建议固定基础检查点和训练材料，比较完整微调与不同秩、精度的方案，记录任务指标、输出分布与失败类型。若适配器只在某类提示上训练，应保留域外测试。不能因为训练损失下降，就认定事实性或通用能力没有退化。",
      []
    ],
    [
      "7. 合并、切换与部署也要验收",
      "LoRA 增量在适当实现中可以与权重合并，但量化权重上的合并、重新量化或多适配器切换具有各自执行细节。应保存基础模型、适配器、量化配置与合并过程的版本，核对部署输出是否与被评测产物一致。对延迟的结论，应由实际部署路径测量，而不是仅从参数数目推断。",
      [
        "lora"
      ]
    ],
    [
      "稀疏更新与实际文件体积也有区别",
      "适配器中的可训练数目可以精确计数，但保存文件还含有张量名称、元数据与精度信息。将一个适配器文件压缩后得到的字节数，不能直接与另一个未压缩完整检查点比较后宣称相同的参数节省倍数。同理，峰值显存包含同时存在的多种对象；即使冻结权重变小，激活也可能成为新的主要项。",
      []
    ],
    [
      "后续实验可以如何设计",
      "先核对基础模型与适配器矩阵维度，用脚本统计实际可训练数。随后在固定批量、序列长度与梯度累积设置下测量峰值显存，并分别记录加载阶段、前向、反向与优化器步骤。再在保留任务集上比较质量与延迟。若出现内存不足，应按失败记录保留，而不是不断降低输入条件后仍称原配置通过。",
      []
    ],
    [
      "局限与结论",
      "本稿未运行微调实验，也不提供显卡采购或容量承诺。成本核算需要区分原始权重、训练状态、激活与计算搬运。只有分别列账、对齐质量目标，并在目标环境测量峰值资源，参数高效与内存高效这两个判断才有可检查的含义。",
      []
    ]
  ],
  "diagram": [
    "原始权重 W",
    "冻结部分仍需存储并参与计算",
    "LoRA 低秩更新",
    "BA 的参数量为 r(d_in+d_out)",
    "QLoRA 量化存储",
    "冻结权重压缩，计算类型另记",
    "训练状态",
    "梯度、优化器和适配器分别计入",
    "注意力与激活",
    "IO 优化针对另一项内存成本",
    "部署产物验收",
    "核对基础版本、适配器与输出"
  ],
  "figureNote": "图 1 为成本项分解示意。图 2 按 4096×4096 单矩阵计算 LoRA 参数；另列恰有 70 亿权重在 16/4 位下的理想原始存储（十进制 GB），未计元数据、激活或运行开销，非显卡容量承诺。",
  "table": {
    "headers": [
      "成本项",
      "计数或测量方式",
      "不能直接替代"
    ],
    "rows": [
      [
        "可训练参数",
        "实际矩阵元素计数",
        "总训练显存"
      ],
      [
        "冻结权重",
        "位宽、元数据与存储",
        "算子计算精度"
      ],
      [
        "峰值运行内存",
        "目标配置真实测量",
        "单纯权重文件体积"
      ]
    ]
  },
  "version": 1,
  "kind": "ai_assisted_research_guide",
  "author": "Codex（AI）",
  "published_date": "2026-10-16",
  "disclosure": "本站用户明确委托撰写与公开。由 Codex（AI）依据所列一手资料整理，配图由代码绘制；未经独立同行评审。文中原论文的实验结果归属于原作者，本文未运行新的模型训练或基准测试。",
  "limitations": "文献导读与方法分析；不是原创实验论文、正式期刊录用或专业资质认证。解析示例不代表本站或当前产品的实测表现。",
  "sources": [
    {
      "id": "lora",
      "title": "LoRA: Low-Rank Adaptation of Large Language Models",
      "authors": "Hu et al.",
      "date": "2021-06-17",
      "url": "https://arxiv.org/html/2106.09685v2",
      "locator": "v2; low-rank update formulation",
      "note": "A frozen matrix W receives a trainable low-rank BA update. The number of trainable values r(d_in+d_out) differs from the full matrix size.",
      "accessed_at": "2026-10-15",
      "rights": "Original source remains under its own license; only attributed factual synthesis, no source figures or long excerpts."
    },
    {
      "id": "qlora",
      "title": "QLoRA: Efficient Finetuning of Quantized LLMs",
      "authors": "Dettmers et al.",
      "date": "2023-05-23",
      "url": "https://arxiv.org/html/2305.14314v1",
      "locator": "v1; NF4, double quantization, paged optimizers and limitations",
      "note": "4-bit frozen weights and low-rank adapters reduce finetuning memory under the paper's setup; total memory includes more than raw weight storage.",
      "accessed_at": "2026-10-15",
      "rights": "Original source remains under its own license; only attributed factual synthesis, no source figures or long excerpts."
    },
    {
      "id": "flash",
      "title": "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness",
      "authors": "Dao et al.",
      "date": "2022-05-27",
      "url": "https://arxiv.org/html/2205.14135v1",
      "locator": "v1; sections 2-3",
      "note": "Exact attention reorganizes memory IO using tiling; this is not a linear-time approximation of attention.",
      "accessed_at": "2026-10-15",
      "rights": "Original source remains under its own license; only attributed factual synthesis, no source figures or long excerpts."
    }
  ],
  "figures": [
    {
      "slug": "efficient-adaptation-lora",
      "number": 1,
      "file": "efficient-adaptation-lora-1.png",
      "sha256": "2673dedc8c563d54852407445d5acc7e62e312aaf5a6c9dc14fd65b18ec58a52",
      "alt": "六个要点：原始权重 W；LoRA 低秩更新；QLoRA 量化存储；训练状态；注意力与激活；部署产物验收",
      "data": {
        "kind": "conceptual",
        "items": [
          "原始权重 W",
          "冻结部分仍需存储并参与计算",
          "LoRA 低秩更新",
          "BA 的参数量为 r(d_in+d_out)",
          "QLoRA 量化存储",
          "冻结权重压缩，计算类型另记",
          "训练状态",
          "梯度、优化器和适配器分别计入",
          "注意力与激活",
          "IO 优化针对另一项内存成本",
          "部署产物验收",
          "核对基础版本、适配器与输出"
        ]
      },
      "ai_generated": true,
      "disclosure": "AI-assisted code rendering; original schematic or explicitly attributed data; not a new model experiment.",
      "url": "/guides/assets/efficient-adaptation-lora-1-2673dedc8c56.png"
    },
    {
      "slug": "efficient-adaptation-lora",
      "number": 2,
      "file": "efficient-adaptation-lora-2.png",
      "sha256": "996b40d39588f3210d0ea5be113b22a93eeffdd4c638942d5c90fbeaa00180a2",
      "alt": "图 1 为成本项分解示意。图 2 按 4096×4096 单矩阵计算 LoRA 参数；另列恰有 70 亿权重在 16/4 位下的理想原始存储（十进制 GB），未计元数据、激活或运行开销，非显卡容量承诺。",
      "data": {
        "kind": "analytical",
        "matrix": [
          4096,
          4096
        ],
        "full_parameters": 16777216,
        "ranks": [
          4,
          8,
          16
        ],
        "adapter_parameters": [
          32768,
          65536,
          131072
        ],
        "weights": 7000000000,
        "bits": [
          16,
          4
        ],
        "raw_GB": [
          14,
          3.5
        ],
        "excludes": "metadata, gradients, activations, optimizer, framework"
      },
      "ai_generated": true,
      "disclosure": "AI-assisted code rendering; original schematic or explicitly attributed data; not a new model experiment.",
      "url": "/guides/assets/efficient-adaptation-lora-2-996b40d39588.png"
    }
  ],
  "peer_reviewed": false
}