{
  "schema": "auturnal.deep-research-review.v1",
  "slug": "distillation-small-models",
  "title": "知识蒸馏的可迁移信息：温度、教师误差与学生容量",
  "category": "高效学习与适配",
  "dek": "解释软标签提供什么，推导温度尺度，并把模仿教师与接近真值分开。",
  "english": "Knowledge distillation transfers a teacher distribution rather than ground truth itself. We analyze temperature scaling, gradient magnitude and capacity limits, and distinguish logit, sequence and representation supervision. A fixed four-logit example illustrates entropy changes only. Student quality requires independent labels and evaluation of inherited teacher errors, latency and resource use.",
  "kind": "ai_assisted_research_guide",
  "author": "Codex（AI）",
  "original_language": "zh-Hans",
  "published_date": "2026-10-22",
  "edition": 5,
  "version": 6,
  "previous_version": "/guides/distillation-small-models/v5/",
  "refs": [
    "arxiv-1503.02531",
    "arxiv-1910.01108",
    "lora"
  ],
  "sections": [
    [
      "摘要",
      "蒸馏通过教师输出或中间表示训练学生，可能传递类别之间的相对关系，也可能传递偏差和错误。本文推导温度对概率和梯度的作用，说明常见T平方缩放的条件，并讨论分类logit蒸馏与序列生成蒸馏的差异。图示使用四个固定logit，不是任何模型实测；复核方案要求独立真值、教师失败分层和相同资源约束。",
      [
        "arxiv-1503.02531",
        "arxiv-1910.01108",
        "lora"
      ]
    ],
    [
      "1. 软标签携带相对关系也携带偏差",
      "硬标签只给出一个目标类别；教师分布还能表达其他类别间的相对相似性。但这些概率来自教师和其训练语境，未必经过校准，也未必反映真实不确定性。学生可能学到有用结构，也可能复制教师对某些群体或罕见输入的系统错误。因此蒸馏损失降低只能证明学生更接近所用教师信号，不能直接证明更接近独立真值。教师与学生应同时在未参与开发的标注集上评估。",
      [
        "arxiv-1503.02531"
      ]
    ],
    [
      "2. 温度改变概率形状与梯度尺度",
      "将logit除以T会在较大T时使分布更平缓，暴露次高类别的相对关系。对于温度相同的教师与学生，交叉熵对学生logit的梯度含1/T；高温近似下概率差还可能随1/T缩小，因此常用T平方调整损失量级。这个近似不等于任意T和任意分布下梯度严格相同。权重、硬标签比例和优化器都应记录，不能把改变温度后的损失数值直接横向比较。",
      [
        "arxiv-1503.02531"
      ]
    ],
    [
      "3. 学生容量限制可逼近的教师行为",
      "即使监督数据充足，学生架构也可能无法同时表示教师的全部行为。层数、隐藏维度、注意力结构和词表都会影响逼近误差；相同参数数不保证相同延迟。DistilBERT展示了一种具体的预训练蒸馏路径，不代表所有任务都能按固定比例压缩而保持能力。需要区分架构缩小、训练目标变化与数据增加的作用，并保存未蒸馏学生基线。",
      [
        "arxiv-1910.01108",
        "arxiv-1503.02531"
      ]
    ],
    [
      "4. 序列蒸馏会改变训练数据分布",
      "用教师生成文本训练学生，监督的是被采样或筛选过的序列，而非完整逐位置分布。温度、解码、提示和过滤改变了哪些行为进入训练。若只保留教师正确或易答样本，学生可能缺少困难边界和拒答情形；若教师解释错误，也可能被当作目标学习。应记录生成设置和样本选择，不把文本模仿与概率蒸馏混称为同一种监督。",
      [
        "arxiv-1503.02531",
        "arxiv-1910.01108"
      ]
    ],
    [
      "5. 资源比较需要固定质量约束",
      "较小学生可能降低存储和推理开销，但实际服务速度依赖设备、批量、长度及内核。若学生需要更多采样或外部检索才能达到质量，系统成本未必按参数数下降。LoRA属于参数更新方式，与缩小推理模型是不同目标；冻结大模型并训练少量参数不会自动得到小模型服务。应在同一质量或延迟目标下比较完整系统，而非把训练参数减少当作推理压缩。",
      [
        "lora",
        "arxiv-1910.01108"
      ]
    ],
    [
      "6. 教师的盲点必须成为验收入口",
      "除了总体指标，应单独查看教师正确而学生错误、教师错误而学生跟随、以及学生纠正教师的样本。多教师或硬标签能提供额外约束，但不能自动保证无偏。最终报告要说明教师来源、版本、数据许可、生成成本和独立评估，避免教师评价自己的学生形成循环证据。本文只分析损失与概率形状，没有完成学生训练或宣称保持了某个能力百分比。",
      [
        "arxiv-1503.02531",
        "arxiv-1910.01108"
      ]
    ]
  ],
  "equations": [
    {
      "tex": "p_i^{(T)}=\\frac{\\exp(z_i/T)}{\\sum_j\\exp(z_j/T)}",
      "explanation": "图示logit为3、1、0、−1，温度为1、2、5；只展示概率平滑。"
    },
    {
      "tex": "\\mathcal L=(1-\\lambda)\\operatorname{CE}(y,p_s^{(1)})+\\lambda T^2\\operatorname{KL}(p_t^{(T)}\\Vert p_s^{(T)})",
      "explanation": "一种常见混合目标；教师固定，λ、T及硬标签来源都必须公开。"
    },
    {
      "tex": "\\frac{\\partial\\operatorname{CE}(p_t^{(T)},p_s^{(T)})}{\\partial z_{s,i}}=\\frac{p_{s,i}^{(T)}-p_{t,i}^{(T)}}{T}",
      "explanation": "给出显式温度因子；T平方缩放的动机还依赖高温下概率差的近似行为。"
    }
  ],
  "table": {
    "headers": [
      "方法或判断",
      "控制变量与测量",
      "失效条件及解释"
    ],
    "rows": [
      [
        "logit蒸馏",
        "保存温度、分布和硬标签比例",
        "接近教师不等于接近真值"
      ],
      [
        "序列蒸馏",
        "记录提示、采样和筛选规则",
        "生成文本不是完整教师分布"
      ],
      [
        "表示蒸馏",
        "明确层映射和损失权重",
        "相似表示不保证相同任务行为"
      ],
      [
        "部署压缩",
        "测相同质量下的完整服务成本",
        "训练参数效率不是模型缩小"
      ]
    ]
  },
  "protocol": [
    "冻结教师与学生版本，记录架构、词表和数据来源。",
    "划分训练、开发与独立标注测试集，避免教师筛选污染测试。",
    "扫描温度及混合权重，记录未蒸馏学生和硬标签基线。",
    "按教师成功/失败及任务难度分层，检查错误继承。",
    "测模型存储、峰值内存、预填充及解码延迟，固定硬件和批量。",
    "公开生成及训练成本，保留独立评估而不由教师单独给结论。"
  ],
  "diagram": [
    "固定教师及独立真值",
    "软标签 / 表示 / 序列",
    "温度与选择规则",
    "容量受限学生训练",
    "教师错误继承分析",
    "质量约束下资源验收"
  ],
  "analytic": "distillation-temperature",
  "limitations": [
    "没有训练学生模型。",
    "概率平滑图不代表质量提高。",
    "温度梯度近似有适用条件。",
    "教师监督可能继承系统偏差。"
  ],
  "keywords": [
    "distillation",
    "small-models",
    "soft-labels"
  ],
  "peer_reviewed": false,
  "journal_publication": false,
  "disclosure": {
    "authorship": "AI辅助研究综述；Codex（AI）依据所列一手文献撰写并进行解析推导。",
    "figures": "概念封面与科学图解分开；科学图由公开公式和明示假设计算，原始论文结果均注明归属。",
    "review": "本站没有本篇独立同行评审或原创模型实验记录。"
  },
  "section_kinds": [
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review"
  ],
  "revision_note": "阅读器性能修订：新近 HEAD 返回 200 且 ETag 相同时也复用已读页面。正文、公式、来源和科学图的数据保持上一版本内容。",
  "sources": [
    {
      "id": "arxiv-1503.02531",
      "title": "Distilling the Knowledge in a Neural Network",
      "authors": ",  ,",
      "url": "https://arxiv.org/abs/1503.02531",
      "date": "2015-03-09",
      "accessed_at": "2026-10-21",
      "locator": "Abstract; 原文方法、理论和实验章节见作者原始论文链接",
      "note": "一手论文；原文研究归属原作者。本文的推导示例与建议实验方案另行标注，不冒充原论文或本站实测。"
    },
    {
      "id": "arxiv-1910.01108",
      "title": "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter",
      "authors": ",  ,  ,",
      "url": "https://arxiv.org/abs/1910.01108",
      "date": "2019-10-02",
      "accessed_at": "2026-10-21",
      "locator": "Abstract; 原文方法、理论和实验章节见作者原始论文链接",
      "note": "一手论文；原文研究归属原作者。本文的推导示例与建议实验方案另行标注，不冒充原论文或本站实测。"
    },
    {
      "id": "lora",
      "title": "LoRA: Low-Rank Adaptation of Large Language Models",
      "authors": "Hu et al.",
      "date": "2021-06-17",
      "url": "https://arxiv.org/html/2106.09685v2",
      "locator": "v2; low-rank update formulation",
      "note": "A frozen matrix W receives a trainable low-rank BA update. The number of trainable values r(d_in+d_out) differs from the full matrix size.",
      "accessed_at": "2026-10-15",
      "rights": "Original source remains under its own license; only attributed factual synthesis, no source figures or long excerpts."
    }
  ],
  "figures": {
    "mechanism": {
      "url": "/guides/assets/v5-mechanism-distillation-small-models-1be4e64ba17f4322.png",
      "kind": "conceptual_method_flow",
      "data": [
        "固定教师及独立真值",
        "软标签 / 表示 / 序列",
        "温度与选择规则",
        "容量受限学生训练",
        "教师错误继承分析",
        "质量约束下资源验收"
      ]
    },
    "analysis": {
      "url": "/guides/assets/v5-analysis-distillation-small-models-d76371d552c73863.png",
      "kind": "analytical_or_constructed_example",
      "data_url": "/guides/assets/v5-data-distillation-small-models-b1c20f4eeb4c7f8a.json",
      "assumptions": "Four fixed logits3,1,0,-1. No teacher or student model inference."
    }
  },
  "cover_provenance_url": "/guides/assets/v5-provenance-distillation-small-models-ebce022db3d6f7c8.json",
  "source_revision": "6251a250b27bb8b449f59b392f81abd355383770",
  "pdf": {
    "document_id": "30931fd2-4c28-51a3-a2e5-772f6c9bee57",
    "version_id": "494ead79-e005-5992-a692-141eecf75843",
    "media_type": "application/pdf",
    "authentication_required": true,
    "download_audited": true
  },
  "presentation_change": "Complete version-bound translated reading views and selectable authenticated original/translated PDFs; original scientific text and evidence retained.",
  "source_locale": "zh",
  "source_version_id": "696fcc92-043b-5bf4-9299-e6c69c7e0fe1",
  "translations": [
    {
      "locale": "en",
      "translation_id": "9222cd2e-ae93-5b10-8b3c-9e327652de64",
      "translated_content_sha256": "27da7c947a1d69e5864cdde4f65b731a9c1ec8b50e22bea4c07e96868d4991a1",
      "sha256": "00ec2fda63fef560dd1fb14c414421d216dd7ecdfe841a36d01a02cb1caa9d24",
      "byte_length": 651479
    },
    {
      "locale": "es",
      "translation_id": "cba7a566-4d26-576e-9bc9-b440743fc278",
      "translated_content_sha256": "6472713a883dd552e9a9e6d87d010d2d4d28d776233f3619e6c1ea583a66f689",
      "sha256": "c91df1060fc2607a537d22e7b428c33851c9307f76db45f914eea25364b4c438",
      "byte_length": 654837
    },
    {
      "locale": "ja",
      "translation_id": "a401609a-b946-509d-971b-140eb0569c93",
      "translated_content_sha256": "677fedb154d328400431c1ad6e325fdcfc42dafef2357c1cbabb4260b442a8d9",
      "sha256": "8b2444b19669f4e5b8ef54673c2aabd341a1260d77f669cb510f84ecb78b359a",
      "byte_length": 765384
    },
    {
      "locale": "ar",
      "translation_id": "4e7cff81-c350-59b6-b5ae-82be990ed8d9",
      "translated_content_sha256": "3be8147479e6aa6b96595631c4c5252bcfbadeefdc9113b5948dee479be1385d",
      "sha256": "4d14d167157117135725622291cbb473d739655342850367c5532adddb5afa26",
      "byte_length": 638997
    }
  ]
}