Auturnal
English
← All research guides

Efficient Learning and Adaptation

Transferable Information in Knowledge Distillation: Temperature, Teacher Errors, and Student Capacity

Explaining what soft labels provide, deriving temperature scaling, and separating teacher imitation from approaching ground truth.

Codex (AI)2026-10-22 · v63 primary references · 3 equations · 2 analytical figures
Cite article

AI translation may be inaccurate. Please verify against the original.

AI-assisted translation · AI translation may be inaccurate. Please verify against the original. Read the original
Mathematical notation, code, bibliographic identities and text inside figures retain their original form.

Transferable Information in Knowledge Distillation: Temperature, Teacher Errors, and Student Capacity — conceptual cover

Cover · Thematic conceptual illustration generated by ImageGen; not an experimental result.

Abstract

Distillation trains students using teacher outputs or intermediate representations, potentially transferring relative relationships between classes as well as biases and errors. This article derives the effects of temperature on probabilities and gradients, explains the conditions for the common T-squared scaling, and discusses differences between classification logit distillation and sequence-generation distillation. The illustration uses four fixed logits, not measurements from any model; the verification plan requires independent ground truth, stratification of teacher failures, and identical resource constraints.[1] Distilling the Knowledge in a Neural Network[2] DistilBERT, a distilled version of BERT: smaller, faste[3] LoRA: Low-Rank Adaptation of Large Language Models

English Abstract

Knowledge distillation transfers a teacher distribution rather than ground truth itself. We analyze temperature scaling, gradient magnitude and capacity limits, and distinguish logit, sequence and representation supervision. A fixed four-logit example illustrates entropy changes only. Student quality requires independent labels and evaluation of inherited teacher errors, latency and resource use.

1. Soft Labels Carry Both Relative Relationships and Biases

Hard labels specify only a single target class; the teacher distribution can also express relative similarities among other classes. However, these probabilities come from the teacher and its training context; they may not be calibrated and may not reflect true uncertainty. Students may learn useful structure, but they may also reproduce the teacher's systematic errors for certain groups or rare inputs. Thus, a decrease in distillation loss demonstrates only that the student is closer to the teacher signal used, not directly that it is closer to independent ground truth. Both teacher and student should be evaluated on a labeled dataset that was not used during development.[1] Distilling the Knowledge in a Neural Network

2. Temperature Changes Probability Shape and Gradient Scale

Dividing logits by T flattens the distribution at larger T, exposing relative relationships among lower-ranked classes. For teachers and students at the same temperature, the cross-entropy gradient with respect to student logits contains 1/T; under a high-temperature approximation, probability differences may also shrink in proportion to 1/T, so T-squared is commonly used to adjust the loss scale. This approximation does not imply strictly identical gradients for arbitrary T and arbitrary distributions. Weights, the hard-label proportion, and the optimizer should all be recorded; loss values after changing temperature cannot be compared directly.[1] Distilling the Knowledge in a Neural Network

3. Student Capacity Limits Which Teacher Behaviors Can Be Approximated

Even with sufficient supervision data, the student architecture may be unable to represent all teacher behaviors simultaneously. The number of layers, hidden dimensions, attention structure, and vocabulary all affect approximation error; identical parameter counts do not guarantee identical latency. DistilBERT demonstrates a specific pretraining distillation approach, not that all tasks permit compression at a fixed ratio while retaining capabilities. The effects of architecture reduction, changes in training objectives, and additional data must be distinguished, and an undistilled student baseline must be retained.[2] DistilBERT, a distilled version of BERT: smaller, faste[1] Distilling the Knowledge in a Neural Network

4. Sequence Distillation Changes the Training Data Distribution

Training students on teacher-generated text provides supervision through sampled or filtered sequences, rather than the complete distribution at each position. Temperature, decoding, prompts, and filtering change which behaviors enter training. If only samples that the teacher answers correctly or that are easy to answer are retained, students may lack difficult boundary cases and refusal scenarios; incorrect teacher explanations may also be learned as targets. Generation settings and sample selection should be recorded, and text imitation and probability distillation should not be conflated as the same form of supervision.[1] Distilling the Knowledge in a Neural Network[2] DistilBERT, a distilled version of BERT: smaller, faste

5. Resource Comparisons Require Fixed Quality Constraints

Smaller students may reduce storage and inference overhead, but actual serving speed depends on the device, batch size, sequence length, and kernels. If students require more sampling or external retrieval to reach the required quality, system cost may not decrease in proportion to parameter count. LoRA is a parameter-update method, which serves a different objective from reducing the inference model's size; freezing a large model and training a small number of parameters does not automatically produce a small-model service. Complete systems should be compared under the same quality or latency target, rather than treating fewer training parameters as inference compression.[3] LoRA: Low-Rank Adaptation of Large Language Models[2] DistilBERT, a distilled version of BERT: smaller, faste

6. Teacher Blind Spots Must Be a Starting Point for Acceptance Testing

Beyond aggregate metrics, samples where the teacher is correct but the student is wrong, where the teacher is wrong and the student follows, and where the student corrects the teacher should be examined separately. Multiple teachers or hard labels can provide additional constraints, but do not automatically guarantee freedom from bias. The final report must specify the teacher's source and version, data licensing, generation costs, and independent evaluation, avoiding circular evidence from teachers evaluating their own students. This article analyzes only loss and probability shape; it has not completed student training or claimed to retain any particular percentage of capability.[1] Distilling the Knowledge in a Neural Network[2] DistilBERT, a distilled version of BERT: smaller, faste

Core Equations and Derivations

pi(T)=exp⁡(zi/T)∑jexp⁡(zj/T)p_i^{(T)}=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)}

(1) The illustrated logits are 3, 1, 0, and −1, with temperatures of 1, 2, and 5; only probability smoothing is shown.

LaTeXp_i^{(T)}=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)}
L=(1−λ)CE⁡(y,ps(1))+λT2KL⁡(pt(T)∥ps(T))\mathcal L=(1-\lambda)\operatorname{CE}(y,p_s^{(1)})+\lambda T^2\operatorname{KL}(p_t^{(T)}\Vert p_s^{(T)})

(2) A common mixed objective; the teacher is fixed, and λ, T, and the source of hard labels must all be disclosed.

LaTeX\mathcal L=(1-\lambda)\operatorname{CE}(y,p_s^{(1)})+\lambda T^2\operatorname{KL}(p_t^{(T)}\Vert p_s^{(T)})
∂CE⁡(pt(T),ps(T))∂zs,i=ps,i(T)−pt,i(T)T\frac{\partial\operatorname{CE}(p_t^{(T)},p_s^{(T)})}{\partial z_{s,i}}=\frac{p_{s,i}^{(T)}-p_{t,i}^{(T)}}{T}

(3) Shows the explicit temperature factor; the rationale for T-squared scaling also depends on the approximate behavior of probability differences at high temperatures.

LaTeX\frac{\partial\operatorname{CE}(p_t^{(T)},p_s^{(T)})}{\partial z_{s,i}}=\frac{p_{s,i}^{(T)}-p_{t,i}^{(T)}}{T}

Methods and Evidence Workflow

Fix the teacher and independent ground truth → Soft labels / representations / sequences → Temperature and selection rules → Training a capacity-limited student → Analysis of inherited teacher errors → Resource acceptance testing under quality constraints
Figure 1 · Methods and Evidence WorkflowA conceptual diagram drawn from the structure of this article, showing steps that must be distinguished; it does not indicate that these experiments have been completed.

Analytical Figures and Numerical Examples

Temperature reveals relative logit structure
Figure 2 · Temperature reveals relative logit structureFour fixed logits 3,1,0, -1. No teacher or student model inference. JSON · Equations, Inputs, and Numerical Values

Recomputable Worked Example

With fixed logits, the highest class probability is approximately 0.831 at T=1 and approximately 0.375 at T=5. A flatter distribution does not mean that the student learns better; quality against ground-truth labels must be measured separately.

Method Comparison

Method or JudgmentControlled Variables and MeasurementsFailure Conditions and Interpretation
Logit distillationRetain the temperature, distributions, and hard-label proportionBeing closer to the teacher does not mean being closer to ground truth
Sequence distillationRecord prompts, sampling, and filtering rulesGenerated text is not the complete teacher distribution
Representation distillationSpecify layer mappings and loss weightsSimilar representations do not guarantee identical task behavior
Deployment compressionMeasure the full serving cost at the same qualityTraining parameter efficiency is not model size reduction

Verification and Experimental Plan

  1. Freeze teacher and student versions, and record architectures, vocabularies, and data sources.
  2. Separate training, development, and independently labeled test sets, avoiding test contamination through teacher-based selection.
  3. Sweep temperatures and mixing weights, and record undistilled student and hard-label baselines.
  4. Stratify by teacher success or failure and task difficulty to examine error inheritance.
  5. Measure model storage, peak memory, and prefill and decoding latency with hardware and batch size fixed.
  6. Disclose generation and training costs, and retain independent evaluation rather than leaving conclusions solely to the teacher.

Scope and Limitations

  • No student model was trained.
  • The probability-smoothing figure does not demonstrate improved quality.
  • The temperature-gradient approximation has conditions of applicability.
  • Teacher supervision may pass on systematic biases.

Primary Sources and Corresponding Claims

  1. Distilling the Knowledge in a Neural Network, , · 2015-03-09Abstract; 原文方法、理论和实验章节见作者原始论文链接
  2. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, , , · 2019-10-02Abstract; 原文方法、理论和实验章节见作者原始论文链接
  3. LoRA: Low-Rank Adaptation of Large Language ModelsHu et al. · 2021-06-17v2; low-rank update formulation

Citation and Manuscript Export

BibTeXRISCSL JSONTXTArticle JSON

Author, Version, and Sources

AI-assisted research review; Codex (AI) wrote the article and performed analytical derivations based on the listed primary literature. This site has no record of independent peer review or original model experiments for this article.

Reader performance revision: previously read pages are also reused when a recent HEAD request returns 200 and the ETag is unchanged. The body text, equations, sources, and scientific figure data retain the content of the previous version.

The conceptual cover is separate from the scientific figures; scientific figures are calculated from publicly available equations and explicit assumptions, and all results from original papers are attributed.

Author, Versions, and Correction History · /guides/distillation-small-models/v1/

Cover and Scientific Figure Sources · Data, Sources, and Assumptions · Publication and Version History

Source revision: 6251a250b27bb8b449f59b392f81abd355383770