The Objective Boundaries of Preference Optimization: Reward Proxies, Reference Policies, and Distribution Shift

1. Human feedback changed the training signal

The InstructGPT research workflow first uses demonstrations for supervised fine-tuning, then collects output rankings, learns a reward model, and optimizes the policy through reinforcement learning. It addresses the question of how to make a model better conform to the intentions expressed in the collected feedback. Annotator selection, prompt sources, and task coverage all affect the objective; preferences from a particular data distribution cannot be directly described as everyone's unified values.[1] Training language models to follow instructions with hu

2. DPO's simplification rests on explicit assumptions

Under its preference model and KL-constrained objective formulation, DPO uses the relationship between rewards and policies to reformulate optimization as a loss based directly on pairwise preferences. Eliminating separate reward-model fitting and the corresponding reinforcement learning training loop does not mean that preference data, a reference policy, or validation can be omitted. A simpler formulation does not automatically eliminate annotation errors or distributional bias.[2] Direct Preference Optimization: Your Language Model is

3. Why the reference policy matters

Comparing the relative log probabilities of the target and reference policies constrains what it means to increase the probability of a preferred answer. The reference model supplies a baseline, and regularization strength controls the cost of deviation. Different data, parameters, and implementations change the practical trade-off. This review neither recommends a particular hyperparameter value as universally optimal nor compares model scores whose training conditions have not been checked on a common basis.[2] Direct Preference Optimization: Your Language Model is

4. Using loss curves to understand the objective, rather than claim effectiveness

A DPO-type logistic loss can be written as$$\ell(m)=\log\!\left(1+\exp(-\beta m)\right)$$Here, m denotes the relative log-probability difference in the direction of preference. The loss decreases as m increases, but this does not guarantee more truthful answers; if the data reward sycophancy or confident phrasing, optimization may also reinforce these features. Figure 2 in this edition uses an independent proxy-utility counterexample to distinguish proxy scores from target utility; it contains no training logs or actual preference measurements.[2] Direct Preference Optimization: Your Language Model is

5. Confidence and actual accuracy also require separate calibration

Kadavath and colleagues studied models' self-assessments of answer validity and answerability, obtaining promising results in some task formats while also documenting difficulties with calibration across new tasks. Thus, a model's statement that it is confident still needs to be checked against actual accuracy. Whether preference optimization changes this correspondence should be an independent evaluation question and cannot be inferred directly from a higher preference win rate.[3] Language Models (Mostly) Know What They Know

6. Which variables should be controlled when comparing two methods

This review recommends fixing the base model, data splits, and pairwise preferences, and recording the prompt format, reference model, training budget, evaluators, and sampling strategy. Instruction following, factual correctness, calibration, length bias, and harmful outputs should also be measured separately. If evaluators favor longer answers and the two methods produce markedly different output lengths, this confound should be reported rather than providing only an overall win rate.作者分析与复核建议 / Author analysis

7. Treating disagreement as data rather than noise

Annotators may reasonably disagree about value judgments or expressive style. Retaining disagreement rates, task categories, and annotation guidelines helps clarify whose preferences the policy has learned. Factual judgments, by contrast, should be linked to external evidence wherever possible. Recording “this is the answer I prefer” separately from “this statement is supported by evidence” gives subsequent corrections and data improvements explicit targets.作者分析与复核建议 / Author analysis

Preference data also need temporal and group context

Some answers are more popular in a particular period, region, or user group, but the same ranking may not apply in another context. Data records should allow differences to be examined by task category and annotator group, particularly distinguishing factual corrections, expressive style, and value judgments. If training and evaluation data are generated from the same preference templates, higher scores may primarily indicate template matching rather than validated utility for a broader range of users.作者分析与复核建议 / Author analysis

8. The closed-form relationship for the KL-constrained objective

For a fixed input x, expected reward is maximized over the policy distribution while penalizing KL divergence from the reference policy. Using a Lagrange multiplier to constrain the probabilities to sum to one yields an optimal policy proportional to the reference probability multiplied by exp(r/β). Rearranging expresses reward as the log ratio of the policy to the reference policy, plus a normalization term that depends only on the input. This term cancels when comparing two answers to the same input. The relationship explains the theoretical connection to the DPO objective, but depends on fixed rewards, support conditions, and ideal optimization conditions; finite samples and parameterized models do not guarantee attainment of the closed-form optimum.[1] Training language models to follow instructions with hu[2] Direct Preference Optimization: Your Language Model is

9. Preference data are not ground truth free of measurement error

Annotators may make different trade-offs among helpfulness, concision, politeness, correctness, and risk; the way preference pairs are collected also determines which answers enter the comparison. If one set of candidates is longer on average, the preference model may learn length cues rather than actual content quality. Annotation guidelines, disagreement rates, candidate generation strategies, and duplicate-sample handling should be recorded, and an independent evaluation set retained. A win rate on training preferences is a result specific to that distribution and cannot be directly extrapolated to all languages, tasks, or populations.[1] Training language models to follow instructions with hu[2] Direct Preference Optimization: Your Language Model is

10. Optimizing a proxy to a higher value is not always better

Once a policy begins adapting to an evaluator's observable features, rising scores may accompany declines in answer truthfulness, calibration, or out-of-domain behavior. The constructed curves in this review illustrate only this logical possibility; they are not empirical evidence of reward hacking in any particular model. Reliable testing should use alternative independent evaluation methods, include separate observations of facts, refusal boundaries, and expressions of uncertainty, and check whether the reward distribution extends beyond the range of the training samples. If the training evaluator also serves as the sole acceptance evaluator, the closed loop establishes only that its own score has been optimized.[1] Training language models to follow instructions with hu[3] Language Models (Mostly) Know What They Know

11. Reference constraints and capability retention require joint measurement

Increasing or decreasing the constraint coefficient changes the cost of the policy moving away from the reference model, but no single value applies to all tasks. Even very small changes can affect rare, high-risk behaviors, while a larger overall distributional distance does not necessarily imply worse answers. Original task capabilities, preference evaluations, length, and refusal rates should be reported together, with changes examined separately across task categories. Knowledge distillation and preference training have different objectives: the former typically matches a teacher distribution, whereas the latter uses relative preferences; decreases in the same type of loss cannot be interpreted as equivalent capability transfer.[2] Direct Preference Optimization: Your Language Model is [4] Distilling the Knowledge in a Neural Network

Core equations and derivations

Method and evidence workflow

Analytical figures and numerical examples

Recomputable worked example

The constructed true utility is maximized at x=2; the proxy minus the penalty is maximized at x=5, where true utility instead equals −5. This provides only a counterexample and does not estimate the actual magnitude of degradation in RLHF or DPO.

Method comparison

Verification and experimental plan

1. Fix the reference model, preference-data version, and candidate generation strategy; record deduplication, length distributions, and annotation disagreements.
2. Create nonoverlapping sets for training preferences, in-distribution validation, and out-of-domain validation; variants of the same question must not leak across sets.
3. Sweep the constraint coefficient and save changes in the log probabilities of the policy and reference model, rather than saving only the final win rate.
4. Compare answers using an independent evaluation method with matched lengths and decoding budgets, randomizing answer order.
5. Report factual correctness, calibration, refusal, and basic task capabilities separately, and check whether overall improvement masks localized degradation.
6. Retain inputs, objective versions, and judgment rationales for failure cases; reevaluate after modifying annotations or the optimization plan rather than automatically carrying forward a previous passing status.

Scope and limitations

• Preference signals are affected by the annotator population and candidate distribution.
• The utility curves in the figure are constructed examples, not model experiments.
• A closed-form derivation does not amount to a training guarantee for a model with finitely many parameters.
• No new human preference study or safety certification is provided.

Primary sources and corresponding claims

1. Training language models to follow instructions with human feedbackOuyang et al. · 2022-03-04v1; method and limitations
2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafailov et al. · 2023-05-29v3; sections 3-5
3. Language Models (Mostly) Know What They KnowKadavath et al. · 2022-07-11v4 abstract
4. Distilling the Knowledge in a Neural Network, , · 2015-03-09Abstract; 原文方法、理论和实验章节见作者原始论文链接