{ "schema": "auturnal.deep-research-review.v2", "slug": "preference-optimization-objectives", "title": "Preference Optimization Does Not Identify Human Utility", "category": "Reliability & Interpretation", "dek": "Pairwise feedback identifies a statistical preference model under assumptions; it does not by itself define a universal or uniquely recoverable utility.", "abstract": "Preference optimization changes a generative model by fitting comparative judgments and constraining departure from a reference policy. This article distinguishes the observed comparison process, a latent scalar reward model, and the policy selected by an optimization objective. It derives the two-response solution of KL-regularized reward maximization, analyzes reward-offset invariance, and constructs a three-group preference cycle that no single Bradley–Terry utility can reproduce exactly. Direct Preference Optimization is treated as a useful reparameterization within its assumptions rather than an identification of human values. Executable finite checks verify the analytic policy, offset invariance, and the cycle. The figures illustrate declared losses and policies, not human preference measurements or alignment benchmarks.", "english": "Preference optimization changes a generative model by fitting comparative judgments and constraining departure from a reference policy. This article distinguishes the observed comparison process, a latent scalar reward model, and the policy selected by an optimization objective. It derives the two-response solution of KL-regularized reward maximization, analyzes reward-offset invariance, and constructs a three-group preference cycle that no single Bradley–Terry utility can reproduce exactly. Direct Preference Optimization is treated as a useful reparameterization within its assumptions rather than an identification of human values. Executable finite checks verify the analytic policy, offset invariance, and the cycle. The figures illustrate declared losses and policies, not human preference measurements or alignment benchmarks.", "refs": [ "dl-19-1", "dl-19-2", "dl-19-3" ], "sections": [ [ "1. Separate judgments, rewards, and policies", "A preference dataset contains events: a person or evaluator sees a prompt and two candidate responses under a particular presentation, then supplies a judgment. A reward model assigns numerical scores intended to explain those events. A policy produces future responses. These three objects are connected by modeling choices, but they are not the same object. A policy can improve agreement with collected judgments without establishing that its outputs are true, safe, or optimal for every user.\n\nThe distinction matters because both comparison selection and annotation instructions affect the observations. Showing two concise summaries asks a different question from comparing one concise summary with a long speculative answer. A model trained on the resulting labels inherits this sampling frame. The relevant scientific question is therefore which preference process the model approximates and how the approximation generalizes. Treating a low training loss as discovery of an objective human utility bypasses that question rather than answering it.", [] ], [ "2. A paired-comparison model makes an explicit assumption", "The classical Bradley–Terry paper provides the historical paired-comparison reference. Its bibliographic identity is verified here, while its publisher full text was not accessible during this review; no detailed historical theorem is attributed to that unread text. The logistic comparison form is also explicitly stated in the accessed DPO paper.\n\nFor this article, assume each candidate has a real-valued score and the probability of choosing the first is the logistic function of the score difference. This is a statistical model of repeated judgments, not a definition of preference itself. It implies that the log odds of one choice over another equal a score difference. A common additive shift of all scores cancels. Multiplying scores changes choice probabilities unless the comparison noise scale changes as well. Thus the origin of the scale is unidentifiable from comparisons, and the numerical scale has meaning only relative to a chosen observation model.", [ "dl-19-1", "dl-19-3" ] ], [ "3. Collecting feedback is part of the learning system", "Christiano and colleagues demonstrate learning rewards from human comparisons of trajectory segments and using those learned rewards for reinforcement learning. This separates learning the objective from optimizing behavior. The demonstration does not make every future preference dataset representative or remove the need to evaluate learned behavior.\n\nConsider a comparison collection process that draws both candidates from a narrow reference policy. It may obtain highly consistent labels inside that region while leaving other outputs almost unconstrained. Optimizing a fitted reward can then change the output distribution toward poorly observed regions. Collecting fresh comparisons can reduce that gap, but introduces a feedback loop: the policy influences which examples are judged next. An auditable system records the candidate generator, evaluator population, instructions, tie handling, and sampling probabilities. Without those records, disagreement between two trained policies may reflect different annotation populations rather than a superior optimization algorithm.", [ "dl-19-2" ] ], [ "4. KL regularization selects a policy relative to a reference", "For a fixed prompt and a finite response set, maximize expected reward minus beta times the KL divergence from a reference distribution with positive support. A Lagrange multiplier for normalization yields a policy proportional to the reference probability multiplied by the exponential of reward divided by beta. The objective is strictly concave over the supported simplex for positive beta, so this solution is unique there.\n\nIn a two-response construction, let the reference assign probability one half to each response and let their reward difference be log three. At beta one, the optimal policy assigns probability three quarters to the preferred response. Increasing beta moves the policy toward the reference; decreasing it concentrates probability on the higher reward. Our policy figure shows this exact construction. It is not a fitted human reward scale. Changing the reference probabilities changes the selected policy even if the comparative score difference stays fixed, demonstrating that the optimization objective contains more than the observed ordering.", [] ], [ "5. Direct optimization changes the parameterization", "Rafailov and colleagues derive Direct Preference Optimization by expressing rewards through policy-to-reference log probability ratios. In a Bradley–Terry comparison, the prompt-dependent normalization term cancels. This gives a policy objective based on comparative likelihood without a separate learned reward network.\n\nThe cancellation is algebraic; it does not show that an arbitrary parameterized language model reaches the global optimum or that finite feedback satisfies the assumed comparison law. In the two-response case, the loss can be studied directly as a function of a log-ratio margin. A larger positive margin lowers the preferred-pair loss, with beta controlling its scale. The first figure plots this declared function at several beta values. At zero margin every curve equals log two. This common intersection is a useful implementation check, but identical pair loss can coexist with different generation behavior on prompts or responses absent from the comparison data.", [ "dl-19-3" ] ], [ "6. Reward origins disappear, but support still matters", "If a prompt-specific constant is added to every response reward, both its comparison probabilities and its KL-optimal policy remain unchanged. The exponential normalization absorbs the constant exactly. The executable check verifies this invariance with several finite reward vectors and offsets. Such nonuniqueness is harmless when the aim is selecting the same optimal policy; it becomes misleading if the reward's absolute magnitude is presented as a directly measured quantity.\n\nThe support condition is equally important. A reference probability of zero makes positive policy mass at that response incompatible with finite forward KL. Formal formulas involving a log policy-to-reference ratio need a domain where the denominator is positive. Language models often assign nonzero token probabilities mathematically, but truncation, candidate filtering, and finite precision can create effective exclusions. The deployed generation procedure therefore deserves its own support description. A training objective over sequence probabilities does not automatically characterize a filtered decoder.", [] ], [ "7. Aggregated preferences can be cyclic", "Construct three equally sized evaluator groups with deterministic rankings A above B above C, B above C above A, and C above A above B. Across the combined population, two thirds prefer A to B, two thirds prefer B to C, and two thirds prefer C to A. Each individual ranking is transitive; the aggregate pair probabilities form a cycle.\n\nA scalar logistic-score model cannot fit all three probabilities exactly. The sum of the three score differences around a cycle must be zero, while the three observed log odds sum to three times log two. This is a finite model-misspecification witness, not a claim that every real preference dataset contains such a cycle. A fitted scalar model must compromise, and a single reward number hides which group loses in that compromise. Reporting subgroup disagreement or allowing a richer conditional model can reveal information that a pooled win rate suppresses. Optimization precision cannot repair an incompatible statistical representation.", [] ], [ "8. Preference agreement is not factual verification", "A fluent but unsupported explanation can be preferred over an accurate but awkward one under some instructions. Conversely, an evaluator asked to check citations may favor a different response. These possibilities follow from the distinction between the observation process and the task property, not from an assumption that annotators are irrational. A preference label measures the judgment that was elicited.\n\nFor a scholarly assistant, factual support should therefore have its own observable checks: source existence, claim-to-source alignment, equation consistency, and correction behavior. These can be included in evaluation or annotation instructions, but their success should still be reported separately from an overall preference score. Likewise, harmlessness, brevity, and task completion need not be perfectly correlated. A composite reward encodes a tradeoff among them. The weights and evaluator population are governance and product choices, not quantities that a pairwise likelihood can uniquely infer from incomplete observations.", [] ], [ "9. Verify the finite mechanism before scaling the claim", "The companion code checks the two-response optimum against a dense grid of candidate policy probabilities, using exact rewards and the declared reference. It verifies that the analytic solution attains at least as much objective value as every grid point, up to numerical tolerance. It also checks normalization and additive-reward invariance for small finite distributions. This is a numerical consistency test of a concave finite objective, not evidence of convergence in a neural parameter space.\n\nA separate enumeration constructs all three evaluator rankings and counts their pairwise choices. The resulting cycle is compared with the necessary zero-sum condition for scalar log odds. The plotted DPO loss is checked at zero and at selected margins, and the policy curve at beta one is checked against three quarters. No human labels, generated completions, reward-model training, or policy fine-tuning were performed for this article. The proposed experimental protocol below requires those additional observations.", [] ], [ "10. Evaluate a process, not a scalar slogan", "A defensible preference-optimization study fixes the comparison process, reference model, support, regularization, and held-out evaluation population. It reports agreement with labels and behavior under actual decoding, including prompts outside the feedback collection distribution. It also measures the task properties that matter independently of preference. An overall win rate is informative only alongside its comparison set and the uncertainty and disagreements it aggregates.\n\nThe resulting interpretation is precise. Comparative data can identify some score differences under a chosen observation model. KL regularization selects a policy using both those differences and a reference distribution. A direct policy loss can simplify optimization without solving every problem of representation, support, or plural preferences. Those boundaries are not reasons to abandon preference learning; they specify what evidence a stronger claim would need. For a reliable system, preserving the provenance of judgments and the limits of the fitted model is as important as reducing its loss.", [] ] ], "section_kinds": [ "author_analysis_or_verification_proposal", "source_based_review", "source_based_review", "author_analysis_or_verification_proposal", "source_based_review", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal", "author_analysis_or_verification_proposal" ], "equations": [ { "tex": "P(a\\succ b\\mid x)=\\sigma\\!\\left(r(x,a)-r(x,b)\\right),\\qquad \\sigma(t)=(1+e^{-t})^{-1}", "explanation": "Assumed Bradley–Terry logistic observation model, also explicit in the accessed DPO paper. It represents pairwise probabilities; it is not a universal law of human judgment." }, { "tex": "\\pi^*(y\\mid x)=\\frac{\\pi_{\\rm ref}(y\\mid x)e^{r(x,y)/\\beta}}{\\sum_{y'}\\pi_{\\rm ref}(y'\\mid x)e^{r(x,y')/\\beta}},\\qquad\\beta>0", "explanation": "Finite-response optimum of expected reward minus beta times forward KL to a positive reference. With equal reference masses and reward gap log(3), the preferred probability is 3/4 at beta=1." }, { "tex": "\\mathcal L_{\\rm DPO}=-\\mathbb E_{\\mathcal D}\\log\\sigma\\!\\left(\\beta\\left[\\log\\frac{\\pi_\\theta(y_w\\mid x)}{\\pi_{\\rm ref}(y_w\\mid x)}-\\log\\frac{\\pi_\\theta(y_l\\mid x)}{\\pi_{\\rm ref}(y_l\\mid x)}\\right]\\right)", "explanation": "DPO objective of Rafailov et al.; requires supported comparisons. Its model equivalence does not certify neural optimization convergence or representative feedback." }, { "tex": "\\operatorname{logit}P(A\\succ B)+\\operatorname{logit}P(B\\succ C)+\\operatorname{logit}P(C\\succ A)=0", "explanation": "Necessary condition for one scalar logistic-score model at a fixed prompt. The constructed equal mixture of three cyclic rankings yields 3 log(2), violating it." } ], "table": { "headers": [ "Object", "What it records", "Additional assumption needed" ], "rows": [ [ "Pairwise judgment", "A choice under a presentation and instruction", "Representativeness of the comparison process" ], [ "Scalar reward", "A fitted explanation of choice probabilities", "Adequacy and identifiability of the observation model" ], [ "KL-optimal policy", "Tradeoff between reward and reference departure", "Support, reward scale, and optimization conditions" ], [ "Held-out win rate", "Agreement relative to a comparator and evaluator population", "Generalization to new users, tasks, and independent properties" ] ] }, "protocol": [ "Record candidate generation, evaluator population, instructions, ties, and judgment provenance.", "Separate feedback used for fitting, model selection, and final behavioral evaluation.", "Freeze reference policy, sequence probability convention, decoder, and regularization scale.", "Measure subgroup disagreement, repeated-label consistency, and cyclic preference patterns.", "Report factual or task-specific verification separately from preference agreement and overall win rates.", "Stress-test new prompt populations and unsupported outputs, publishing uncertainty and failure examples." ], "limitations": [ "The examples use finite response sets and assigned rewards rather than trained language models.", "The Bradley–Terry original publication was verified bibliographically; its full text was inaccessible during this review.", "The three-group cycle is a constructed incompatibility witness, not an empirical estimate of human disagreement.", "No policy fine-tuning or human annotation experiment was run; these are proposed evaluations." ], "evidence": [ { "source": "dl-19-1", "result": "Identifies the classical paired-comparison publication used as the historical model reference.", "boundary": "Publisher full text was inaccessible; no unread theorem is claimed, and the logistic model is independently specified here.", "locator": "Bibliographic record: Biometrika 39(3/4), 324–345." }, { "source": "dl-19-2", "result": "Demonstrates learning a reward from human trajectory comparisons and optimizing behavior with it.", "boundary": "The task-specific demonstrations do not identify universal preferences.", "locator": "Method and preference-learning loop." }, { "source": "dl-19-3", "result": "Reparameterizes a KL-regularized preference objective through policy-to-reference ratios.", "boundary": "Requires its comparison-model and support assumptions; finite neural fitting adds estimation and optimization error.", "locator": "Sections 3–5 and Appendix A." } ], "research_questions": [ "Which annotation metadata is sufficient to distinguish model error from evaluator-population disagreement?", "How should feedback collection allocate comparisons outside the current policy's familiar support?", "When are conditional or non-scalar preference models preferable to a pooled scalar reward?" ], "kind": "ai_assisted_research_guide", "author": "Codex (AI)", "original_language": "en", "created_date": "2026-10-22", "published_date": null, "publication_status": "local_draft_pending_owner_upload", "version": 1, "previous_version": null, "analytic": "preference-optimization-objectives", "peer_reviewed": false, "journal_publication": false, "disclosure": { "authorship": "Written and technically self-reviewed by Codex (AI). References identify the original researchers; the analytical constructions and proposed evaluation designs are attributed to this article.", "figures": "Covers are conceptual ImageGen illustrations. Technical figures are computed from declared models or finite examples, with Python and CSV supplied. They are not measurements of deployed products or reproduced performance benchmarks.", "review": "This local draft has no independent peer-review or journal-acceptance record. Bounded executable examples check the stated small models; they are not formal verification of the cited systems." }, "review_scope": "A focused analytical review of three selected foundational primary sources, with original worked examples. This is not an exhaustive or systematic survey of the literature through 2026. Source access and version notes are recorded with the bibliography.", "revision_note": "Initial local English manuscript; no server upload or publication action." }