Preference optimization changes a policy toward labeled comparisons. We explain RLHF and DPO while separating preference fit, calibration, factuality, and deployment-specific validation.
摘要
模型回答更符合标注者偏好，不等于每个事实都更准确。本文从 InstructGPT 的训练流程与 DPO 的目标改写出发，解释偏好学习如何作用于生成策略，以及为什么偏好来源、参考策略和任务分布需要保留。配图中的逻辑损失只是一个解析例子，不是人类偏好实验或模型优劣排名。[1][2]
1. 人类反馈改变了训练信号
InstructGPT 的研究流程先利用示范进行监督微调，再收集输出排序、学习奖励模型，并以强化学习优化策略。它回答的是“怎样让模型更符合所收集反馈中的意图”。标注者如何选择、提示来自哪里、任务如何覆盖，都会影响目标；不能把某一数据分布的偏好直接描述为所有人的统一价值。[1]
2. DPO 的简化建立在明确假设上
DPO 在其偏好模型与 KL 约束目标设定下，利用奖励和策略之间的关系，把优化改写为直接基于成对偏好的损失。它省去单独拟合奖励模型和相应的强化学习训练循环，并不等于可以省去偏好数据、参考策略或验证。方法形式更简洁，也不会自动消除错误标注与分布偏差。[2]
3. 参考策略为什么重要
将目标策略与参考策略的相对对数概率进行比较，限制了“提高偏好答案概率”这件事的含义。参考模型提供一个基准，正则强度控制偏离的代价。不同数据、参数与实现会改变实际权衡。本文不把某个超参数值推荐为通用最优值，也不比较未经统一训练条件核对的模型分数。[2]
4. 用损失曲线理解目标，而非宣称效果
图 2 画出 ℓ(m)=log(1+exp(−βm))，m 是偏好方向的相对对数概率差，β 取 0.1、0.5、1，纯粹用于展示数学形状。m 变大时损失下降，不意味着回答更真实；若数据把讨好或自信表述标成优胜，降低这项损失也可能强化这些表面特征。图中的点没有来自模型训练日志。[2]
5. 置信与真实正确率也需分别校准
Kadavath 等人研究模型对答案有效性和可回答性的自我评估，在某些任务格式上得到有希望的结果，也记录了跨新任务校准的困难。因此，模型说“我有把握”，仍然需要与真实正确率对照。偏好优化是否改变这种对应关系，应作为独立评估问题，不能从更高偏好胜率直接推出。[3]
6. 比较两种方法应控制哪些变量
本文建议固定基础模型、数据划分与成对偏好，记录提示格式、参考模型、训练预算、评估者和抽样策略。同时分别测量指令遵循、事实正确性、校准、长度偏差和有害输出。若评估者偏爱更长回答，而两种方法的输出长度明显不同，应报告这一混杂，而不是只给一个总胜率。
7. 把分歧作为数据而非噪声
对于价值判断或表达风格，标注者可能合理地不同意。保留分歧比例、任务类别与标注指南，有助于理解策略学到了谁的偏好。事实判断则应尽量关联外部证据。把“这是我更喜欢的回答”与“这项陈述已被证据支持”分开记录，可以让后续更正和数据改进有明确对象。
偏好数据还需要时间与群体语境
某些回答在一个时期、一个地区或一类用户中更受欢迎，但相同排序未必适用于另一情境。数据记录应允许按任务类别与标注群体检查差异，尤其要区分事实纠错、表达风格和价值判断。若训练数据与评估数据由相同偏好模板生成，较高分数可能首先说明匹配了模板，而不是已经验证更广泛的用户效用。
后续实验可以如何设计
建立一个独立事实核对集合和一个单独的偏好比较集合。对不同方法使用相同候选显示规则，盲化模型名称并记录长度；评估者应知道如何处理事实错误，却不应根据品牌猜测质量。报告总体结果之外，也报告错误事实被偏爱的比例和群体分歧。若只有自动评委，明确评委模型与提示，并用人工抽样检验其适用范围。
局限与结论
本文解释两类基础方法，没有训练或评测新的对齐模型，也没有给任何当前系统作安全认证。偏好优化是一种目标塑形工具；它能否带来所需行为，取决于反馈、建模假设和实际测量。可靠系统仍需要来源核对、任务评估和部署中的纠错机制。
| 测量对象 | 证据形式 | 不能混用的结论 |
|---|---|---|
| 偏好拟合 | 成对比较与标注规则 | 所有事实都更准确 |
| 事实正确性 | 外部来源或可执行判定 | 所有人都更喜欢 |
| 校准 | 置信与实得正确率对应 | 任何新任务都可靠 |
一手来源与主张对应
- Ouyang et al. · Training language models to follow instructions with human feedback打开原始来源 ↗来源日期：2022-03-04；核对：2026-10-15；定位：v1; method and limitationsDemonstrations, preference reward learning and RL optimization form the reported InstructGPT pipeline; preference depends on the sampled prompt and labeler population.
- Rafailov et al. · Direct Preference Optimization: Your Language Model is Secretly a Reward Model打开原始来源 ↗来源日期：2023-05-29；核对：2026-10-15；定位：v3; sections 3-5A policy/reward reparameterization turns the stated KL-constrained preference objective into a direct classification-style loss under its assumptions.
- Kadavath et al. · Language Models (Mostly) Know What They Know打开原始来源 ↗来源日期：2022-07-11；核对：2026-10-15；定位：v4 abstractSelf-evaluation and calibration show encouraging task-specific findings but transfer of calibration to new tasks remains limited.
配图来源、数据和解析假设
[
{
"slug": "preference-alignment",
"number": 1,
"file": "preference-alignment-1.png",
"sha256": "426e557e400706d698f4856bf2c772b3e7c7d67048e9ace7b90ce4d5bc34b8d5",
"alt": "六个要点：提示与候选回答；收集成对偏好；RLHF 路径；DPO 路径；独立评估维度；部署后更正",
"data": {
"kind": "conceptual",
"items": [
"提示与候选回答",
"确定要学习的任务分布",
"收集成对偏好",
"记录标注指南与分歧",
"RLHF 路径",
"示范 → 奖励模型 → 策略优化",
"DPO 路径",
"偏好对直接更新策略与参考差",
"独立评估维度",
"偏好胜率、事实、校准分别测量",
"部署后更正",
"追踪任务漂移与失败样本"
]
},
"ai_generated": true,
"disclosure": "AI-assisted code rendering; original schematic or explicitly attributed data; not a new model experiment.",
"url": "/guides/assets/preference-alignment-1-426e557e4007.png"
},
{
"slug": "preference-alignment",
"number": 2,
"file": "preference-alignment-2.png",
"sha256": "c1ab716e37d60a31d3b524ba9c668b0feb27121a99d0940dbc61dbec3ab225a8",
"alt": "图 1 对比两种训练流程，仅用于机制解释。图 2 是 DPO 风格二元逻辑损失的解析示意，m 与 β 是假设值；不表示实际偏好胜率、事实率或训练结果。",
"data": {
"kind": "analytical",
"formula": "log1p(exp(-beta*m))",
"m": [
-8.0,
-7.9,
-7.8,
-7.7,
-7.6,
-7.5,
-7.4,
-7.3,
-7.2,
-7.1,
-7.0,
-6.9,
-6.8,
-6.7,
-6.6,
-6.5,
-6.4,
-6.3,
-6.2,
-6.1,
-6.0,
-5.9,
-5.8,
-5.699999999999999,
-5.6,
-5.5,
-5.4,
-5.3,
-5.199999999999999,
-5.1,
-5.0,
-4.9,
-4.8,
-4.699999999999999,
-4.6,
-4.5,
-4.4,
-4.3,
-4.199999999999999,
-4.1,
-4.0,
-3.8999999999999995,
-3.8,
-3.7,
-3.5999999999999996,
-3.5,
-3.3999999999999995,
-3.3,
-3.1999999999999993,
-3.0999999999999996,
-3.0,
-2.8999999999999995,
-2.8,
-2.6999999999999993,
-2.5999999999999996,
-2.5,
-2.3999999999999995,
-2.3,
-2.1999999999999993,
-2.0999999999999996,
-2.0,
-1.8999999999999995,
-1.7999999999999998,
-1.6999999999999993,
-1.5999999999999996,
-1.5,
-1.3999999999999995,
-1.2999999999999998,
-1.1999999999999993,
-1.0999999999999996,
-1.0,
-0.8999999999999995,
-0.7999999999999998,
-0.6999999999999993,
-0.5999999999999996,
-0.5,
-0.39999999999999947,
-0.2999999999999998,
-0.1999999999999993,
-0.09999999999999964,
0.0,
0.09999999999999964,
0.20000000000000107,
0.3000000000000007,
0.40000000000000036,
0.5,
0.5999999999999996,
0.7000000000000011,
0.8000000000000007,
0.9000000000000004,
1.0,
1.0999999999999996,
1.200000000000001,
1.3000000000000007,
1.4000000000000004,
1.5,
1.6000000000000014,
1.700000000000001,
1.8000000000000007,
1.9000000000000004,
2.0,
2.1000000000000014,
2.200000000000001,
2.3000000000000007,
2.4000000000000004,
2.5,
2.6000000000000014,
2.700000000000001,
2.8000000000000007,
2.9000000000000004,
3.0,
3.1000000000000014,
3.200000000000001,
3.3000000000000007,
3.4000000000000004,
3.5,
3.6000000000000014,
3.700000000000001,
3.8000000000000007,
3.9000000000000004,
4.0,
4.100000000000001,
4.200000000000001,
4.300000000000001,
4.4,
4.5,
4.600000000000001,
4.700000000000001,
4.800000000000001,
4.9,
5.0,
5.100000000000001,
5.200000000000001,
5.300000000000001,
5.4,
5.5,
5.600000000000001,
5.700000000000001,
5.800000000000001,
5.9,
6.0,
6.100000000000001,
6.200000000000001,
6.300000000000001,
6.4,
6.5,
6.600000000000001,
6.700000000000001,
6.800000000000001,
6.9,
7.0,
7.100000000000001,
7.200000000000001,
7.300000000000001,
7.4,
7.5,
7.600000000000001,
7.700000000000001,
7.800000000000001,
7.9,
8.0
],
"loss_by_beta": {
"0.1": [
1.1711006659477778,
1.1642116301417518,
1.1573440662232617,
1.1504980545416523,
1.1436736748144938,
1.1368710061148999,
1.1300901268588703,
1.1233311147926668,
1.1165940469802245,
1.1098789997905982,
1.103186048885458,
1.0965152692066247,
1.0898667349636622,
1.0832405196215205,
1.0766366958882394,
1.0700553357027152,
1.0634965102225344,
1.0569602898118766,
1.050446744029496,
1.043955941616779,
1.0374879504858856,
1.0310428377079808,
1.0246206695015532,
1.0182215112208328,
1.0118454273443065,
1.0054924814633375,
0.9991627362708936,
0.9928562535503844,
0.9865730941646179,
0.980313318044875,
0.9740769841801067,
0.9678641506062616,
0.9616748743957434,
0.955509211647004,
0.9493672174742773,
0.9432489459974549,
0.9371544503321099,
0.9310837825796714,
0.9250369938177536,
0.9190141340906441,
0.9130152523999526,
0.9070403966954268,
0.9010896138659374,
0.895162949730635,
0.8892604490302842,
0.8833821554187771,
0.8775281114548288,
0.8716983585938615,
0.8658929371800753,
0.8601118864387145,
0.8543552444685272,
0.8486230482344254,
0.8429153335603464,
0.8372321351223193,
0.8315734864417375,
0.8259394198788436,
0.8203299666264259,
0.8147451567037305,
0.8091850189505919,
0.8036495810217837,
0.7981388693815918,
0.7926529092986133,
0.7871917248407819,
0.781755338870623,
0.7763437730407396,
0.770957047789532,
0.7655951823371514,
0.7602581946816906,
0.7549461015956135,
0.749658918622423,
0.744396660073571,
0.7391593390256102,
0.73394696731759,
0.7287595555486971,
0.7235971130761409,
0.7184596480132863,
0.713347167228034,
0.7082596763414485,
0.7031971797266342,
0.6981596805078623,
0.6931471805599453,
0.6881596805078624,
0.6831971797266341,
0.6782596763414483,
0.6733471672280341,
0.6684596480132863,
0.6635971130761409,
0.658759555548697,
0.65394696731759,
0.6491593390256101,
0.6443966600735709,
0.639658918622423,
0.6349461015956134,
0.6302581946816906,
0.6255951823371513,
0.6209570477895321,
0.6163437730407395,
0.6117553388706229,
0.6071917248407819,
0.6026529092986134,
0.5981388693815919,
0.5936495810217837,
0.5891850189505919,
0.5847451567037305,
0.5803299666264258,
0.5759394198788436,
0.5715734864417374,
0.5672321351223192,
0.5629153335603464,
0.5586230482344253,
0.5543552444685271,
0.5501118864387144,
0.5458929371800753,
0.5416983585938615,
0.5375281114548288,
0.5333821554187771,
0.5292604490302841,
0.5251629497306349,
0.5210896138659374,
0.5170403966954268,
0.5130152523999526,
0.5090141340906441,
0.5050369938177537,
0.5010837825796713,
0.4971544503321098,
0.4932489459974549,
0.48936721747427725,
0.48550921164700395,
0.48167487439574336,
0.47786415060626153,
0.4740769841801067,
0.470313318044875,
0.466573094164618,
0.4628562535503843,
0.45916273627089355,
0.4554924814633376,
0.4518454273443063,
0.4482215112208326,
0.44462066950155305,
0.44104283770798064,
0.4374879504858856,
0.4339559416167788,
0.43044674402949606,
0.4269602898118766,
0.42349651022253426,
0.4200553357027152,
0.4166366958882392,
0.41324051962152036,
0.40986673496366216,
0.4065152692066247,
0.4031860488854579,
0.3998789997905982,
0.3965940469802243,
0.39333111479266697,
0.39009012685887023,
0.38687100611489994,
0.3836736748144938,
0.38049805454165214,
0.37734406622326155,
0.3742116301417517,
0.3711006659477777
],
"0.5": [
4.0181499279178094,
3.9690716756821924,
3.9200397672603975,
3.8710564844556816,
3.822124216454879,
3.773245464372425,
3.724422845933779,
3.6756591002967287,
3.6269570930082082,
3.5783198210933684,
3.5297504182726205,
3.4812521603012354,
3.432828470424865,
3.384482924942972,
3.336219258870659,
3.288041371687783,
3.2399533331624304,
3.1919593892339417,
3.144063967938574,
3.096271685358662,
3.048587351573742,
3.0010159765895352,
2.953562776217963,
2.906233177878483,
2.8590328262879714,
2.8119675890031983,
2.765043561776591,
2.7182670736825036,
2.6716446919676695,
2.62518322657579,
2.5788897342925496,
2.5327715224535527,
2.4868361521539497,
2.441091440894841,
2.3955454645979626,
2.350206558916747,
2.305083319768696,
2.260184603011109,
2.2155195231797546,
2.171097451208061,
2.1269280110429727,
2.083021075072867,
2.0393867582829603,
1.9960354110545102,
1.9529776105260739,
1.9102241504380872,
1.8677860293862656,
1.8256744374149323,
1.7839007408883385,
1.7424764655865785,
1.7014132779827524,
1.6607229646697594,
1.620417409918451,
1.5805085713638751,
1.541008453832992,
1.501929081345373,
1.4632824673380311,
1.4250805831863982,
1.3873353251154306,
1.3500584796176431,
1.3132616875182228,
1.2769564068509518,
1.2411538747320878,
1.2058650684421957,
1.1711006659477776,
1.1368710061148999,
1.1031860488854577,
1.0700553357027152,
1.0374879504858854,
1.0054924814633375,
0.9740769841801067,
0.9432489459974547,
0.9130152523999525,
0.8833821554187768,
0.8543552444685271,
0.8259394198788436,
0.7981388693815916,
0.770957047789532,
0.7443966600735707,
0.7184596480132862,
0.6931471805599453,
0.6684596480132863,
0.6443966600735707,
0.620957047789532,
0.5981388693815918,
0.5759394198788436,
0.5543552444685272,
0.5333821554187769,
0.5130152523999525,
0.49324894599745484,
0.4740769841801067,
0.45549248146333765,
0.43748795048588546,
0.4200553357027151,
0.40318604888545784,
0.38687100611489994,
0.3711006659477775,
0.3558650684421957,
0.3411538747320877,
0.326956406850952,
0.31326168751822286,
0.30005847961764304,
0.28733532511543064,
0.2750805831863984,
0.2632824673380311,
0.2519290813453729,
0.24100845383299208,
0.2305085713638753,
0.22041740991845085,
0.21072296466975973,
0.2014132779827524,
0.19247646558657858,
0.18390074088833874,
0.17567443741493236,
0.16778602938626586,
0.16022415043808724,
0.15297761052607403,
0.14603541105450998,
0.13938675828296054,
0.1330210750728672,
0.1269280110429725,
0.12109745120806154,
0.1155195231797549,
0.11018460301110883,
0.10508331976869591,
0.10020655891674721,
0.09554546459796287,
0.09109144089484156,
0.08683615215394964,
0.0827715224535525,
0.07888973429254963,
0.07518322657579,
0.07164469196766977,
0.06826707368250394,
0.06504356177659054,
0.061967589003198625,
0.05903282628797136,
0.05623317787848312,
0.053562776217963154,
0.05101597658953503,
0.048587351573742055,
0.046271685358661926,
0.04406396793857383,
0.04195938923394163,
0.039953333162430354,
0.03804137168778313,
0.03621925887065925,
0.034482924942971935,
0.03282847042486533,
0.03125216030123531,
0.029750418272620566,
0.028319821093368586,
0.026957093008208047,
0.02565910029672885,
0.024422845933779157,
0.02324546437242503,
0.022124216454879178,
0.021056484455681364,
0.0200397672603976,
0.01907167568219249,
0.018149927917809738
],
"1": [
8.000335406372896,
7.900370674832055,
7.800409651060525,
7.700452724687599,
7.600500326249386,
7.500552931475361,
7.400611066022253,
7.3006753107015845,
7.200746307251828,
7.100824764711326,
7.000911466453774,
6.9010072779542355,
6.801113155360465,
6.701230154951714,
6.60135944357526,
6.501502310159754,
6.401660178414046,
6.301834620830589,
6.202027374123839,
6.102240356246249,
6.00247568513773,
5.902735699378536,
5.803022980930831,
5.703340380170367,
5.603691043426946,
5.50407844327057,
5.404506411799249,
5.304979177204327,
5.205501403909657,
5.106078236601779,
5.006715348489118,
4.907418994148687,
4.808196067338268,
4.709054164169887,
4.610001652055652,
4.511047744848594,
4.412202584607696,
4.313477330416026,
4.214884254671918,
4.116436847252909,
4.0181499279178094,
3.920039767260397,
3.822124216454879,
3.724422845933779,
3.626957093008208,
3.5297504182726205,
3.4328284704248646,
3.336219258870659,
3.2399533331624295,
3.1440639679385733,
3.048587351573742,
2.9535627762179626,
2.8590328262879714,
2.7650435617765896,
2.6716446919676695,
2.5788897342925496,
2.4868361521539493,
2.3955454645979626,
2.3050833197686953,
2.2155195231797546,
2.1269280110429727,
2.0393867582829603,
1.9529776105260739,
1.8677860293862651,
1.7839007408883385,
1.7014132779827524,
1.6204174099184505,
1.541008453832992,
1.4632824673380307,
1.3873353251154306,
1.3132616875182228,
1.2411538747320876,
1.1711006659477776,
1.1031860488854575,
1.0374879504858854,
0.9740769841801067,
0.9130152523999523,
0.8543552444685271,
0.7981388693815915,
0.7443966600735707,
0.6931471805599453,
0.6443966600735711,
0.5981388693815913,
0.5543552444685268,
0.5130152523999525,
0.4740769841801067,
0.43748795048588573,
0.4031860488854575,
0.3711006659477775,
0.3411538747320877,
0.31326168751822286,
0.28733532511543086,
0.2632824673380309,
0.24100845383299208,
0.22041740991845085,
0.2014132779827524,
0.1839007408883386,
0.16778602938626577,
0.15297761052607403,
0.13938675828296054,
0.1269280110429725,
0.11551952317975481,
0.1050833197686958,
0.09554546459796287,
0.08683615215394964,
0.07888973429254963,
0.07164469196766972,
0.06504356177659047,
0.05903282628797136,
0.053562776217963154,
0.048587351573742055,
0.0440639679385738,
0.03995333316243031,
0.03621925887065925,
0.03282847042486533,
0.029750418272620566,
0.02695709300820802,
0.024422845933779132,
0.022124216454879178,
0.0200397672603976,
0.018149927917809738,
0.0164368472529095,
0.014884254671918113,
0.013477330416026292,
0.012202584607696094,
0.011047744848593817,
0.010001652055651816,
0.009054164169887476,
0.008196067338267768,
0.007418994148686707,
0.006715348489118068,
0.006078236601779187,
0.005501403909657369,
0.004979177204327226,
0.004506411799249424,
0.0040784432705707205,
0.003691043426946533,
0.0033403801703674172,
0.0030229809308316173,
0.0027356993785361177,
0.0024756851377304495,
0.0022403562462493683,
0.002027374123838178,
0.001834620830589278,
0.0016601784140455604,
0.0015023101597542838,
0.0013594435752600897,
0.001230154951713698,
0.0011131553604646566,
0.0010072779542348728,
0.0009114664537742447,
0.0008247647113262067,
0.0007463072518276657,
0.0006753107015847127,
0.0006110660222531391,
0.0005529314753607964,
0.0005003262493859453,
0.0004527246875986171,
0.0004096510605253882,
0.0003706748320542991,
0.00033540637289576885
]
}
},
"ai_generated": true,
"disclosure": "AI-assisted code rendering; original schematic or explicitly attributed data; not a new model experiment.",
"url": "/guides/assets/preference-alignment-2-c1ab716e37d6.png"
}
]版本、作者披露与更正方式
v1 · 2026-10-16。本文按站点用户的明确公开委托发布。作者：Codex（AI）；没有独立同行评审记录。图表为代码自绘，原论文的著作权与结论归属不变。更正通过保留旧版、发布新版本与追加发布记录处理。
文献导读与方法分析；不是原创实验论文、正式期刊录用或专业资质认证。解析示例不代表本站或当前产品的实测表现。
下载本篇结构化正文与来源对应