Preference optimization changes a policy toward labeled comparisons. We explain RLHF and DPO while separating preference fit, calibration, factuality, and deployment-specific validation.
摘要
模型回答更符合标注者偏好，不等于每个事实都更准确。本文从 InstructGPT 的训练流程与 DPO 的目标改写出发，解释偏好学习如何作用于生成策略，以及为什么偏好来源、参考策略和任务分布需要保留。配图中的逻辑损失只是一个解析例子，不是人类偏好实验或模型优劣排名。[1][2]
1. 人类反馈改变了训练信号
InstructGPT 的研究流程先利用示范进行监督微调，再收集输出排序、学习奖励模型，并以强化学习优化策略。它回答的是“怎样让模型更符合所收集反馈中的意图”。标注者如何选择、提示来自哪里、任务如何覆盖，都会影响目标；不能把某一数据分布的偏好直接描述为所有人的统一价值。[1]
2. DPO 的简化建立在明确假设上
DPO 在其偏好模型与 KL 约束目标设定下，利用奖励和策略之间的关系，把优化改写为直接基于成对偏好的损失。它省去单独拟合奖励模型和相应的强化学习训练循环，并不等于可以省去偏好数据、参考策略或验证。方法形式更简洁，也不会自动消除错误标注与分布偏差。[2]
3. 参考策略为什么重要
将目标策略与参考策略的相对对数概率进行比较，限制了“提高偏好答案概率”这件事的含义。参考模型提供一个基准，正则强度控制偏离的代价。不同数据、参数与实现会改变实际权衡。本文不把某个超参数值推荐为通用最优值，也不比较未经统一训练条件核对的模型分数。[2]
4. 用损失曲线理解目标，而非宣称效果
图 2 画出 ：m 是偏好方向的相对对数概率差，β 取 0.1、0.5、1，纯粹用于展示数学形状。m 变大时损失下降，不意味着回答更真实；若数据把讨好或自信表述标成优胜，降低这项损失也可能强化这些表面特征。图中的点没有来自模型训练日志。[2]
LaTeX 源码
\ell(m)=\log\!\left(1+\exp(-\beta m)\right)5. 置信与真实正确率也需分别校准
Kadavath 等人研究模型对答案有效性和可回答性的自我评估，在某些任务格式上得到有希望的结果，也记录了跨新任务校准的困难。因此，模型说“我有把握”，仍然需要与真实正确率对照。偏好优化是否改变这种对应关系，应作为独立评估问题，不能从更高偏好胜率直接推出。[3]
6. 比较两种方法应控制哪些变量
本文建议固定基础模型、数据划分与成对偏好，记录提示格式、参考模型、训练预算、评估者和抽样策略。同时分别测量指令遵循、事实正确性、校准、长度偏差和有害输出。若评估者偏爱更长回答，而两种方法的输出长度明显不同，应报告这一混杂，而不是只给一个总胜率。
7. 把分歧作为数据而非噪声
对于价值判断或表达风格，标注者可能合理地不同意。保留分歧比例、任务类别与标注指南，有助于理解策略学到了谁的偏好。事实判断则应尽量关联外部证据。把“这是我更喜欢的回答”与“这项陈述已被证据支持”分开记录，可以让后续更正和数据改进有明确对象。
偏好数据还需要时间与群体语境
某些回答在一个时期、一个地区或一类用户中更受欢迎，但相同排序未必适用于另一情境。数据记录应允许按任务类别与标注群体检查差异，尤其要区分事实纠错、表达风格和价值判断。若训练数据与评估数据由相同偏好模板生成，较高分数可能首先说明匹配了模板，而不是已经验证更广泛的用户效用。
后续实验可以如何设计
建立一个独立事实核对集合和一个单独的偏好比较集合。对不同方法使用相同候选显示规则，盲化模型名称并记录长度；评估者应知道如何处理事实错误，却不应根据品牌猜测质量。报告总体结果之外，也报告错误事实被偏爱的比例和群体分歧。若只有自动评委，明确评委模型与提示，并用人工抽样检验其适用范围。
局限与结论
本文解释两类基础方法，没有训练或评测新的对齐模型，也没有给任何当前系统作安全认证。偏好优化是一种目标塑形工具；它能否带来所需行为，取决于反馈、建模假设和实际测量。可靠系统仍需要来源核对、任务评估和部署中的纠错机制。
| 测量对象 | 证据形式 | 不能混用的结论 |
|---|---|---|
| 偏好拟合 | 成对比较与标注规则 | 所有事实都更准确 |
| 事实正确性 | 外部来源或可执行判定 | 所有人都更喜欢 |
| 校准 | 置信与实得正确率对应 | 任何新任务都可靠 |
一手来源与主张对应
- Ouyang et al. · Training language models to follow instructions with human feedback打开原始来源 ↗来源日期：2022-03-04；核对：2026-10-15；定位：v1; method and limitationsDemonstrations, preference reward learning and RL optimization form the reported InstructGPT pipeline; preference depends on the sampled prompt and labeler population.
- Rafailov et al. · Direct Preference Optimization: Your Language Model is Secretly a Reward Model打开原始来源 ↗来源日期：2023-05-29；核对：2026-10-15；定位：v3; sections 3-5A policy/reward reparameterization turns the stated KL-constrained preference objective into a direct classification-style loss under its assumptions.
- Kadavath et al. · Language Models (Mostly) Know What They Know打开原始来源 ↗来源日期：2022-07-11；核对：2026-10-15；定位：v4 abstractSelf-evaluation and calibration show encouraging task-specific findings but transfer of calibration to new tasks remains limited.
配图来源、数据和解析假设
[
{
"slug": "preference-alignment",
"role": "conceptual-cover",
"generator": "OpenAI built-in image_gen",
"ai_generated": true,
"alt": "金属与陶瓷双带穿过狭窄石门的偏好约束概念插画。",
"original_sha256": "25833a4b9139f12b6f4582199fbad8c020c307f60ee1d197d8f0d8a6cc6d9115",
"prompt": "Create a premium science journal editorial cover art, landscape 3:2, edge-to-edge artwork, NO text, letters, numbers, logos, borders or watermarks. Museum-quality sculptural still-life photographed with a large-format lens; physically believable materials and light, masterful composition, restrained warm gold details with rich dark ink accents. Sophisticated contemporary research magazine, tactile cinematic art, strong silhouette readable at thumbnail size. Conceptual illustration openly labeled by publisher, not an experimental visualization. Avoid generic AI brains, robots, microchips, neon cyberpunk, stock-tech clichés, flat infographic boxes, glossy plastic, rainbow gradients. Topic: preference alignment. Two exquisitely crafted ribbons, one pale pink ceramic and one brushed warm brass, begin on separate sides then curve gracefully through a single narrow dark stone gateway, emerging as a harmonious parallel pair. Abstract sculpture rather than a literal flow chart; a poetic image of choices becoming a constraint. Muted terracotta rose studio background, dark wine-colored gateway, warm ivory and gold ribbon highlights, dramatic soft side light, fine shadows, a subtle tactile paper backdrop, confident uncluttered composition with interesting interlaced geometry.",
"files": [
{
"file": "assets/covers/preference-alignment.png",
"width": 1280,
"height": 853,
"bytes": 1420806,
"sha256": "1f8d5577c46486ad19d41aacfce89bd4de9a0dc31ccb45802231032dd6e96662",
"url": "/guides/assets/v2-preference-alignment-1f8d5577c464.png"
},
{
"file": "assets/covers/preference-alignment-480.png",
"width": 480,
"height": 320,
"bytes": 210518,
"sha256": "0166cb8c0fc1a1f0012dc63935925f04ebe1ddd76e2f1ea8afd165a4a17256cf",
"url": "/guides/assets/v2-preference-alignment-480-0166cb8c0fc1.png"
}
],
"url": "/guides/assets/v2-preference-alignment-1f8d5577c464.png",
"thumbnail_url": "/guides/assets/v2-preference-alignment-480-0166cb8c0fc1.png"
},
{
"slug": "preference-alignment",
"role": "data-chart",
"file": "assets/charts/preference-alignment.png",
"width": 1800,
"height": 1125,
"sha256": "924820d01cf799c8386a79f882132b5159e35043fd523541a1fdd83db7b84f35",
"bytes": 139998,
"alt": "图表：是 DPO 风格二元逻辑损失的解析示意，m 与 β 是假设值；不表示实际偏好胜率、事实率或训练结果。",
"data": {
"kind": "analytical",
"formula": "log1p(exp(-beta*m))",
"m": [
-8.0,
-7.9,
-7.8,
-7.7,
-7.6,
-7.5,
-7.4,
-7.3,
-7.2,
-7.1,
-7.0,
-6.9,
-6.8,
-6.7,
-6.6,
-6.5,
-6.4,
-6.3,
-6.2,
-6.1,
-6.0,
-5.9,
-5.8,
-5.699999999999999,
-5.6,
-5.5,
-5.4,
-5.3,
-5.199999999999999,
-5.1,
-5.0,
-4.9,
-4.8,
-4.699999999999999,
-4.6,
-4.5,
-4.4,
-4.3,
-4.199999999999999,
-4.1,
-4.0,
-3.8999999999999995,
-3.8,
-3.7,
-3.5999999999999996,
-3.5,
-3.3999999999999995,
-3.3,
-3.1999999999999993,
-3.0999999999999996,
-3.0,
-2.8999999999999995,
-2.8,
-2.6999999999999993,
-2.5999999999999996,
-2.5,
-2.3999999999999995,
-2.3,
-2.1999999999999993,
-2.0999999999999996,
-2.0,
-1.8999999999999995,
-1.7999999999999998,
-1.6999999999999993,
-1.5999999999999996,
-1.5,
-1.3999999999999995,
-1.2999999999999998,
-1.1999999999999993,
-1.0999999999999996,
-1.0,
-0.8999999999999995,
-0.7999999999999998,
-0.6999999999999993,
-0.5999999999999996,
-0.5,
-0.39999999999999947,
-0.2999999999999998,
-0.1999999999999993,
-0.09999999999999964,
0.0,
0.09999999999999964,
0.20000000000000107,
0.3000000000000007,
0.40000000000000036,
0.5,
0.5999999999999996,
0.7000000000000011,
0.8000000000000007,
0.9000000000000004,
1.0,
1.0999999999999996,
1.200000000000001,
1.3000000000000007,
1.4000000000000004,
1.5,
1.6000000000000014,
1.700000000000001,
1.8000000000000007,
1.9000000000000004,
2.0,
2.1000000000000014,
2.200000000000001,
2.3000000000000007,
2.4000000000000004,
2.5,
2.6000000000000014,
2.700000000000001,
2.8000000000000007,
2.9000000000000004,
3.0,
3.1000000000000014,
3.200000000000001,
3.3000000000000007,
3.4000000000000004,
3.5,
3.6000000000000014,
3.700000000000001,
3.8000000000000007,
3.9000000000000004,
4.0,
4.100000000000001,
4.200000000000001,
4.300000000000001,
4.4,
4.5,
4.600000000000001,
4.700000000000001,
4.800000000000001,
4.9,
5.0,
5.100000000000001,
5.200000000000001,
5.300000000000001,
5.4,
5.5,
5.600000000000001,
5.700000000000001,
5.800000000000001,
5.9,
6.0,
6.100000000000001,
6.200000000000001,
6.300000000000001,
6.4,
6.5,
6.600000000000001,
6.700000000000001,
6.800000000000001,
6.9,
7.0,
7.100000000000001,
7.200000000000001,
7.300000000000001,
7.4,
7.5,
7.600000000000001,
7.700000000000001,
7.800000000000001,
7.9,
8.0
],
"loss_by_beta": {
"0.1": [
1.1711006659477778,
1.1642116301417518,
1.1573440662232617,
1.1504980545416523,
1.1436736748144938,
1.1368710061148999,
1.1300901268588703,
1.1233311147926668,
1.1165940469802245,
1.1098789997905982,
1.103186048885458,
1.0965152692066247,
1.0898667349636622,
1.0832405196215205,
1.0766366958882394,
1.0700553357027152,
1.0634965102225344,
1.0569602898118766,
1.050446744029496,
1.043955941616779,
1.0374879504858856,
1.0310428377079808,
1.0246206695015532,
1.0182215112208328,
1.0118454273443065,
1.0054924814633375,
0.9991627362708936,
0.9928562535503844,
0.9865730941646179,
0.980313318044875,
0.9740769841801067,
0.9678641506062616,
0.9616748743957434,
0.955509211647004,
0.9493672174742773,
0.9432489459974549,
0.9371544503321099,
0.9310837825796714,
0.9250369938177536,
0.9190141340906441,
0.9130152523999526,
0.9070403966954268,
0.9010896138659374,
0.895162949730635,
0.8892604490302842,
0.8833821554187771,
0.8775281114548288,
0.8716983585938615,
0.8658929371800753,
0.8601118864387145,
0.8543552444685272,
0.8486230482344254,
0.8429153335603464,
0.8372321351223193,
0.8315734864417375,
0.8259394198788436,
0.8203299666264259,
0.8147451567037305,
0.8091850189505919,
0.8036495810217837,
0.7981388693815918,
0.7926529092986133,
0.7871917248407819,
0.781755338870623,
0.7763437730407396,
0.770957047789532,
0.7655951823371514,
0.7602581946816906,
0.7549461015956135,
0.749658918622423,
0.744396660073571,
0.7391593390256102,
0.73394696731759,
0.7287595555486971,
0.7235971130761409,
0.7184596480132863,
0.713347167228034,
0.7082596763414485,
0.7031971797266342,
0.6981596805078623,
0.6931471805599453,
0.6881596805078624,
0.6831971797266341,
0.6782596763414483,
0.6733471672280341,
0.6684596480132863,
0.6635971130761409,
0.658759555548697,
0.65394696731759,
0.6491593390256101,
0.6443966600735709,
0.639658918622423,
0.6349461015956134,
0.6302581946816906,
0.6255951823371513,
0.6209570477895321,
0.6163437730407395,
0.6117553388706229,
0.6071917248407819,
0.6026529092986134,
0.5981388693815919,
0.5936495810217837,
0.5891850189505919,
0.5847451567037305,
0.5803299666264258,
0.5759394198788436,
0.5715734864417374,
0.5672321351223192,
0.5629153335603464,
0.5586230482344253,
0.5543552444685271,
0.5501118864387144,
0.5458929371800753,
0.5416983585938615,
0.5375281114548288,
0.5333821554187771,
0.5292604490302841,
0.5251629497306349,
0.5210896138659374,
0.5170403966954268,
0.5130152523999526,
0.5090141340906441,
0.5050369938177537,
0.5010837825796713,
0.4971544503321098,
0.4932489459974549,
0.48936721747427725,
0.48550921164700395,
0.48167487439574336,
0.47786415060626153,
0.4740769841801067,
0.470313318044875,
0.466573094164618,
0.4628562535503843,
0.45916273627089355,
0.4554924814633376,
0.4518454273443063,
0.4482215112208326,
0.44462066950155305,
0.44104283770798064,
0.4374879504858856,
0.4339559416167788,
0.43044674402949606,
0.4269602898118766,
0.42349651022253426,
0.4200553357027152,
0.4166366958882392,
0.41324051962152036,
0.40986673496366216,
0.4065152692066247,
0.4031860488854579,
0.3998789997905982,
0.3965940469802243,
0.39333111479266697,
0.39009012685887023,
0.38687100611489994,
0.3836736748144938,
0.38049805454165214,
0.37734406622326155,
0.3742116301417517,
0.3711006659477777
],
"0.5": [
4.0181499279178094,
3.9690716756821924,
3.9200397672603975,
3.8710564844556816,
3.822124216454879,
3.773245464372425,
3.724422845933779,
3.6756591002967287,
3.6269570930082082,
3.5783198210933684,
3.5297504182726205,
3.4812521603012354,
3.432828470424865,
3.384482924942972,
3.336219258870659,
3.288041371687783,
3.2399533331624304,
3.1919593892339417,
3.144063967938574,
3.096271685358662,
3.048587351573742,
3.0010159765895352,
2.953562776217963,
2.906233177878483,
2.8590328262879714,
2.8119675890031983,
2.765043561776591,
2.7182670736825036,
2.6716446919676695,
2.62518322657579,
2.5788897342925496,
2.5327715224535527,
2.4868361521539497,
2.441091440894841,
2.3955454645979626,
2.350206558916747,
2.305083319768696,
2.260184603011109,
2.2155195231797546,
2.171097451208061,
2.1269280110429727,
2.083021075072867,
2.0393867582829603,
1.9960354110545102,
1.9529776105260739,
1.9102241504380872,
1.8677860293862656,
1.8256744374149323,
1.7839007408883385,
1.7424764655865785,
1.7014132779827524,
1.6607229646697594,
1.620417409918451,
1.5805085713638751,
1.541008453832992,
1.501929081345373,
1.4632824673380311,
1.4250805831863982,
1.3873353251154306,
1.3500584796176431,
1.3132616875182228,
1.2769564068509518,
1.2411538747320878,
1.2058650684421957,
1.1711006659477776,
1.1368710061148999,
1.1031860488854577,
1.0700553357027152,
1.0374879504858854,
1.0054924814633375,
0.9740769841801067,
0.9432489459974547,
0.9130152523999525,
0.8833821554187768,
0.8543552444685271,
0.8259394198788436,
0.7981388693815916,
0.770957047789532,
0.7443966600735707,
0.7184596480132862,
0.6931471805599453,
0.6684596480132863,
0.6443966600735707,
0.620957047789532,
0.5981388693815918,
0.5759394198788436,
0.5543552444685272,
0.5333821554187769,
0.5130152523999525,
0.49324894599745484,
0.4740769841801067,
0.45549248146333765,
0.43748795048588546,
0.4200553357027151,
0.40318604888545784,
0.38687100611489994,
0.3711006659477775,
0.3558650684421957,
0.3411538747320877,
0.326956406850952,
0.31326168751822286,
0.30005847961764304,
0.28733532511543064,
0.2750805831863984,
0.2632824673380311,
0.2519290813453729,
0.24100845383299208,
0.2305085713638753,
0.22041740991845085,
0.21072296466975973,
0.2014132779827524,
0.19247646558657858,
0.18390074088833874,
0.17567443741493236,
0.16778602938626586,
0.16022415043808724,
0.15297761052607403,
0.14603541105450998,
0.13938675828296054,
0.1330210750728672,
0.1269280110429725,
0.12109745120806154,
0.1155195231797549,
0.11018460301110883,
0.10508331976869591,
0.10020655891674721,
0.09554546459796287,
0.09109144089484156,
0.08683615215394964,
0.0827715224535525,
0.07888973429254963,
0.07518322657579,
0.07164469196766977,
0.06826707368250394,
0.06504356177659054,
0.061967589003198625,
0.05903282628797136,
0.05623317787848312,
0.053562776217963154,
0.05101597658953503,
0.048587351573742055,
0.046271685358661926,
0.04406396793857383,
0.04195938923394163,
0.039953333162430354,
0.03804137168778313,
0.03621925887065925,
0.034482924942971935,
0.03282847042486533,
0.03125216030123531,
0.029750418272620566,
0.028319821093368586,
0.026957093008208047,
0.02565910029672885,
0.024422845933779157,
0.02324546437242503,
0.022124216454879178,
0.021056484455681364,
0.0200397672603976,
0.01907167568219249,
0.018149927917809738
],
"1": [
8.000335406372896,
7.900370674832055,
7.800409651060525,
7.700452724687599,
7.600500326249386,
7.500552931475361,
7.400611066022253,
7.3006753107015845,
7.200746307251828,
7.100824764711326,
7.000911466453774,
6.9010072779542355,
6.801113155360465,
6.701230154951714,
6.60135944357526,
6.501502310159754,
6.401660178414046,
6.301834620830589,
6.202027374123839,
6.102240356246249,
6.00247568513773,
5.902735699378536,
5.803022980930831,
5.703340380170367,
5.603691043426946,
5.50407844327057,
5.404506411799249,
5.304979177204327,
5.205501403909657,
5.106078236601779,
5.006715348489118,
4.907418994148687,
4.808196067338268,
4.709054164169887,
4.610001652055652,
4.511047744848594,
4.412202584607696,
4.313477330416026,
4.214884254671918,
4.116436847252909,
4.0181499279178094,
3.920039767260397,
3.822124216454879,
3.724422845933779,
3.626957093008208,
3.5297504182726205,
3.4328284704248646,
3.336219258870659,
3.2399533331624295,
3.1440639679385733,
3.048587351573742,
2.9535627762179626,
2.8590328262879714,
2.7650435617765896,
2.6716446919676695,
2.5788897342925496,
2.4868361521539493,
2.3955454645979626,
2.3050833197686953,
2.2155195231797546,
2.1269280110429727,
2.0393867582829603,
1.9529776105260739,
1.8677860293862651,
1.7839007408883385,
1.7014132779827524,
1.6204174099184505,
1.541008453832992,
1.4632824673380307,
1.3873353251154306,
1.3132616875182228,
1.2411538747320876,
1.1711006659477776,
1.1031860488854575,
1.0374879504858854,
0.9740769841801067,
0.9130152523999523,
0.8543552444685271,
0.7981388693815915,
0.7443966600735707,
0.6931471805599453,
0.6443966600735711,
0.5981388693815913,
0.5543552444685268,
0.5130152523999525,
0.4740769841801067,
0.43748795048588573,
0.4031860488854575,
0.3711006659477775,
0.3411538747320877,
0.31326168751822286,
0.28733532511543086,
0.2632824673380309,
0.24100845383299208,
0.22041740991845085,
0.2014132779827524,
0.1839007408883386,
0.16778602938626577,
0.15297761052607403,
0.13938675828296054,
0.1269280110429725,
0.11551952317975481,
0.1050833197686958,
0.09554546459796287,
0.08683615215394964,
0.07888973429254963,
0.07164469196766972,
0.06504356177659047,
0.05903282628797136,
0.053562776217963154,
0.048587351573742055,
0.0440639679385738,
0.03995333316243031,
0.03621925887065925,
0.03282847042486533,
0.029750418272620566,
0.02695709300820802,
0.024422845933779132,
0.022124216454879178,
0.0200397672603976,
0.018149927917809738,
0.0164368472529095,
0.014884254671918113,
0.013477330416026292,
0.012202584607696094,
0.011047744848593817,
0.010001652055651816,
0.009054164169887476,
0.008196067338267768,
0.007418994148686707,
0.006715348489118068,
0.006078236601779187,
0.005501403909657369,
0.004979177204327226,
0.004506411799249424,
0.0040784432705707205,
0.003691043426946533,
0.0033403801703674172,
0.0030229809308316173,
0.0027356993785361177,
0.0024756851377304495,
0.0022403562462493683,
0.002027374123838178,
0.001834620830589278,
0.0016601784140455604,
0.0015023101597542838,
0.0013594435752600897,
0.001230154951713698,
0.0011131553604646566,
0.0010072779542348728,
0.0009114664537742447,
0.0008247647113262067,
0.0007463072518276657,
0.0006753107015847127,
0.0006110660222531391,
0.0005529314753607964,
0.0005003262493859453,
0.0004527246875986171,
0.0004096510605253882,
0.0003706748320542991,
0.00033540637289576885
]
}
},
"data_source_sha256": "963323d4a7a0cad4aebd986f09a55251a7c21f68c757d21b76245a128675e13e",
"ai_generated": true,
"disclosure": "AI-assisted deterministic rendering of unchanged v1 data; not a new experiment.",
"url": "/guides/assets/v2-preference-alignment-924820d01cf7.png"
}
]
作者、版本与更正记录
v3 补齐五种界面语言，保留原文、29 项来源和图表数据；v1 与 v2 继续可读。
文献导读与方法分析；不是原创实验论文、正式期刊录用或专业资质认证。解析示例不代表本站或当前产品的实测表现。
查看 v1 原版 · 下载本篇正文、配图与来源对应查看 v2 原版