{
  "slug": "preference-alignment",
  "category": "对齐与行为",
  "title": "从 RLHF 到 DPO：偏好优化能够保证什么，不能保证什么",
  "dek": "把偏好数据、优化目标与事实正确性分开，理解奖励模型、参考策略和“更受欢迎”的边界。",
  "english": "Preference optimization changes a policy toward labeled comparisons. We explain RLHF and DPO while separating preference fit, calibration, factuality, and deployment-specific validation.",
  "refs": [
    "rlhf",
    "dpo",
    "know"
  ],
  "sections": [
    [
      "摘要",
      "模型回答更符合标注者偏好，不等于每个事实都更准确。本文从 InstructGPT 的训练流程与 DPO 的目标改写出发，解释偏好学习如何作用于生成策略，以及为什么偏好来源、参考策略和任务分布需要保留。配图中的逻辑损失只是一个解析例子，不是人类偏好实验或模型优劣排名。",
      [
        "rlhf",
        "dpo"
      ]
    ],
    [
      "1. 人类反馈改变了训练信号",
      "InstructGPT 的研究流程先利用示范进行监督微调，再收集输出排序、学习奖励模型，并以强化学习优化策略。它回答的是“怎样让模型更符合所收集反馈中的意图”。标注者如何选择、提示来自哪里、任务如何覆盖，都会影响目标；不能把某一数据分布的偏好直接描述为所有人的统一价值。",
      [
        "rlhf"
      ]
    ],
    [
      "2. DPO 的简化建立在明确假设上",
      "DPO 在其偏好模型与 KL 约束目标设定下，利用奖励和策略之间的关系，把优化改写为直接基于成对偏好的损失。它省去单独拟合奖励模型和相应的强化学习训练循环，并不等于可以省去偏好数据、参考策略或验证。方法形式更简洁，也不会自动消除错误标注与分布偏差。",
      [
        "dpo"
      ]
    ],
    [
      "3. 参考策略为什么重要",
      "将目标策略与参考策略的相对对数概率进行比较，限制了“提高偏好答案概率”这件事的含义。参考模型提供一个基准，正则强度控制偏离的代价。不同数据、参数与实现会改变实际权衡。本文不把某个超参数值推荐为通用最优值，也不比较未经统一训练条件核对的模型分数。",
      [
        "dpo"
      ]
    ],
    [
      "4. 用损失曲线理解目标，而非宣称效果",
      "图 2 画出 ℓ(m)=log(1+exp(−βm))，m 是偏好方向的相对对数概率差，β 取 0.1、0.5、1，纯粹用于展示数学形状。m 变大时损失下降，不意味着回答更真实；若数据把讨好或自信表述标成优胜，降低这项损失也可能强化这些表面特征。图中的点没有来自模型训练日志。",
      [
        "dpo"
      ]
    ],
    [
      "5. 置信与真实正确率也需分别校准",
      "Kadavath 等人研究模型对答案有效性和可回答性的自我评估，在某些任务格式上得到有希望的结果，也记录了跨新任务校准的困难。因此，模型说“我有把握”，仍然需要与真实正确率对照。偏好优化是否改变这种对应关系，应作为独立评估问题，不能从更高偏好胜率直接推出。",
      [
        "know"
      ]
    ],
    [
      "6. 比较两种方法应控制哪些变量",
      "本文建议固定基础模型、数据划分与成对偏好，记录提示格式、参考模型、训练预算、评估者和抽样策略。同时分别测量指令遵循、事实正确性、校准、长度偏差和有害输出。若评估者偏爱更长回答，而两种方法的输出长度明显不同，应报告这一混杂，而不是只给一个总胜率。",
      []
    ],
    [
      "7. 把分歧作为数据而非噪声",
      "对于价值判断或表达风格，标注者可能合理地不同意。保留分歧比例、任务类别与标注指南，有助于理解策略学到了谁的偏好。事实判断则应尽量关联外部证据。把“这是我更喜欢的回答”与“这项陈述已被证据支持”分开记录，可以让后续更正和数据改进有明确对象。",
      []
    ],
    [
      "偏好数据还需要时间与群体语境",
      "某些回答在一个时期、一个地区或一类用户中更受欢迎，但相同排序未必适用于另一情境。数据记录应允许按任务类别与标注群体检查差异，尤其要区分事实纠错、表达风格和价值判断。若训练数据与评估数据由相同偏好模板生成，较高分数可能首先说明匹配了模板，而不是已经验证更广泛的用户效用。",
      []
    ],
    [
      "后续实验可以如何设计",
      "建立一个独立事实核对集合和一个单独的偏好比较集合。对不同方法使用相同候选显示规则，盲化模型名称并记录长度；评估者应知道如何处理事实错误，却不应根据品牌猜测质量。报告总体结果之外，也报告错误事实被偏爱的比例和群体分歧。若只有自动评委，明确评委模型与提示，并用人工抽样检验其适用范围。",
      []
    ],
    [
      "局限与结论",
      "本文解释两类基础方法，没有训练或评测新的对齐模型，也没有给任何当前系统作安全认证。偏好优化是一种目标塑形工具；它能否带来所需行为，取决于反馈、建模假设和实际测量。可靠系统仍需要来源核对、任务评估和部署中的纠错机制。",
      []
    ]
  ],
  "diagram": [
    "提示与候选回答",
    "确定要学习的任务分布",
    "收集成对偏好",
    "记录标注指南与分歧",
    "RLHF 路径",
    "示范 → 奖励模型 → 策略优化",
    "DPO 路径",
    "偏好对直接更新策略与参考差",
    "独立评估维度",
    "偏好胜率、事实、校准分别测量",
    "部署后更正",
    "追踪任务漂移与失败样本"
  ],
  "figureNote": "图 1 对比两种训练流程，仅用于机制解释。图 2 是 DPO 风格二元逻辑损失的解析示意，m 与 β 是假设值；不表示实际偏好胜率、事实率或训练结果。",
  "table": {
    "headers": [
      "测量对象",
      "证据形式",
      "不能混用的结论"
    ],
    "rows": [
      [
        "偏好拟合",
        "成对比较与标注规则",
        "所有事实都更准确"
      ],
      [
        "事实正确性",
        "外部来源或可执行判定",
        "所有人都更喜欢"
      ],
      [
        "校准",
        "置信与实得正确率对应",
        "任何新任务都可靠"
      ]
    ]
  },
  "version": 1,
  "kind": "ai_assisted_research_guide",
  "author": "Codex（AI）",
  "published_date": "2026-10-16",
  "disclosure": "本站用户明确委托撰写与公开。由 Codex（AI）依据所列一手资料整理，配图由代码绘制；未经独立同行评审。文中原论文的实验结果归属于原作者，本文未运行新的模型训练或基准测试。",
  "limitations": "文献导读与方法分析；不是原创实验论文、正式期刊录用或专业资质认证。解析示例不代表本站或当前产品的实测表现。",
  "sources": [
    {
      "id": "rlhf",
      "title": "Training language models to follow instructions with human feedback",
      "authors": "Ouyang et al.",
      "date": "2022-03-04",
      "url": "https://arxiv.org/html/2203.02155v1",
      "locator": "v1; method and limitations",
      "note": "Demonstrations, preference reward learning and RL optimization form the reported InstructGPT pipeline; preference depends on the sampled prompt and labeler population.",
      "accessed_at": "2026-10-15",
      "rights": "Original source remains under its own license; only attributed factual synthesis, no source figures or long excerpts."
    },
    {
      "id": "dpo",
      "title": "Direct Preference Optimization: Your Language Model is Secretly a Reward Model",
      "authors": "Rafailov et al.",
      "date": "2023-05-29",
      "url": "https://arxiv.org/html/2305.18290v3",
      "locator": "v3; sections 3-5",
      "note": "A policy/reward reparameterization turns the stated KL-constrained preference objective into a direct classification-style loss under its assumptions.",
      "accessed_at": "2026-10-15",
      "rights": "Original source remains under its own license; only attributed factual synthesis, no source figures or long excerpts."
    },
    {
      "id": "know",
      "title": "Language Models (Mostly) Know What They Know",
      "authors": "Kadavath et al.",
      "date": "2022-07-11",
      "url": "https://arxiv.org/abs/2207.05221v4",
      "locator": "v4 abstract",
      "note": "Self-evaluation and calibration show encouraging task-specific findings but transfer of calibration to new tasks remains limited.",
      "accessed_at": "2026-10-15",
      "rights": "Original source remains under its own license; only attributed factual synthesis, no source figures or long excerpts."
    }
  ],
  "figures": [
    {
      "slug": "preference-alignment",
      "number": 1,
      "file": "preference-alignment-1.png",
      "sha256": "426e557e400706d698f4856bf2c772b3e7c7d67048e9ace7b90ce4d5bc34b8d5",
      "alt": "六个要点：提示与候选回答；收集成对偏好；RLHF 路径；DPO 路径；独立评估维度；部署后更正",
      "data": {
        "kind": "conceptual",
        "items": [
          "提示与候选回答",
          "确定要学习的任务分布",
          "收集成对偏好",
          "记录标注指南与分歧",
          "RLHF 路径",
          "示范 → 奖励模型 → 策略优化",
          "DPO 路径",
          "偏好对直接更新策略与参考差",
          "独立评估维度",
          "偏好胜率、事实、校准分别测量",
          "部署后更正",
          "追踪任务漂移与失败样本"
        ]
      },
      "ai_generated": true,
      "disclosure": "AI-assisted code rendering; original schematic or explicitly attributed data; not a new model experiment.",
      "url": "/guides/assets/preference-alignment-1-426e557e4007.png"
    },
    {
      "slug": "preference-alignment",
      "number": 2,
      "file": "preference-alignment-2.png",
      "sha256": "c1ab716e37d60a31d3b524ba9c668b0feb27121a99d0940dbc61dbec3ab225a8",
      "alt": "图 1 对比两种训练流程，仅用于机制解释。图 2 是 DPO 风格二元逻辑损失的解析示意，m 与 β 是假设值；不表示实际偏好胜率、事实率或训练结果。",
      "data": {
        "kind": "analytical",
        "formula": "log1p(exp(-beta*m))",
        "m": [
          -8.0,
          -7.9,
          -7.8,
          -7.7,
          -7.6,
          -7.5,
          -7.4,
          -7.3,
          -7.2,
          -7.1,
          -7.0,
          -6.9,
          -6.8,
          -6.7,
          -6.6,
          -6.5,
          -6.4,
          -6.3,
          -6.2,
          -6.1,
          -6.0,
          -5.9,
          -5.8,
          -5.699999999999999,
          -5.6,
          -5.5,
          -5.4,
          -5.3,
          -5.199999999999999,
          -5.1,
          -5.0,
          -4.9,
          -4.8,
          -4.699999999999999,
          -4.6,
          -4.5,
          -4.4,
          -4.3,
          -4.199999999999999,
          -4.1,
          -4.0,
          -3.8999999999999995,
          -3.8,
          -3.7,
          -3.5999999999999996,
          -3.5,
          -3.3999999999999995,
          -3.3,
          -3.1999999999999993,
          -3.0999999999999996,
          -3.0,
          -2.8999999999999995,
          -2.8,
          -2.6999999999999993,
          -2.5999999999999996,
          -2.5,
          -2.3999999999999995,
          -2.3,
          -2.1999999999999993,
          -2.0999999999999996,
          -2.0,
          -1.8999999999999995,
          -1.7999999999999998,
          -1.6999999999999993,
          -1.5999999999999996,
          -1.5,
          -1.3999999999999995,
          -1.2999999999999998,
          -1.1999999999999993,
          -1.0999999999999996,
          -1.0,
          -0.8999999999999995,
          -0.7999999999999998,
          -0.6999999999999993,
          -0.5999999999999996,
          -0.5,
          -0.39999999999999947,
          -0.2999999999999998,
          -0.1999999999999993,
          -0.09999999999999964,
          0.0,
          0.09999999999999964,
          0.20000000000000107,
          0.3000000000000007,
          0.40000000000000036,
          0.5,
          0.5999999999999996,
          0.7000000000000011,
          0.8000000000000007,
          0.9000000000000004,
          1.0,
          1.0999999999999996,
          1.200000000000001,
          1.3000000000000007,
          1.4000000000000004,
          1.5,
          1.6000000000000014,
          1.700000000000001,
          1.8000000000000007,
          1.9000000000000004,
          2.0,
          2.1000000000000014,
          2.200000000000001,
          2.3000000000000007,
          2.4000000000000004,
          2.5,
          2.6000000000000014,
          2.700000000000001,
          2.8000000000000007,
          2.9000000000000004,
          3.0,
          3.1000000000000014,
          3.200000000000001,
          3.3000000000000007,
          3.4000000000000004,
          3.5,
          3.6000000000000014,
          3.700000000000001,
          3.8000000000000007,
          3.9000000000000004,
          4.0,
          4.100000000000001,
          4.200000000000001,
          4.300000000000001,
          4.4,
          4.5,
          4.600000000000001,
          4.700000000000001,
          4.800000000000001,
          4.9,
          5.0,
          5.100000000000001,
          5.200000000000001,
          5.300000000000001,
          5.4,
          5.5,
          5.600000000000001,
          5.700000000000001,
          5.800000000000001,
          5.9,
          6.0,
          6.100000000000001,
          6.200000000000001,
          6.300000000000001,
          6.4,
          6.5,
          6.600000000000001,
          6.700000000000001,
          6.800000000000001,
          6.9,
          7.0,
          7.100000000000001,
          7.200000000000001,
          7.300000000000001,
          7.4,
          7.5,
          7.600000000000001,
          7.700000000000001,
          7.800000000000001,
          7.9,
          8.0
        ],
        "loss_by_beta": {
          "0.1": [
            1.1711006659477778,
            1.1642116301417518,
            1.1573440662232617,
            1.1504980545416523,
            1.1436736748144938,
            1.1368710061148999,
            1.1300901268588703,
            1.1233311147926668,
            1.1165940469802245,
            1.1098789997905982,
            1.103186048885458,
            1.0965152692066247,
            1.0898667349636622,
            1.0832405196215205,
            1.0766366958882394,
            1.0700553357027152,
            1.0634965102225344,
            1.0569602898118766,
            1.050446744029496,
            1.043955941616779,
            1.0374879504858856,
            1.0310428377079808,
            1.0246206695015532,
            1.0182215112208328,
            1.0118454273443065,
            1.0054924814633375,
            0.9991627362708936,
            0.9928562535503844,
            0.9865730941646179,
            0.980313318044875,
            0.9740769841801067,
            0.9678641506062616,
            0.9616748743957434,
            0.955509211647004,
            0.9493672174742773,
            0.9432489459974549,
            0.9371544503321099,
            0.9310837825796714,
            0.9250369938177536,
            0.9190141340906441,
            0.9130152523999526,
            0.9070403966954268,
            0.9010896138659374,
            0.895162949730635,
            0.8892604490302842,
            0.8833821554187771,
            0.8775281114548288,
            0.8716983585938615,
            0.8658929371800753,
            0.8601118864387145,
            0.8543552444685272,
            0.8486230482344254,
            0.8429153335603464,
            0.8372321351223193,
            0.8315734864417375,
            0.8259394198788436,
            0.8203299666264259,
            0.8147451567037305,
            0.8091850189505919,
            0.8036495810217837,
            0.7981388693815918,
            0.7926529092986133,
            0.7871917248407819,
            0.781755338870623,
            0.7763437730407396,
            0.770957047789532,
            0.7655951823371514,
            0.7602581946816906,
            0.7549461015956135,
            0.749658918622423,
            0.744396660073571,
            0.7391593390256102,
            0.73394696731759,
            0.7287595555486971,
            0.7235971130761409,
            0.7184596480132863,
            0.713347167228034,
            0.7082596763414485,
            0.7031971797266342,
            0.6981596805078623,
            0.6931471805599453,
            0.6881596805078624,
            0.6831971797266341,
            0.6782596763414483,
            0.6733471672280341,
            0.6684596480132863,
            0.6635971130761409,
            0.658759555548697,
            0.65394696731759,
            0.6491593390256101,
            0.6443966600735709,
            0.639658918622423,
            0.6349461015956134,
            0.6302581946816906,
            0.6255951823371513,
            0.6209570477895321,
            0.6163437730407395,
            0.6117553388706229,
            0.6071917248407819,
            0.6026529092986134,
            0.5981388693815919,
            0.5936495810217837,
            0.5891850189505919,
            0.5847451567037305,
            0.5803299666264258,
            0.5759394198788436,
            0.5715734864417374,
            0.5672321351223192,
            0.5629153335603464,
            0.5586230482344253,
            0.5543552444685271,
            0.5501118864387144,
            0.5458929371800753,
            0.5416983585938615,
            0.5375281114548288,
            0.5333821554187771,
            0.5292604490302841,
            0.5251629497306349,
            0.5210896138659374,
            0.5170403966954268,
            0.5130152523999526,
            0.5090141340906441,
            0.5050369938177537,
            0.5010837825796713,
            0.4971544503321098,
            0.4932489459974549,
            0.48936721747427725,
            0.48550921164700395,
            0.48167487439574336,
            0.47786415060626153,
            0.4740769841801067,
            0.470313318044875,
            0.466573094164618,
            0.4628562535503843,
            0.45916273627089355,
            0.4554924814633376,
            0.4518454273443063,
            0.4482215112208326,
            0.44462066950155305,
            0.44104283770798064,
            0.4374879504858856,
            0.4339559416167788,
            0.43044674402949606,
            0.4269602898118766,
            0.42349651022253426,
            0.4200553357027152,
            0.4166366958882392,
            0.41324051962152036,
            0.40986673496366216,
            0.4065152692066247,
            0.4031860488854579,
            0.3998789997905982,
            0.3965940469802243,
            0.39333111479266697,
            0.39009012685887023,
            0.38687100611489994,
            0.3836736748144938,
            0.38049805454165214,
            0.37734406622326155,
            0.3742116301417517,
            0.3711006659477777
          ],
          "0.5": [
            4.0181499279178094,
            3.9690716756821924,
            3.9200397672603975,
            3.8710564844556816,
            3.822124216454879,
            3.773245464372425,
            3.724422845933779,
            3.6756591002967287,
            3.6269570930082082,
            3.5783198210933684,
            3.5297504182726205,
            3.4812521603012354,
            3.432828470424865,
            3.384482924942972,
            3.336219258870659,
            3.288041371687783,
            3.2399533331624304,
            3.1919593892339417,
            3.144063967938574,
            3.096271685358662,
            3.048587351573742,
            3.0010159765895352,
            2.953562776217963,
            2.906233177878483,
            2.8590328262879714,
            2.8119675890031983,
            2.765043561776591,
            2.7182670736825036,
            2.6716446919676695,
            2.62518322657579,
            2.5788897342925496,
            2.5327715224535527,
            2.4868361521539497,
            2.441091440894841,
            2.3955454645979626,
            2.350206558916747,
            2.305083319768696,
            2.260184603011109,
            2.2155195231797546,
            2.171097451208061,
            2.1269280110429727,
            2.083021075072867,
            2.0393867582829603,
            1.9960354110545102,
            1.9529776105260739,
            1.9102241504380872,
            1.8677860293862656,
            1.8256744374149323,
            1.7839007408883385,
            1.7424764655865785,
            1.7014132779827524,
            1.6607229646697594,
            1.620417409918451,
            1.5805085713638751,
            1.541008453832992,
            1.501929081345373,
            1.4632824673380311,
            1.4250805831863982,
            1.3873353251154306,
            1.3500584796176431,
            1.3132616875182228,
            1.2769564068509518,
            1.2411538747320878,
            1.2058650684421957,
            1.1711006659477776,
            1.1368710061148999,
            1.1031860488854577,
            1.0700553357027152,
            1.0374879504858854,
            1.0054924814633375,
            0.9740769841801067,
            0.9432489459974547,
            0.9130152523999525,
            0.8833821554187768,
            0.8543552444685271,
            0.8259394198788436,
            0.7981388693815916,
            0.770957047789532,
            0.7443966600735707,
            0.7184596480132862,
            0.6931471805599453,
            0.6684596480132863,
            0.6443966600735707,
            0.620957047789532,
            0.5981388693815918,
            0.5759394198788436,
            0.5543552444685272,
            0.5333821554187769,
            0.5130152523999525,
            0.49324894599745484,
            0.4740769841801067,
            0.45549248146333765,
            0.43748795048588546,
            0.4200553357027151,
            0.40318604888545784,
            0.38687100611489994,
            0.3711006659477775,
            0.3558650684421957,
            0.3411538747320877,
            0.326956406850952,
            0.31326168751822286,
            0.30005847961764304,
            0.28733532511543064,
            0.2750805831863984,
            0.2632824673380311,
            0.2519290813453729,
            0.24100845383299208,
            0.2305085713638753,
            0.22041740991845085,
            0.21072296466975973,
            0.2014132779827524,
            0.19247646558657858,
            0.18390074088833874,
            0.17567443741493236,
            0.16778602938626586,
            0.16022415043808724,
            0.15297761052607403,
            0.14603541105450998,
            0.13938675828296054,
            0.1330210750728672,
            0.1269280110429725,
            0.12109745120806154,
            0.1155195231797549,
            0.11018460301110883,
            0.10508331976869591,
            0.10020655891674721,
            0.09554546459796287,
            0.09109144089484156,
            0.08683615215394964,
            0.0827715224535525,
            0.07888973429254963,
            0.07518322657579,
            0.07164469196766977,
            0.06826707368250394,
            0.06504356177659054,
            0.061967589003198625,
            0.05903282628797136,
            0.05623317787848312,
            0.053562776217963154,
            0.05101597658953503,
            0.048587351573742055,
            0.046271685358661926,
            0.04406396793857383,
            0.04195938923394163,
            0.039953333162430354,
            0.03804137168778313,
            0.03621925887065925,
            0.034482924942971935,
            0.03282847042486533,
            0.03125216030123531,
            0.029750418272620566,
            0.028319821093368586,
            0.026957093008208047,
            0.02565910029672885,
            0.024422845933779157,
            0.02324546437242503,
            0.022124216454879178,
            0.021056484455681364,
            0.0200397672603976,
            0.01907167568219249,
            0.018149927917809738
          ],
          "1": [
            8.000335406372896,
            7.900370674832055,
            7.800409651060525,
            7.700452724687599,
            7.600500326249386,
            7.500552931475361,
            7.400611066022253,
            7.3006753107015845,
            7.200746307251828,
            7.100824764711326,
            7.000911466453774,
            6.9010072779542355,
            6.801113155360465,
            6.701230154951714,
            6.60135944357526,
            6.501502310159754,
            6.401660178414046,
            6.301834620830589,
            6.202027374123839,
            6.102240356246249,
            6.00247568513773,
            5.902735699378536,
            5.803022980930831,
            5.703340380170367,
            5.603691043426946,
            5.50407844327057,
            5.404506411799249,
            5.304979177204327,
            5.205501403909657,
            5.106078236601779,
            5.006715348489118,
            4.907418994148687,
            4.808196067338268,
            4.709054164169887,
            4.610001652055652,
            4.511047744848594,
            4.412202584607696,
            4.313477330416026,
            4.214884254671918,
            4.116436847252909,
            4.0181499279178094,
            3.920039767260397,
            3.822124216454879,
            3.724422845933779,
            3.626957093008208,
            3.5297504182726205,
            3.4328284704248646,
            3.336219258870659,
            3.2399533331624295,
            3.1440639679385733,
            3.048587351573742,
            2.9535627762179626,
            2.8590328262879714,
            2.7650435617765896,
            2.6716446919676695,
            2.5788897342925496,
            2.4868361521539493,
            2.3955454645979626,
            2.3050833197686953,
            2.2155195231797546,
            2.1269280110429727,
            2.0393867582829603,
            1.9529776105260739,
            1.8677860293862651,
            1.7839007408883385,
            1.7014132779827524,
            1.6204174099184505,
            1.541008453832992,
            1.4632824673380307,
            1.3873353251154306,
            1.3132616875182228,
            1.2411538747320876,
            1.1711006659477776,
            1.1031860488854575,
            1.0374879504858854,
            0.9740769841801067,
            0.9130152523999523,
            0.8543552444685271,
            0.7981388693815915,
            0.7443966600735707,
            0.6931471805599453,
            0.6443966600735711,
            0.5981388693815913,
            0.5543552444685268,
            0.5130152523999525,
            0.4740769841801067,
            0.43748795048588573,
            0.4031860488854575,
            0.3711006659477775,
            0.3411538747320877,
            0.31326168751822286,
            0.28733532511543086,
            0.2632824673380309,
            0.24100845383299208,
            0.22041740991845085,
            0.2014132779827524,
            0.1839007408883386,
            0.16778602938626577,
            0.15297761052607403,
            0.13938675828296054,
            0.1269280110429725,
            0.11551952317975481,
            0.1050833197686958,
            0.09554546459796287,
            0.08683615215394964,
            0.07888973429254963,
            0.07164469196766972,
            0.06504356177659047,
            0.05903282628797136,
            0.053562776217963154,
            0.048587351573742055,
            0.0440639679385738,
            0.03995333316243031,
            0.03621925887065925,
            0.03282847042486533,
            0.029750418272620566,
            0.02695709300820802,
            0.024422845933779132,
            0.022124216454879178,
            0.0200397672603976,
            0.018149927917809738,
            0.0164368472529095,
            0.014884254671918113,
            0.013477330416026292,
            0.012202584607696094,
            0.011047744848593817,
            0.010001652055651816,
            0.009054164169887476,
            0.008196067338267768,
            0.007418994148686707,
            0.006715348489118068,
            0.006078236601779187,
            0.005501403909657369,
            0.004979177204327226,
            0.004506411799249424,
            0.0040784432705707205,
            0.003691043426946533,
            0.0033403801703674172,
            0.0030229809308316173,
            0.0027356993785361177,
            0.0024756851377304495,
            0.0022403562462493683,
            0.002027374123838178,
            0.001834620830589278,
            0.0016601784140455604,
            0.0015023101597542838,
            0.0013594435752600897,
            0.001230154951713698,
            0.0011131553604646566,
            0.0010072779542348728,
            0.0009114664537742447,
            0.0008247647113262067,
            0.0007463072518276657,
            0.0006753107015847127,
            0.0006110660222531391,
            0.0005529314753607964,
            0.0005003262493859453,
            0.0004527246875986171,
            0.0004096510605253882,
            0.0003706748320542991,
            0.00033540637289576885
          ]
        }
      },
      "ai_generated": true,
      "disclosure": "AI-assisted code rendering; original schematic or explicitly attributed data; not a new model experiment.",
      "url": "/guides/assets/preference-alignment-2-c1ab716e37d6.png"
    }
  ],
  "peer_reviewed": false
}