{
  "schema": "auturnal.deep-research-review.v1",
  "slug": "data-dedup-contamination",
  "title": "数据去重与评测污染：从样本家族到可信泛化分数",
  "category": "数据与评测",
  "dek": "分析重复、近似变体和测试泄漏，建立能复核误报与漏报的污染审计。",
  "english": "Deduplication and benchmark contamination are separate audit questions. We analyze exact matches, near-duplicate families, privacy-related memorization and evaluation leakage. A declared mixture model illustrates how contaminated items can inflate a score without improving uncontaminated accuracy. The figure is not evidence about any existing model or benchmark.",
  "kind": "ai_assisted_research_guide",
  "author": "Codex（AI）",
  "original_language": "zh-Hans",
  "published_date": "2026-10-22",
  "edition": 5,
  "version": 6,
  "previous_version": "/guides/data-dedup-contamination/v5/",
  "refs": [
    "arxiv-2107.06499",
    "arxiv-2104.08758",
    "helm"
  ],
  "sections": [
    [
      "摘要",
      "重复训练样本可能改变记忆行为，测试内容进入训练又可能使泛化评估失真，但二者不是同一个问题。本文区分精确重复、近似家族、语义重叠和评测泄漏，提出先按来源家族划分、再审计相似匹配的流程。解析图仅展示假设污染比例下的分数混合，不指控任何真实模型或数据集。",
      [
        "arxiv-2107.06499",
        "arxiv-2104.08758",
        "helm"
      ]
    ],
    [
      "1. 去重目标应先写清楚",
      "减少重复训练、降低隐私记忆风险与防止测试污染，对重复的定义可能不同。同一公开事实出现在多个来源不必然构成评测作弊；同一道题的轻微改写却可能泄漏答案。去重研究展示了重复数据对训练与评估的影响，训练数据提取研究则关注记忆泄露风险。设计审计时应明确保护对象和判断单位，避免用一个相似度阈值代替所有政策。",
      [
        "arxiv-2107.06499",
        "arxiv-2104.08758"
      ]
    ],
    [
      "2. 精确匹配可靠但覆盖有限",
      "规范化后哈希能高效识别完全一致文本，但空白、格式、翻译和改写都会改变字节。过强规范化又可能把本来不同的代码或数学题合并。应保存原文哈希、规范化规则和匹配位置，使每次删除可回溯。近似匹配可以补充覆盖，但必须估计误报和漏报，特别是模板相同而实例不同的题目，不能只展示几个成功找到的重复案例。",
      [
        "arxiv-2107.06499"
      ]
    ],
    [
      "3. 家族划分比逐条随机划分更重要",
      "同一原始文档、题目或实验的改写、翻译与解释应作为相关家族处理。若先逐条随机划分，再去除精确重复，变体可能已横跨训练和测试。应在划分前建立来源或生成关系，并把整组分配到同一侧。无法确定家族时，要保留不确定标记和人工复核，而不是把未命中检测器直接称为无污染。检索增强系统的索引也属于需要审计的数据来源。",
      [
        "arxiv-2107.06499",
        "helm"
      ]
    ],
    [
      "4. 混合分数掩盖了未污染能力",
      "设污染样本占比c，其准确率为a1，其他样本为a0，总分是二者的加权平均。即便a0完全不变，c增加也能让总分上升。本篇图固定a0=0.5、a1=0.99，展示这个恒等关系；它不估计任何真实污染率。实际审计应按匹配强度和时间来源分层，分别报告排除前后分数与样本数，并承认检测不完整。",
      [
        "helm",
        "arxiv-2107.06499"
      ]
    ],
    [
      "5. 隐私记忆与答题泄漏需要不同证据",
      "模型能输出训练片段，不自动证明某次 benchmark 分数由污染造成；测试题出现在训练中，也不意味着其中一定含个人隐私。前者需要受控提取与隐私评估，后者需要时间、来源和答案关系的证据。本篇不执行任何真实个人数据提取。研究报告应仅公开最小必要的匹配元数据和合法可展示片段，避免为了证明风险再次传播敏感材料。",
      [
        "arxiv-2104.08758",
        "arxiv-2107.06499"
      ]
    ],
    [
      "6. 一个可信审计应允许复核删除决定",
      "保存检测版本、阈值、匹配家族、复核结论和删除理由，抽样检查保留与删除两侧。评测同时使用较新、来源受控或私有但合法授权的独立样本，并固定评估协议。HELM强调多维评价与透明性，本任务中应据此同时报告污染检测局限和指标差异。没有命中不等于证明没有重叠，污染审计也不能由一个总体分数代替。",
      [
        "helm",
        "arxiv-2107.06499"
      ]
    ]
  ],
  "equations": [
    {
      "tex": "A_{\\mathrm{mix}}=(1-c)A_0+cA_1",
      "explanation": "分层加权恒等式；图示A0=0.5、A1=0.99均为假设，不是模型测量。"
    },
    {
      "tex": "\\Delta A=c(A_1-A_0)",
      "explanation": "只有在所列分层定义成立时，表示混合分数相对于A0的差异，不能据此反推真实污染率。"
    },
    {
      "tex": "\\mathrm{precision}=\\frac{TP}{TP+FP},\\qquad\\mathrm{recall}=\\frac{TP}{TP+FN}",
      "explanation": "匹配检测器需以人工标注或受控注入样本估计误报漏报，并说明抽样方式。"
    }
  ],
  "table": {
    "headers": [
      "方法或判断",
      "控制变量与测量",
      "失效条件及解释"
    ],
    "rows": [
      [
        "精确重复",
        "保留规范化规则与哈希",
        "漏掉翻译和改写"
      ],
      [
        "近似匹配",
        "分层抽查精确率和召回率",
        "模板相似可能造成误删"
      ],
      [
        "家族划分",
        "先聚合同源变体再划分",
        "逐条随机划分可能泄漏"
      ],
      [
        "污染分层评估",
        "列出排除前后分数与样本数",
        "未命中不能证明无污染"
      ]
    ]
  },
  "protocol": [
    "定义去重与污染的不同目标，固定检测单位及政策。",
    "记录原文哈希、来源时间、规范化和检测器版本。",
    "先按文档/题目/生成家族划分，再执行精确与近似匹配。",
    "对保留和删除两侧分层抽样，估计误报、漏报及不确定性。",
    "报告污染分层与排除前后指标，使用独立受控测试集复核。",
    "发布最小必要审计元数据，保留合法复核路径并避免泄露私人材料。"
  ],
  "diagram": [
    "来源与时间证据",
    "原文 / 规范化哈希",
    "同源变体家族",
    "精确及近似匹配",
    "人工抽查与分层评估",
    "版本化删除和复核记录"
  ],
  "analytic": "contamination-mixture",
  "limitations": [
    "混合图不估计真实污染率。",
    "相似度不自动证明答案泄漏。",
    "检测可能同时误删和漏检。",
    "没有进行真实隐私内容提取。"
  ],
  "keywords": [
    "deduplication",
    "contamination",
    "evaluation"
  ],
  "peer_reviewed": false,
  "journal_publication": false,
  "disclosure": {
    "authorship": "AI辅助研究综述；Codex（AI）依据所列一手文献撰写并进行解析推导。",
    "figures": "概念封面与科学图解分开；科学图由公开公式和明示假设计算，原始论文结果均注明归属。",
    "review": "本站没有本篇独立同行评审或原创模型实验记录。"
  },
  "section_kinds": [
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review"
  ],
  "revision_note": "阅读器性能修订：新近 HEAD 返回 200 且 ETag 相同时也复用已读页面。正文、公式、来源和科学图的数据保持上一版本内容。",
  "sources": [
    {
      "id": "arxiv-2107.06499",
      "title": "Deduplicating Training Data Makes Language Models Better",
      "authors": ",  ,  ,  ,  ,  ,",
      "url": "https://arxiv.org/abs/2107.06499",
      "date": "2021-07-14",
      "accessed_at": "2026-10-21",
      "locator": "Abstract; 原文方法、理论和实验章节见作者原始论文链接",
      "note": "一手论文；原文研究归属原作者。本文的推导示例与建议实验方案另行标注，不冒充原论文或本站实测。"
    },
    {
      "id": "arxiv-2104.08758",
      "title": "Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus",
      "authors": ",  ,  ,  ,  ,  ,  ,",
      "url": "https://arxiv.org/abs/2104.08758",
      "date": "2021-04-18",
      "accessed_at": "2026-10-21",
      "locator": "Abstract; 原文方法、理论和实验章节见作者原始论文链接",
      "note": "一手论文；原文研究归属原作者。本文的推导示例与建议实验方案另行标注，不冒充原论文或本站实测。"
    },
    {
      "id": "helm",
      "title": "Holistic Evaluation of Language Models",
      "authors": "Liang et al.",
      "date": "2022-11-16",
      "url": "https://arxiv.org/abs/2211.09110v2",
      "locator": "v2 abstract; seven dimensions and evaluation coverage",
      "note": "HELM combines accuracy, calibration, robustness, fairness, bias, toxicity and efficiency where feasible; coverage and tradeoffs are explicit.",
      "accessed_at": "2026-10-15",
      "rights": "Original source remains under its own license; only attributed factual synthesis, no source figures or long excerpts."
    }
  ],
  "figures": {
    "mechanism": {
      "url": "/guides/assets/v5-mechanism-data-dedup-contamination-58df23d273c94b46.png",
      "kind": "conceptual_method_flow",
      "data": [
        "来源与时间证据",
        "原文 / 规范化哈希",
        "同源变体家族",
        "精确及近似匹配",
        "人工抽查与分层评估",
        "版本化删除和复核记录"
      ]
    },
    "analysis": {
      "url": "/guides/assets/v5-analysis-data-dedup-contamination-0e12fd7858762e10.png",
      "kind": "analytical_or_constructed_example",
      "data_url": "/guides/assets/v5-data-data-dedup-contamination-5141908e5c6a76a6.json",
      "assumptions": "Constructed clean accuracy0.5 and contaminated accuracy0.99; no inference about any real model or benchmark."
    }
  },
  "cover_provenance_url": "/guides/assets/v5-provenance-data-dedup-contamination-88389a26c6d7ca6e.json",
  "source_revision": "6251a250b27bb8b449f59b392f81abd355383770",
  "pdf": {
    "document_id": "de689f1f-fdf6-585f-b5f9-26dd35fe66bf",
    "version_id": "0dddfc90-6ed5-5852-916d-edf304243a48",
    "media_type": "application/pdf",
    "authentication_required": true,
    "download_audited": true
  },
  "presentation_change": "Complete version-bound translated reading views and selectable authenticated original/translated PDFs; original scientific text and evidence retained.",
  "source_locale": "zh",
  "source_version_id": "66795541-5d1e-555c-bfdb-e702e8d2eb24",
  "translations": [
    {
      "locale": "en",
      "translation_id": "28ce5550-e599-521b-b59a-894c56ba49d4",
      "translated_content_sha256": "2751d171b7d5f74f2086acbcaae8824744eb29c6985071dfe6e6175eb49fda82",
      "sha256": "18b73a0611eb811dd6fc7ad3333eda6f49cc038fb06fca0197fe79df85920719",
      "byte_length": 638239
    },
    {
      "locale": "es",
      "translation_id": "1f518925-09ff-55c8-99b6-9ce497a1cedc",
      "translated_content_sha256": "88ac5e54218d547451730348bd52ea0ee3fc102f1a5c28183fb879e22326b68e",
      "sha256": "98b49ed250d580c0f7f6d5ec51adacff58dc8c964e77895e544a14fd12b1817e",
      "byte_length": 645588
    },
    {
      "locale": "ja",
      "translation_id": "809be7d0-41c6-520d-a73e-784024e1c77b",
      "translated_content_sha256": "ebbad8b21c504e274ac55cc1e6bc01d53f3471d86e94696b5e9313202550e429",
      "sha256": "5e4e4c048b6a23572e6807bc78ff579d0ed6d257d8bd9298a1495848e5012cb7",
      "byte_length": 748859
    },
    {
      "locale": "ar",
      "translation_id": "cf8d1c04-0278-50bb-a860-6e1137b46ad4",
      "translated_content_sha256": "ac93216a151f674a8f49afe629bbb4dcca6ae68fe8a0d8736d6a6eb91b886623",
      "sha256": "00bd59ea58aeca2dafbdcfb5aa40adbd785332f8f189a12a7ecaf6d13007fcb3",
      "byte_length": 623320
    }
  ]
}