{
  "schema": "auturnal.deep-research-review.v1",
  "slug": "sparse-mixture-experts",
  "title": "稀疏专家模型的真实成本：路由、容量约束与分布式通信",
  "category": "模型架构与系统",
  "dek": "从token流量核算专家负载，解释总参数、激活参数与真实训练成本为何不能互换。",
  "english": "Sparse mixture-of-experts models allocate token computation through learned routing. This review separates total capacity, activated computation, expert overflow and communication. A deterministic load calculation compares uniform and skewed routing under a declared capacity factor; it is not a trained-router experiment. Evaluation must jointly report model quality, dropped or rerouted tokens, communication and device utilization.",
  "kind": "ai_assisted_research_guide",
  "author": "Codex（AI）",
  "original_language": "zh-Hans",
  "published_date": "2026-10-22",
  "edition": 5,
  "version": 6,
  "previous_version": "/guides/sparse-mixture-experts/v5/",
  "refs": [
    "arxiv-2101.03961",
    "arxiv-2006.16668",
    "attention"
  ],
  "sections": [
    [
      "摘要",
      "稀疏专家以按token选择部分专家的方式扩大参数容量，但选择本身引入了负载、容量和跨设备通信问题。本文分析路由权重、容量因子、辅助均衡目标与专家放置之间的关系，并提出将有效计算、丢弃或重路由、等待时间和质量联合记录的验证方案。配图以8个专家和1000个token进行确定性容量核算，所有负载均为显式构造，不是任何已训练模型的路由分布或性能测量。",
      [
        "arxiv-2101.03961",
        "arxiv-2006.16668",
        "attention"
      ]
    ],
    [
      "1. 总容量与每步激活计算是两个预算",
      "将一个稠密前馈层替换成多个专家后，每个token仅进入少数专家，可以在不同比例增加激活计算的情况下增加总参数。但其余注意力、路由、归一化和共享层仍需运行。总参数影响存储、检查点和优化器状态，激活参数更接近部分算术成本；两者都不能单独表示训练吞吐。对比时应明确层数、专家大小、top-k、共享专家以及稠密基线，防止把更多训练资源带来的收益完全归给稀疏机制。",
      [
        "arxiv-2101.03961",
        "arxiv-2006.16668"
      ]
    ],
    [
      "2. 路由把局部选择变成全局排队问题",
      "路由器为输入产生专家分数，再选出一个或多个专家并组合结果。某个token的分数很高，并不意味着对应专家在这个批次还有容量。容量通常按一个路由分组内的token总数、top-k和专家数确定；分组边界、设备划分与padding都会改变实际利用率。若大量token选择同一专家，会造成溢出或等待，其他专家却可能空闲。因此只给平均负载会掩盖热点，至少还应报告最大负载、尾部分位数和容量溢出数量。",
      [
        "arxiv-2101.03961"
      ]
    ],
    [
      "3. 均衡目标改善利用率但改变优化问题",
      "辅助均衡损失鼓励专家使用率与路由概率更加均匀，可以缓解少数专家长期拥塞；它同时也限制了路由器根据数据自由形成分工的方式。均衡系数、噪声、选择规则及训练阶段会影响折中。负载均匀不证明专家学习到了有用差异，也不证明质量最高。需要在相同数据和计算预算下扫描均衡系数，同时测任务质量、溢出和专家使用，观察收益是否来自减少丢弃，而非仅有路由统计上的变化。",
      [
        "arxiv-2101.03961",
        "arxiv-2006.16668"
      ]
    ],
    [
      "4. all-to-all通信可能吞掉算术节省",
      "专家分布到多张设备时，token表示需要按路由重排、发送、计算再返回。通信量与隐藏维度、精度、top-k及专家放置有关，且小消息、不同长度批次和设备间带宽会影响时延。算术操作数相同的两个部署，可能有完全不同的通信等待。应分别记录分发、专家计算、合并及同步时间，并报告实际拓扑和并行策略；单卡稀疏乘法结果不能直接代表多机训练或在线服务表现。",
      [
        "arxiv-2006.16668",
        "arxiv-2101.03961"
      ]
    ],
    [
      "5. 容量超限必须成为质量记录的一部分",
      "当专家满载时，实现可能丢弃该路径、只保留残差，或重路由到次选专家。不同策略改变了实际模型计算，可能对少数语种、罕见任务或长序列产生不均匀影响。总体丢弃率较低也可能隐藏特定群体高丢弃。评估时要按任务、长度和语言分层，并保存从候选路由到真正执行路由的差异。训练和推理若采用不同容量或选择算法，应视为不同运行条件，不能只比较相同权重文件。",
      [
        "arxiv-2101.03961"
      ]
    ],
    [
      "6. 成本结论应包含系统和质量的联合约束",
      "合适的比较问题是：在达到同一质量与服务目标时，稀疏方案需要多少训练计算、设备内存、通信和延迟。参数扩大不自动证明样本效率提高，稀疏激活也不自动证明总成本降低。本篇构造负载图只用于展示容量溢出如何出现；对真实路由器，应保留每层分组统计而不记录私有token正文，再用重复运行和设备级测量确认瓶颈。把可复核计数与系统实测分开，才能解释性能差异。",
      [
        "arxiv-2101.03961",
        "arxiv-2006.16668",
        "attention"
      ]
    ]
  ],
  "equations": [
    {
      "tex": "y(x)=\\sum_{e\\in\\operatorname{TopK}(g(x))}p_e(x)E_e(x)",
      "explanation": "专家输出按选择后的权重组合；归一化方式、容量溢出和共享路径由实现另外规定。"
    },
    {
      "tex": "C_e=\\left\\lceil c\\frac{Tk}{E}\\right\\rceil,\\qquad O=\\sum_e\\max(0,L_e-C_e)",
      "explanation": "T是同一路由组token数，k是每token派发次数，E是专家数，c是容量因子；O计算超过容量的派发，而非自动等于最终丢弃的token数。"
    },
    {
      "tex": "\\rho_{\\max}=\\frac{\\max_e L_e}{Tk/E}",
      "explanation": "用最繁忙专家相对于理想平均负载的比例衡量热点。平均负载完全相同的两个批次也可能有不同热点。"
    }
  ],
  "table": {
    "headers": [
      "方法或判断",
      "控制变量与测量",
      "失效条件及解释"
    ],
    "rows": [
      [
        "总参数",
        "分列全部专家、共享层与优化器状态",
        "不等于每token激活计算"
      ],
      [
        "路由均衡",
        "记录每专家负载与分层溢出",
        "均匀不保证专家有效分工"
      ],
      [
        "容量因子",
        "固定路由组与溢出处理规则",
        "增加容量也增加内存或padding"
      ],
      [
        "分布式吞吐",
        "测分发、计算、合并和尾延迟",
        "FLOPs无法刻画通信与等待"
      ]
    ]
  },
  "protocol": [
    "固定稠密基线、层结构、专家大小、top-k和训练数据，公开匹配预算规则。",
    "明确token路由组、容量因子、权重归一化和满载后的处理。",
    "保存各层每专家负载、最大/平均比与溢出，按语种和长度分层。",
    "扫描均衡系数和容量，联合比较质量及真正执行的token路径。",
    "记录设备拓扑、通信字节、padding比例、峰值内存和各阶段耗时。",
    "用真实服务批量及长度分布复测，区分本篇计数示意与生产吞吐。"
  ],
  "diagram": [
    "token及路由分组",
    "router分数 / top-k",
    "专家容量检查",
    "跨设备分发与排队",
    "专家计算 / 结果合并",
    "质量、溢出与系统联合评估"
  ],
  "analytic": "moe-routing",
  "limitations": [
    "示意负载没有来自已训练模型。",
    "容量公式依赖路由分组定义。",
    "总参数与激活计算不代表总费用。",
    "均衡、专业化和任务质量需联合实测。"
  ],
  "keywords": [
    "mixture-of-experts",
    "routing",
    "distributed-training"
  ],
  "peer_reviewed": false,
  "journal_publication": false,
  "disclosure": {
    "authorship": "AI辅助研究综述；Codex（AI）依据所列一手文献撰写并进行解析推导。",
    "figures": "概念封面与科学图解分开；科学图由公开公式和明示假设计算，原始论文结果均注明归属。",
    "review": "本站没有本篇独立同行评审或原创模型实验记录。"
  },
  "section_kinds": [
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review"
  ],
  "revision_note": "阅读器性能修订：新近 HEAD 返回 200 且 ETag 相同时也复用已读页面。正文、公式、来源和科学图的数据保持上一版本内容。",
  "sources": [
    {
      "id": "arxiv-2101.03961",
      "title": "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity",
      "authors": ",  ,",
      "url": "https://arxiv.org/abs/2101.03961",
      "date": "2021-01-11",
      "accessed_at": "2026-10-21",
      "locator": "Abstract; 原文方法、理论和实验章节见作者原始论文链接",
      "note": "一手论文；原文研究归属原作者。本文的推导示例与建议实验方案另行标注，不冒充原论文或本站实测。"
    },
    {
      "id": "arxiv-2006.16668",
      "title": "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding",
      "authors": ",  ,  ,  ,  ,  ,  ,  ,",
      "url": "https://arxiv.org/abs/2006.16668",
      "date": "2020-06-30",
      "accessed_at": "2026-10-21",
      "locator": "Abstract; 原文方法、理论和实验章节见作者原始论文链接",
      "note": "一手论文；原文研究归属原作者。本文的推导示例与建议实验方案另行标注，不冒充原论文或本站实测。"
    },
    {
      "id": "attention",
      "title": "Attention Is All You Need",
      "authors": "Vaswani et al.",
      "date": "2017-06-12",
      "url": "https://arxiv.org/html/1706.03762v7",
      "locator": "v7; sections 3-4; Table 2 (EN-DE only)",
      "note": "Transformer uses scaled dot-product attention; full attention has quadratic pair complexity. EN-DE big model 28.4 BLEU; no cross-version EN-FR claim.",
      "accessed_at": "2026-10-15",
      "rights": "Original source remains under its own license; only attributed factual synthesis, no source figures or long excerpts."
    }
  ],
  "figures": {
    "mechanism": {
      "url": "/guides/assets/v5-mechanism-sparse-mixture-experts-f55a3f4fb954294f.png",
      "kind": "conceptual_method_flow",
      "data": [
        "token及路由分组",
        "router分数 / top-k",
        "专家容量检查",
        "跨设备分发与排队",
        "专家计算 / 结果合并",
        "质量、溢出与系统联合评估"
      ]
    },
    "analysis": {
      "url": "/guides/assets/v5-analysis-sparse-mixture-experts-8aaa4a588715ca9f.png",
      "kind": "analytical_or_constructed_example",
      "data_url": "/guides/assets/v5-data-sparse-mixture-experts-b7bcc75000802a31.json",
      "assumptions": "8 experts; 1000 token assignments; top-k=1; capacity factor 1.25, rounded up to157. No routing model trained."
    }
  },
  "cover_provenance_url": "/guides/assets/v5-provenance-sparse-mixture-experts-31ad78c89acf0840.json",
  "source_revision": "6251a250b27bb8b449f59b392f81abd355383770",
  "pdf": {
    "document_id": "9106208c-8aff-56e5-829c-05b9014ae0a0",
    "version_id": "7ea2cf22-17e8-5f30-bc3e-a73e77f177a1",
    "media_type": "application/pdf",
    "authentication_required": true,
    "download_audited": true
  },
  "presentation_change": "Complete version-bound translated reading views and selectable authenticated original/translated PDFs; original scientific text and evidence retained.",
  "source_locale": "zh",
  "source_version_id": "f0b5ecb3-caa1-5614-aeb6-7b6c6a28fcf2",
  "translations": [
    {
      "locale": "en",
      "translation_id": "5bb0bdac-608c-5a14-9c0d-3db77c6270f5",
      "translated_content_sha256": "7160b6e6c9d308986dcb39bc80a9db2fff9a664b480f3c20dbbb4d7bfb7db418",
      "sha256": "99957a7fc159919f549b14724943db900e1ec6684df0ccbafc09f91acb910b9d",
      "byte_length": 664784
    },
    {
      "locale": "es",
      "translation_id": "58f84221-1988-582c-89df-b6b91f596f85",
      "translated_content_sha256": "48ead623007986d91dc238f16543752f9d172b408ecd30824ca1362975b7a620",
      "sha256": "cd18ff07fef97e583f90e8d760675b4906caec10b20000d64ec9c5aea2bd31b1",
      "byte_length": 670845
    },
    {
      "locale": "ja",
      "translation_id": "ae206e9a-f5d9-5c65-abe0-4d9124db36d2",
      "translated_content_sha256": "83c061e39f9bb409784e2374e1b55049b7970e1ed82502a8e400400276190d7b",
      "sha256": "fde3f95f69df5a8f3b58eb5f6d66a953e0501544eb220e08152902967fd22014",
      "byte_length": 780799
    },
    {
      "locale": "ar",
      "translation_id": "4f286112-bdf5-5925-9578-da89e76a10f1",
      "translated_content_sha256": "f762140de2b8f20b932a7761ac3a6a471fa66bd749c37a75f82566b1c5247fde",
      "sha256": "7e72690e8573835d1ca298b31b03b4aba70c80769475165f1ef3d0c052447f28",
      "byte_length": 651010
    }
  ]
}