[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-moe-neuron-specialization-rampr":3,"topics-all":38,"news-related-25191dae-d51d-4fdd-a23c-3f1a46563c8a":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"25191dae-d51d-4fdd-a23c-3f1a46563c8a","MoE剪枝别看专家看神经元:留一半,显存省36-43%","MoE模型的专家分工之争有了新答案:arXiv论文提出RAMP度量,发现领域分工真实存在于神经元层——同域神经元集合跨样本重叠58-74%,跨域仅15-24%,专家层把信号平均掉了。据此的RAMPr剪枝在六个模型上保留75%神经元可保住91-98%性能,保留一半时显存降36-43%、批处理预填充吞吐升17-39%。","MoE（混合专家）模型有个流传很广的直觉：每个专家各管一摊，代码专家管代码、医学专家管医学。但另一派研究一直唱反调：路由决策更多由 token 身份和隐状态几何决定，看不出领域专长。这个争论持续了好几年，arXiv 上一篇 55 页的新论文给出了调停答案：两派都没错，只是都看错了层——真正的领域分工不在专家那层，在神经元那层。\n\n## 神经元才是分工单位\n\n作者提出 RAMP（Routing-Aware Magnitude Profiling）度量：按神经元所在专家的路由系数加权，衡量每个神经元对计算的真实贡献。用覆盖对话、数学、医学、中文、代码的九组混合语料做校准，在 Qwen3-30B-A3B、Qwen3.5-35B-A3B、Qwen3.6-35B-A3B、GLM-4.7-Flash、Qwen1.5-MoE-A2.7B-Chat、OLMoE-1B-7B-0125-Instruct 六个模型上跑下来，结论非常整齐：选取前 6.25% 神经元时，同一领域的神经元集合跨样本重叠 58-74%；跨领域的重叠只有 15-24%。领域信号在神经元层清晰存在。\n\n分布方式更能说明问题：62.64-82.50% 的领域专属神经元集中在少数格外偏爱它们的专家里，这种富集与专家路由偏好正相关（中位 Spearman 相关系数 0.42-0.71）。这就解释了为什么一派能测出领域偏好、另一派测不出功能边界——领域神经元散布在多个专家中，专家级统计把信号平均掉了。\n\n## 训练动态:表面稳定,底层翻腾\n\n论文还追踪了训练全程的神经元重要性演化（OLMoE 完整训练轨迹，加上 OpenMoE 与 Instella 的分阶段组合）。预训练相邻 checkpoint 之间，专家级重要性排名已经趋于稳定，神经元重要性排名相关性却只有 0.842-0.939——底层一直在重排；进入后训练阶段，相关性升至 0.997-0.998，基本冻结。只盯专家层，会严重低估训练中实际发生的重组。\n\n## 剪枝的实用账本\n\n分析工具直接变成了压缩工具 RAMPr：不再按专家整体删除，而是跨专家挑神经元保留，分累积重要性版（RAMPr-L）与领域均衡版（RAMPr-D）。账本如下：保留 75% 神经元，六个模型平均保住原始性能的 91-98%；保留 50%，常驻显存降低 36-43%，批处理预填充吞吐反而提升 17-39%。在 GLM-4.7-Flash 上，RAMPr-D 的 11 任务宏平均成绩比专家剪枝基线 REAP 高 2.91 个百分点。论文附录注明，全部任务分数用 lm-eval 0.4.13 在 vLLM 0.29.0 上以 BF16 精度统一评测，原始模型与每个剪枝变体采用完全相同的设置。\n\n## 所以呢\n\n这篇论文对两类人直接有用。做部署的：显存是硬通货，砍一半神经元只掉个位数百分比的成绩，私有化部署和资源受限场景直接受益。做可解释性研究的：它把专家分工之争从口水仗变成可测量问题——粒度选错了，结论自然打架。顺带一提，这种训练全轨迹分析只在 OLMoE、OpenMoE、Instella 这类完全开源的 MoE 家族上做得出来，开放权重给社区的研究红利，这篇论文本身就是证据。下一个问题也许是：既然神经元才是功能单位，MoE 按专家划分容量的设计，是不是一开始就切错了粒度？\n\n论文: https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13975","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13975","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"d11f0044-8aef-487c-bebe-89ce4683a4a3","moe",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"972cc190-dba7-4962-8a10-66c5b2bc9c8e","en","Pruning MoE Models Neuron-by-Neuron Saves 36-43% GPU Memory","MoE specialization lives in neurons, not experts. RAMPr pruning keeps 91-98% performance at 75% retention; at 50%, GPU memory drops 36-43%.","Mixture-of-Experts (MoE) models carry a popular intuition: each expert handles its own turf — a code expert for code, a medical expert for medicine. A rival line of research keeps pushing back, arguing that routing decisions are driven mostly by token identity and hidden-state geometry, not domain expertise. A new 55-page arXiv paper offers a verdict that lets both sides keep their dignity: they were just looking at the wrong layer. The real domain organization lives not at the expert level but at the neuron level.\n\n## Neurons Are the Unit of Specialization\n\nThe authors propose Routing-Aware Magnitude Profiling (RAMP), which weighs each neuron's contribution magnitude by its expert's routing coefficient. Calibrating on nine groups of mixed-domain text spanning chat, mathematics, medicine, Chinese content and code, across six models — Qwen3-30B-A3B, Qwen3.5-35B-A3B, Qwen3.6-35B-A3B, GLM-4.7-Flash, Qwen1.5-MoE-A2.7B-Chat and OLMoE-1B-7B-0125-Instruct — the result is remarkably consistent: at a 6.25% selection rate, neuron sets from the same domain overlap 58-74% across samples, while cross-domain overlap is only 15-24%. The domain signal clearly exists — at the neuron level.\n\nThe distribution pattern is even more telling: 62.64-82.50% of domain-exclusive neurons reside in experts where they are disproportionately represented, and this enrichment correlates positively with expert-level routing preferences (median Spearman correlations of 0.42-0.71). This is exactly why one camp measures domain preferences while the other finds no functional boundaries: domain neurons are scattered across multiple experts, and expert-level statistics average the signal away.\n\n## Training Dynamics: Calm on Top, Churning Below\n\nThe paper also traces neuron importance across training — following the full OLMoE training trajectory, plus staged combinations of OpenMoE and Instella. Between adjacent pre-training checkpoints, expert-level importance rankings look stable, yet neuron-importance rank correlations sit at only 0.842-0.939: the layer below keeps reorganizing. During post-training, correlations rise to 0.997-0.998, essentially frozen. Watch only the expert level and you badly underestimate how much reorganization training actually does.\n\n## The Practical Ledger for Pruning\n\nThe analysis tool becomes a compression tool, RAMPr: instead of deleting whole experts, it selects neurons to retain across expert boundaries, in a cumulative-importance variant (RAMPr-L) and a domain-balanced variant (RAMPr-D). The ledger: at 75% neuron retention, six models preserve 91-98% of original full performance on average; at 50% retention, resident GPU memory drops 36-43% while batched prefill throughput improves 17-39%. On GLM-4.7-Flash, RAMPr-D beats the expert-pruning baseline REAP by 2.91 percentage points on an eleven-task macro average.\n\n## So What\n\nThis paper pays off for two audiences. For deployment folks: GPU memory is hard currency, and dropping half the neurons costs only single-digit percentages of score — private deployments and resource-constrained settings benefit immediately. For interpretability researchers: it turns the expert-specialization fight into a measurable question — pick the wrong granularity and the conclusions will simply clash. Worth noting: this kind of full-trajectory analysis is only possible on fully open MoE families like OLMoE, OpenMoE and Instella; the paper itself is evidence of the research dividend of open weights. The next question almost asks itself: if neurons are the functional unit, was partitioning capacity by experts the wrong granularity from day one?\n\nPaper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13975","moe-neuron-specialization-rampr","2026-11-11T15:30:00Z","2026-11-11T15:10:55.666560Z","2026-11-11T15:10:55.666567Z",true,"agent",204,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"f02185e7-5f4c-46c6-bb18-575fbf3b1c0a","摩根大通实测:五项前沿配方,四项真有用","jpmorgan-factorial-retest-llm-ingredient-gains","2026-11-19T17:25:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"9af27b13-b1a9-47e3-a143-fd8a6c1ee5a9","Gemma 4省下的37.5%,推理引擎还没兑现","gemma-4-kv-cache-k-equals-v-compact-row","2026-11-18T11:40:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"f828f611-b57a-4855-9246-3609c2495faa","老师折进自己:2.27B用22B token留住84%","concertina-folding-llm-teacher-weights","2026-11-16T08:50:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"f28966de-08b7-442e-ba3a-1d644131df26","工具输出不必是人话:清华潜token压缩最多省89%","coca-cola-latent-tool-context-compression","2026-11-07T23:05:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"9260ffaf-bab9-4444-8852-b2689df55ab3","MoE剪枝英语校准:立陶宛语掉21.4分,英语仅1.7","moe-pruning-english-calibration-language-erosion","2026-11-07T13:30:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"b8ca86ba-2401-441d-ba5c-06cb6ef09420","随机抽样几乎就够了:高误差风险再砍50.2%","bestrand-benchmark-compression-random-sampling","2026-11-06T15:40:00+00:00"]