[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-ultrabert-encoder-dual-lm-heads":3,"topics-all":38,"news-related-5a5efb41-7037-46cb-b93e-6842f9ff2d62":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"5a5efb41-7037-46cb-b93e-6842f9ff2d62","UltraBERT:91.0 GLUE+47.7 BEIR,BERT 配方重审后 encoder 双榜登顶","dmis-lab 开源 UltraBERT 编码器，150M\u002F380M 双尺寸 Apache 2.0：70T token 消融显示编码器 1-2T 饱和，mask 率高低分别利好分类与检索。双 LM head 与级联蒸馏让单模型兼顾两端，380M 拿下 91.0 GLUE 和 47.7 BEIR。","当所有算力涌向 decoder 时，dmis-lab 做了件逆行的事：用累计超 70 万亿 token 的消融重审了 BERT 时代的编码器预训练配方——结论是老配方一直在错位烧卡。UltraBERT（150M\u002F380M，Apache 2.0，原生 32K 上下文）就此让单一编码器登顶两榜：380M 版 91.0 GLUE、47.7 BEIR。\n\n## 编码器的老问题\n\n编码器至今是性价比之王：RAG 管线用它做嵌入，FineWeb-Edu 靠它过滤质量，双向读一遍直接出向量，推理天然便宜。但顽疾是 GLUE 第一从来不是 BEIR 第一——架构早追上来了，训练配方还停在 10B-100B token 时代。\n\n## 70 万亿 token 换来三条规律\n\n团队在 150M\u002F380M\u002F1.3B 三档规模上同数据训练、同流程评测，累计消融超 70 万亿 token，最长单次 8.8T：\n\n其一，decoder 一直涨，编码器会饱和：GLUE 和 SQuAD 约 1-2T token 进平台期，BEIR 更早，训练 loss 却一路降到 8.8T，多烧的卡基本白烧。\n\n其二，mask 率分裂：低比率（约 5%）利好分类，高比率（约 50%）利好检索，与旧结论相反。\n\n其三，带剪枝的蒸馏只在低 mask 率提升分类，高比率反而损害检索。\n\n## 解法：一个编码器，两个 LM head\n\n混着训的直觉做法不行：均匀采样表现得像平均值，ratio 衰减拿不回低比率的分类质量（GLUE 88.8 对 89.5）。原因是两种 mask 率下输入嵌入几乎一样，LM head 要的形态却完全不同，共享即打架。\n\n解法是 LM head 解耦：batch 按 4:1 混入 5% 与 50% mask 序列，主体共享、各配独立 head。150M 编码器就此拿到 89.4 GLUE 与 45.1 BEIR：分类匹配最佳单比率（89.5），检索反超（45.1 对 45.0）。再叠加级联蒸馏（1.3B 教师→380M→150M，学生直接学教师）；5% head 吃 KL 损失，50% head 用标准 MLM。训练 2T token，尾部 100B 切 32K 长上下文，2.5% 指令份额让基座零样本跟 MCQA 提示。\n\n## 成绩单\n\n两个尺寸各拿下 10 项基准中的 9 项第一，提升最猛的是推理型检索（base 尺寸 RAR-b Code\u002FMath +7.3\u002F+8.3）。380M 版 91.0 GLUE、47.7 BEIR，作者称（to our knowledge）这是首个在分类与检索同时 SOTA 的编码器；passkey 测试在 32K 全长保持 99%。配套 UltraBERT-Embed-150M 经两阶段对比训练（约 5 亿弱监督对 + 570 万精选难负例），MTEB(eng, v2) 67.24，为 200M 以下稠密模型最高分。\n\n## 所以呢\n\n最值钱的不是榜单，而是算力分配地图：编码器 1-2T token 到天花板，decoder 的 scaling law 不能照搬，做嵌入、过滤、质检的生产线值得按饱和点重算预算。官方博客里还有一层信号：Jev 类决策模型（读一遍、从固定选项挑一个、不生成）正让编码器本行重新吃香，微软 Decision-1 也建在 Qwen3.5-9B 上。编码器复兴不是怀旧，是成本结构在说话：当生成越来越贵，「不生成」本身就是竞争力。\n\n参考：arXiv:2610.15028（huggingface.co\u002Fblog\u002Fdmis-lab\u002Fultrabert）","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.15028","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"7e89b5cc-57db-4f37-bc6d-28919a73931c","model-release",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"3a3f0197-e5ee-491f-9196-6e94d505ea89","en","UltraBERT: One Encoder Tops Both GLUE (91.0) and BEIR (47.7)","Open-source UltraBERT from dmis-lab: encoders saturate at 1-2T (70T-token ablations); dual LM heads take one model to 91.0 GLUE and 47.7 BEIR.","While the industry's compute chases ever-larger decoders, the dmis-lab team did something contrarian: they re-audited the BERT-era encoder pretraining recipe with ablations totaling more than 70 trillion training tokens. The verdict is uncomfortable — the old recipe has been burning GPUs in the wrong places. The result is UltraBERT (150M\u002F380M, Apache 2.0, native 32K context), the first single encoder to top both leaderboards: 91.0 GLUE and 47.7 BEIR for the 380M model.\n\n## The Old Encoder Problem: Single-Board Champions\n\nEncoders remain the workhorse of cost-effective text processing: RAG pipelines embed documents with them, and pretraining datasets like FineWeb-Edu were quality-filtered by one. An encoder reads input once, bidirectionally, and returns a vector or score — no token-by-token generation, hence cheap inference. The chronic flaw: the model ranking first on GLUE was never the one ranking first on BEIR, and no single model led both. Architecturally, encoders caught up over the past two years (RoPE, SwiGLU, alternating local\u002Fglobal attention), but the training recipe still followed conventions summarized in the 10B-100B token era.\n\n## Three Lessons Bought With 70 Trillion Tokens\n\nThe team trained encoders at 150M, 380M and 1.3B on the same data mixture, fine-tuning and evaluating every checkpoint identically. The ablations add up to more than 70T tokens, with the longest single run reaching 8.8T. Three findings:\n\n**First, decoders keep scaling; encoders saturate.** GLUE and SQuAD plateau around 1-2T tokens, BEIR even earlier, while training and validation loss keep dropping all the way to 8.8T. Loss falls, downstream does not — the extra compute buys almost nothing. Past saturation, changing the recipe beats adding tokens.\n\n**Second, the masking ratio is split.** Lower ratios (around 5%) benefit classification while higher ones (around 50%) benefit retrieval, contradicting prior work suggesting larger ratios for larger models.\n\n**Third, distillation is not universal.** Distillation with pruning improves classification at low masking ratios but degrades retrieval at high ones.\n\n## The Fix: One Encoder, Two LM Heads\n\nSince 5% and 50% each excel at different tasks, the obvious move is training on both. The team tried ratio decay and uniform per-batch sampling — both fail. Uniform sampling just behaves like its average, and decay never recovers low-ratio classification quality (88.8 vs 89.5 GLUE). Looking inside the model revealed why: input embeddings at 6% and 50% masking look nearly identical, but the LM heads need completely different shapes — sharing the output layer forces the two objectives to fight.\n\nSo the UltraBERT recipe is LM head decoupling: each batch mixes 5% and 50% masked sequences at 4:1, the encoder body is shared, and each ratio gets its own LM head. One 150M encoder then reaches 89.4 GLUE and 45.1 BEIR — matching the best single-ratio model on classification (89.5) while beating the best one on retrieval (45.0 vs 45.0 for the 50%-only model). Cascade distillation follows: the 1.3B teacher is width-pruned to 380M and distilled, then pruned to 150M and distilled again, with both students learning directly from the 1.3B teacher; the 5% head trains on the teacher's logits with a KL loss, the 50% head with standard MLM. Training runs on 2T tokens, the last 100B at 32K context, and a 2.5% slice of instruction data lets the base follow MCQA prompts zero-shot.\n\n## The Scoreboard: Nine of Ten Benchmarks\n\nIn the official evaluation, UltraBERT takes the top score on 9 of 10 benchmarks at both sizes, with the biggest jumps on reasoning-style retrieval: +7.3 and +8.3 on RAR-b Code and Math at base size. The 380M model hits 91.0 GLUE and 47.7 BEIR, which the authors describe as, to their knowledge, the first encoder to achieve state of the art on both classification and retrieval. On long-context reliability, the passkey test holds 99% across the full 32K window. The companion UltraBERT-Embed-150M, trained in two contrastive stages (about 500M weakly supervised text and code pairs, then 5.7M curated examples with mined hard negatives), scores 67.24 on MTEB(eng, v2) — the highest of any dense model under 200M parameters.\n\n## So What\n\nThe real value here is not the leaderboard but the compute-allocation map: encoders top out at 1-2T tokens, and decoder scaling laws do not transfer. Any production line still using encoders for embedding, filtering, or quality control should re-run the budget against the saturation point. A second signal sits in the team's own blog: decision models (the Jev style of \"read once, choose from fixed options, generate nothing\") are making the encoder's core skill fashionable again — connect that to Microsoft's Qwen3.5-9B-based Decision-1, and the encoder revival is not nostalgia but new cost structure talking. As generative models get more expensive, not generating is itself a competitive edge.\n\nReference: arXiv:2610.15028 (huggingface.co\u002Fblog\u002Fdmis-lab\u002Fultrabert)","ultrabert-encoder-dual-lm-heads","2026-10-14T17:10:59Z","2026-10-14T17:11:02.584176Z","2026-10-14T17:11:02.584184Z",true,"agent",95,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"4c4a2a9e-f69b-4985-bd42-97ab2ef4e2ac","Spark-X2.5-4B 开源:4B 跑 1M 上下文,22 项基准打 9B 级 Qwen3.5","spark-x2-5-4b-apache-open-source","2026-09-16T01:30:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"c745abb4-d608-4ea6-884b-5176d7134d71","IFM 开源 K2 Horizon 六模型：训练数据全放，7B 刷榜成绩 82 被自己砍到 70.6","ifm-k2-horizon-open-fleet-audit","2026-09-05T23:07:55+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"741bd34c-7134-4e8e-ab45-4f53dc576a6b","腾讯 Hy4 登顶 9 月开源榜:79.87 分超 Qwen3.8 Max,Anthropic 包揽总榜前三","tencent-hy4-tops-open-source-benchlm-september","2026-09-01T17:10:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"f9cf9f03-6aca-4d29-94d3-5c6acfeaf435","匿名模型 OX Alpha 短暂登顶 OpenRouter 编码榜:研究者推测底座指向智谱 GLM-5.x","ox-alpha-stealth-openrouter-glm-5-zhipu","2026-08-24T03:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"d4fa7e14-8fbd-4940-93a6-3dd6f0a3991d","DeepSeek V4 Pro 正式版：1.6T MoE，1M 上下文","deepseek-v4-pro-0813-ga-1m-context-moe","2026-08-13T02:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"c94766df-827e-4e4e-a006-b6639ec76722","DeepSeek V4-Flash-0731 转正观察:权重不动,后训练把 Agent 分数打到 V4-Pro 之上","deepseek-v4-flash-0731-agent-benchmark-official-aug2026","2026-08-01T02:00:00+00:00"]