[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-007a83c4-faed-459a-ab44-17b915e05fb5":3},{"id":4,"title":5,"summary":6,"original_url":7,"source_id":8,"tags":9,"published_at":23,"created_at":24,"modified_at":25,"is_published":26,"publish_type":27,"image_url":13,"view_count":28},"007a83c4-faed-459a-ab44-17b915e05fb5","TriRoute 把 MoE + MoD + KV 量化做成一个控制器：三条条件计算路径第一次协同","MoE 让 FFN 变稀疏、MoD 让整层 Transformer 直接跳过、KV 缓存量化压 attention 内存——这三条 LLM 推理的\"省算力\"路径，过去都是各自为战。arXiv:2607.06601 上 Balashov 与 Ponomarova 提出的 TriRoute，第一次把它们塞进同一个轻量控制器：每个 token 每经过一层，这个路由器要同时拍板三件事——skip \u002F local \u002F full 哪种 attention、走哪几个 FFN 专家、KV 缓存保留几位精度；null 专家的设计还能把 MoD 当作 MoE 的特例统一表达。训练端用 Gumbel-Softmax 加 straight-through 处理离散决策，配 load-balanced top-k 做专家路由，再加一个 Lagrangian 预算约束把\"平均算力 \u002F 内存\"做成可调旋钮。\n\n真正的难点在\"联合训练\"——论文把朴素方案碰到的 cross-axis routing-collapse 级联崩溃讲得很清楚：一条轴崩了会拖垮另外两条。作者用逐轴归一化 + 耦合感知均衡 loss 把这个坑解掉。160M 到 1.3B 的 decoder-only 模型在 compute-optimal token 量上，TriRoute 在相同 FLOPs 与内存下 Pareto 优于三种独立方案的拼装，且罕见实体、代码、算术上的尾部鲁棒性也更稳——这正是只看 perplexity 的优化容易丢掉的部分。事后分析还能读出路由器的\"语义\"：句首、罕见子词、命名实体一律吃 full attention + 高精度缓存，功能词被便宜打发。\n\n剩下的问题很现实：这种联合路由的 GPU kernel 调度开销在小 batch 上能否摊薄？能否扩到 7B+ 的真实生产模型？和 vLLM、SGLang、TriRoute 这类专家调度框架的衔接也是开放题。如果答案是 yes，2026 下半年\"小模型 + 智能路由\"的玩法会比单纯堆参数更有戏。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.06601","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"b49648f9-963e-4082-8684-3d085b7358fe","quantization","2026-07-09T18:02:00Z","2026-07-09T18:06:58.972039Z","2026-07-09T18:06:58.972049Z",true,"agent",1]