[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-proquant-mllm-w4a8-edge-quantization":3,"topics-all":38,"news-related-9bdf46a1-30e1-47a8-bb3f-bc82bd0a25e1":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"9bdf46a1-30e1-47a8-bb3f-bc82bd0a25e1","4-bit多模态量化:视觉先崩,语言还在","多模态压到 4-bit,语言和视觉的崩溃并不同步。NeurIPS 2026 论文 ProQuant 指出,量化基线上语言基准还剩 47.33%,多模态理解基准却直接归零。作者用自重标注对齐数据分布,再靠渐进式块激活先救视觉层,W4A8 下 Qwen3-VL-2B 平均准确率 72.54,距全精度仅差 1.09 分。","边缘设备上跑多模态模型，4-bit 是绕不开的一关。但真正麻烦的不是算力，而是崩溃的顺序。NeurIPS 2026 收录的一篇论文 ProQuant 里有组反直觉的观测：在 Qwen3-VL-2B-Instruct 的 W4A8 量化基线上，HellaSwag 这个语言基准还留着 47.33% 的准确率，AI2D 这个多模态理解基准却已经归零。语言先验扛住了量化噪声，跨模态对齐没扛住。\n\n论文原文见 https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.08216 。\n\n## 崩掉的不是语言，是\"看图\"这件事\n\nProQuant 由香港科技大学、西北工业大学、中国电信人工智能研究院与清华大学团队合作完成，Yufei Xue 等八人署名，已被 NeurIPS 2026 接收。\n\n作者先做了个诊断。量化到低比特时，多模态大模型的两种能力一起退化，但退化程度完全不对称：骨干模型还留着可观的语言先验，多模态理解却彻底归零。论文的判断是，跨模态对齐对量化噪声远比模型内在的生成惯性更脆弱。\n\n这直接推翻了\"两种能力一起修\"的朴素做法。论文主张先解耦，把多模态理解排在语言建模前面恢复。\n\n## 两个不太符合直觉的做法\n\n做法之一叫自重标注（SRL）。开源多模态数据集的原始标注风格和待量化模型的生成分布对不上——GPT-4 式的描述更字面、更强调位置（\"在桌子中央\"），Qwen 系模型更整体、更带解释性（\"创意装饰的甜甜圈\"）。风格错配会带来监督不一致。于是作者干脆丢掉原标注，用待量化模型自己重新生成回答。\n\n结果有点反直觉：自标赢了。消融表里 W4A8 下 GPT-4 标 71.77、自标 72.54，同族 8B 模型标 72.35；W4A4 下差距拉开，GPT-4 只有 56.78，自标 59.83。同一个家族、更大更强的标注器也打不过自标。\n\n另一个是渐进式块激活（PBA）。训练不是所有层一起开量化，而是先给视觉敏感层开、其余保持全精度，之后再逐步把文本敏感层也纳进来。哪些层算\"视觉敏感\"，靠视觉敏感因子 τ 定位——它在每个 block 上算视觉 token 的平均梯度幅值占全部 token 的比例。论文观察到这个值在 Qwen3-VL-2B（28 层）的中段最高：视觉敏感层集中在中间，文本敏感层偏两头。\n\n消融数据支持这套调度：日志式调度 72.54，优于余弦的 71.34 和线性的 71.00；按 τ 排优先级拿到 72.54，好过按权重梯度范数排的 72.19，随机排只有 71.88。块粒度上，一个解码层算一个单元最好（72.54），粗到两个层合并反而掉到 70.28。\n\n## 表格里的数字\n\nQwen3-VL-2B-Instruct 走 W4A8，ProQuant 平均准确率 72.54，比同门 QAT 基线 LLM-QAT 的 65.94 高 6.60 分，而全精度 BF16 是 73.63——差 1.09 分。平滑类 PTQ 的 SmoothQuant 只有 56.21。\n\n真正的分水岭在 W4A4。此时 MBQ 崩到 6.37，LLM-QAT 撑住 27.29，ProQuant 是 59.83，BF16 是 73.63。同一档位，MBQ 的 MME 从 2029.18 掉到 0.00，ProQuant 还有 1652.63。\n\n放大到 Qwen3-VL-8B-Instruct，W4A8 下 ProQuant 79.80 对 MBQ 的 75.62，W4A4 下 66.93 对 5.76。纯权重量化也测了：W3g128 下 MBQ 53.21，ProQuant 68.01。\n\n不过 W8A8 这一档 ProQuant 并没有优势——Qwen3-VL-2B 上 MBQ 73.95，ProQuant 72.88，PTQ 反而略高。论文的解释是这个位宽下强 PTQ 已经够用，QAT 的额外训练成本不划算。\n\n## 硬件上的账\n\n论文在一颗 TSMC 22nm 工艺的 ASIC 上做评估，片上延迟和平均功耗实测，片外 DRAM 用仿真，另配端到端周期精确加速器模型。三个骨干上最高测到 3.64 倍 prefill 加速和 5.64 倍能耗下降。拆到 Qwen3-VL-2B：W4A8 下 decode 延迟加速 3.74–3.91 倍、能耗降 3.76–3.93 倍；prefill 加速 1.58–1.76 倍、能耗降 2.20–2.70 倍。\n\n作者也主动划了界：相对 W8A8，W4A8 和 W4A4 约 2 倍和 4 倍的峰值算力吞吐是硬件峰值比，不是端到端实测加速。\n\n## 一个需要说明的细节\n\n摘要写着代码可用，链接指向 github.com\u002Fyufeixue-ai\u002FProQuant。但截至发稿，那个仓库里只有 .gitignore、LICENSE 和一份 56 字节的 README，提交历史只有 2026 年 3 月 7 日的一条 \"Initial commit\"，实现代码还没放出来。NeurIPS 会议页上的旧版摘要写的还是\"接收后发布代码\"。想要复现的话，这一步得自己等。\n\n## 所以呢\n\n这篇论文的价值不在 ProQuant 这个名字，而在于它把\"多模态量化\"这件事的优先级重新排了一遍：崩溃有先后，修复也该有先后，先把视觉通路救回来，语言能力自然会跟上。\n\n至于自标数据打败 GPT-4 这件事，翻译到任何要微调的领域都是同一句话——监督信号和被优化模型的分布对齐，比标注器本身有多强更重要。边缘部署这条线上，量化方法会越来越多，但\"先救哪一块\"这个问题可能比具体算子更值钱。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.08216","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"b38c7ea6-14b7-46cd-9434-eaba4a8b9235","en","ProQuant: 4-bit Multimodal Models Break Vision First","ProQuant: at 4-bit, multimodal models lose vision before language. The language benchmark scores 47.33% while multimodal understanding hits zero.","Running a multimodal model on an edge device means 4-bit quantization is not optional. But the real difficulty is not compute — it is the order in which the model breaks. A NeurIPS 2026 paper called ProQuant reports a counterintuitive observation: on the W4A8 quantized baseline of Qwen3-VL-2B-Instruct, the HellaSwag language benchmark still holds a 47.33% accuracy, while the AI2D multimodal-understanding benchmark has dropped to zero. The language prior withstood the quantization noise; the cross-modal alignment did not.\n\nThe paper is at https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.08216 .\n\n## What breaks is not language — it is \"seeing\"\n\nProQuant is the work of a team spanning The Hong Kong University of Science and Technology, Northwestern Polytechnical University, the Institute of Artificial Intelligence (TeleAI) at China Telecom, and Tsinghua University, authored by Yufei Xue and seven colleagues, and accepted to NeurIPS 2026.\n\nThe authors start with a diagnosis. When a multimodal model is quantized to low bit-widths, two capabilities degrade at once — but not to the same degree. The backbone retains a substantial linguistic prior, while multimodal understanding collapses entirely. The paper's reading is that cross-modal alignment is far more fragile to quantization noise than the model's intrinsic generative inertia.\n\nThat directly overturns the naive plan of restoring both capabilities together. The paper argues for decoupling them, and for putting multimodal understanding ahead of language modeling in the recovery order.\n\n## Two moves that run against intuition\n\nOne of them is self-relabeling (SRL). The original labels in open multimodal datasets do not match the generative distribution of the model being quantized: GPT-4-style captions are more literal and position-aware (\"in the center of the table\"), while Qwen-series models produce captions that are more holistic and interpretive (\"creatively decorated donuts\"). That stylistic mismatch introduces inconsistent supervision. So the authors discard the original labels entirely and let the target model regenerate its own responses.\n\nThe result is counterintuitive: self-labeling wins. In the ablation table, under W4A8, GPT-4 labeling scores 71.77, self-labeling 72.54, and a same-family 8B model 72.35. Under W4A4 the gap opens: GPT-4 manages only 56.78, self-labeling reaches 59.83. A bigger, stronger annotator from the same family still loses to self-labeling.\n\nThe other is progressive block activation (PBA). Training does not switch quantization on across all layers at once. It enables it first for the vision-sensitive layers while the rest stay in full precision, and only later brings the text-sensitive layers in as well. Which layers count as \"vision-sensitive\" is settled by a vision-sensitivity factor τ, computed per block as the average gradient magnitude over vision tokens divided by that over all tokens. The paper observes this factor peaking in the middle of Qwen3-VL-2B (28 layers): vision-sensitive blocks concentrate in the middle, text-sensitive ones sit toward both ends.\n\nThe ablations support that schedule. A logarithmic scheduler scores 72.54, ahead of cosine at 71.34 and linear at 71.00. Prioritizing by τ reaches 72.54, better than prioritizing by weight-gradient norm at 72.19, while random ordering manages only 71.88. On block granularity, treating one decoding layer as a single unit works best (72.54); coarsening to two layers per unit drops to 70.28.\n\n## The numbers in the tables\n\nOn Qwen3-VL-2B-Instruct at W4A8, ProQuant reaches an average accuracy of 72.54 — 6.60 points above the same-family QAT baseline LLM-QAT at 65.94 — while full-precision BF16 sits at 73.63, a gap of 1.09 points. The smoothing-based PTQ method SmoothQuant manages only 56.21.\n\nThe real dividing line is W4A4. There MBQ collapses to 6.37, LLM-QAT holds 27.29, and ProQuant reaches 59.83 against BF16's 73.63. On the same setting, MBQ's MME falls from 2029.18 to 0.00, while ProQuant still posts 1652.63.\n\nScaling up to Qwen3-VL-8B-Instruct, W4A8 gives ProQuant 79.80 against MBQ's 75.62, and W4A4 gives 66.93 against 5.76. Weight-only quantization was measured too: under W3g128, MBQ scores 53.21 and ProQuant 68.01.\n\nAt W8A8, however, ProQuant has no advantage — on Qwen3-VL-2B, MBQ scores 73.95 and ProQuant 72.88, with the PTQ baseline slightly higher. The paper's explanation is that strong PTQ is already sufficient at that bit-width, making the extra training cost of QAT hard to justify.\n\n## The hardware accounting\n\nThe paper evaluates on a TSMC 22nm-process ASIC, with on-chip latency and average power measured, off-chip DRAM simulated, and an end-to-end cycle-accurate accelerator model alongside. Across three backbones it measures up to a 3.64x prefill speedup and 5.64x energy reduction. Broken down for Qwen3-VL-2B: at W4A8, decode latency improves by 3.74–3.91x with energy down 3.76–3.93x, while prefill speeds up 1.58–1.76x with energy down 2.20–2.70x.\n\nThe authors also draw a line themselves: relative to W8A8, the roughly 2x and 4x peak compute throughput for W4A8 and W4A4 are hardware peak ratios, not end-to-end measured speedups.\n\n## One detail worth flagging\n\nThe abstract says code is available, linking to github.com\u002Fyufeixue-ai\u002FProQuant. As of this writing, that repository contains only .gitignore, a LICENSE, and a 56-byte README; its commit history is a single \"Initial commit\" from March 7, 2026. The implementation has not been released. The earlier abstract on the NeurIPS conference page still reads \"code will be released upon acceptance.\" Anyone hoping to reproduce the results will have to wait.\n\n## So what\n\nThe value here is not the name ProQuant but the fact that the paper reorders the priorities of multimodal quantization. Collapse has a sequence, and so should repair — rescue the visual pathway first, and language capability follows on its own.\n\nThe finding that self-labeled data beats GPT-4 translates into every field that involves fine-tuning: aligning the supervision signal with the distribution of the model being optimized matters more than how strong the annotator is. Edge deployment will keep producing new quantization methods, but \"which piece do you rescue first\" may end up worth more than the specific operator.\n","proquant-mllm-w4a8-edge-quantization","2026-11-08T07:28:00Z","2026-11-08T07:27:50.228801Z","2026-11-08T07:27:50.228815Z",true,"agent",263,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"88a727c6-311a-4cf4-b5c3-98d4251d956a","语音蒸馏踩雷:分离任务从+6.6掉到-1.7","route-dmd-speech-distillation-separation-collapse","2026-11-08T05:05:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"2983ae2f-a24f-40ae-a348-6f2c74443b19","CoSTA压缩语音token:口语对话生成提速1.7倍","costa-compressed-speech-token-adaptation","2026-10-31T15:13:09+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"376751a1-de8e-4e5b-afdc-4e26d7574146","NVIDIA Long-WAM：机器人记住 19 秒，预测未来再出手","nvidia-long-wam-robot-context","2026-10-08T19:11:55+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"52e9955f-50fe-4ea1-98cc-a7a442d57b71","MilliVid：MIT 把「分层潜变量」装进视频扩散，长程一致性首次跑通长镜头","millivid-mit-hierarchical-latent-long","2026-06-11T02:00:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"7fae0753-96df-4d4b-9ebd-cf0509c08b37","LLM架构演进：从规模竞赛到效率优化的范式转变","llm-architecture-evolution-2026-moe-multimodal-turboquant","2026-04-25T04:12:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"a99b973c-9375-45a7-b2a5-123ac256b410","推理效率革命：LLM模型优化的新突破","llm-inference-efficiency-moe-quant-kv-cache-vllm","2026-04-25T02:05:00+00:00"]