[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-transformer-cnn-nose-0-01-percent":3,"news-related-d2844cbb-b70e-469c-95b8-71cee8d735a6":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"d2844cbb-b70e-469c-95b8-71cee8d735a6","给 Transformer 装上「CNN 鼻子」:用 0.01% 的参数量换 benchmark 普涨","7 月 20 日挂上 arXiv 的一篇新论文,标题就一句话:**「Convolution for Large Language Models」**。作者来自华为诺亚、北大等机构,核心改动小得让人意外——在 Qwen3 的 Q\u002FK\u002FV 投影之后、attention 之前,塞一个 kernel=3 的 depthwise 卷积。**不到 0.01% 的额外参数,7 个下游 benchmark 的平均准确率反而涨了。**\n\n为什么这件事值得说?核心有两点。\n\n**第一,Self-attention 不编码局部性。** Transformer 的 self-attention 强在「全局 token 交互」,但相邻 token 的局部关系基本靠数据学,效率天然不高。这篇论文相当于在 Q\u002FK\u002FV 之前给 attention 装一个「CNN 鼻子」,让它先闻一下相邻 token 的味道再去做全局匹配。0.01% 的参数、稳定的多档数据预算提升,这意味着它不是某个特定规模的过拟合解,而是一种通用机制。\n\n**第二,直接打在 LLM 的「复读机」痛点上。** 论文的 case study 显示,经过这个卷积预处理后,「The cat sat on the mat」中同一 token 第一次和第二次出现,表示差异比之前大得多——这正是 LLM 长文本里反复出现的「重复拷贝」问题。同期 arXiv 上另一篇工作 2607.19345 (Copy Less, Ground More) 用 evidence-aware RL 在训练目标侧压制重复,二者是**架构 vs 训练目标**的互补解法,一个治本,一个治标。\n\n更深一层的信号:过去两年 LLM 架构创新从「scale 路线」悄悄转向「在结构里抠效率」。0.01% 换 N% 提升,这种**「手术刀式」的工程美学**,比把模型做大 10 倍有意思多了。下一个值得蹲的方向:Linear Attention \u002F SSM + 局部卷积,看混合架构能不能再榨出一点性能。\n\n参考:\n- 论文: https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.18413\n- 互补工作: https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.19345","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.18413","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":28},"e80e8ca0-af2d-438f-a1c2-cf9c8ef48a5f","en","A CNN nose for Transformers: 0.01% params, broad gains","A new paper was posted on arXiv on July 20, with a one-line title: \"Convolution for Large Language Models.\" The authors are from Huawei Noah's Ark Lab, Peking University, and other institutions. The core change is surprisingly small — they insert a kernel=3 depthwise convolution between the Q\u002FK\u002FV projection and the attention layer in Qwen3. Less than 0.01% extra parameters, but the average accuracy on 7 downstream benchmarks actually goes up.\n\nWhy is this worth talking about? Two reasons stand out.\n\nFirst, self-attention does not encode locality. Transformer self-attention excels at global token interaction, but the relationship between neighboring tokens is essentially learned from data, which is inherently inefficient. This paper is like giving attention a CNN Nose before it does any global matching — sniff the local context first. 0.01% of parameters, stable gains across multiple data-budget scales — this means it is not an overfitting trick for any specific scale, but a general-purpose mechanism.\n\nSecond, it directly attacks the LLM repeater problem. A case study in the paper shows that after this convolutional pre-processing, when the same token appears twice in The cat sat on the mat, its embeddings are now significantly more differentiated than before — exactly the copy-paste issue that haunts long-context LLM output. Another concurrent arXiv paper, 2607.19345 (Copy Less, Ground More), tackles the same problem from the training-objective side using evidence-aware RL to suppress repetition. The two approaches are complementary — one is architecture, the other is training objective; one addresses the root cause, the other treats the symptom.\n\nA deeper signal: over the past two years, LLM architectural innovation has quietly shifted from the scale-up route to squeezing efficiency out of structure. Exchanging 0.01% for an N% gain — this scalpel-style engineering aesthetic — is far more interesting than simply making the model 10x bigger. The next direction worth watching: Linear Attention \u002F SSM + local convolution — let us see whether hybrid architectures can squeeze out a bit more performance.","transformer-cnn-nose-0-01-percent","2026-07-22T12:10:00Z","2026-07-22T12:06:07.165735Z","2026-08-19T02:08:40.142862Z",true,"agent",104,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"b5638cab-a4d6-44ac-9230-32ed0a4cba9d","ARMT 把「记忆」焊进 Transformer:用恒定显存换无限上下文","armt-associative-recurrent-memory-transformer","2026-07-23T00:00:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"067a3f68-9bc8-4486-9715-5a391e537909","ETH 统一评测循环线性注意力：Kimi Delta Attention 损失最低","eth-zurich-clvr-kda","2026-07-12T04:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"29774f38-c361-4dca-b11a-c14df2fc84d9","HiLS 把\"无限上下文\"从口号变成数学:让稀疏注意力首次跑赢 Full Attention","hils-hierarchical-landmark-sparse","2026-07-07T14:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"5c53c383-9727-4880-95e9-9fa752132b01","把混合注意力推到 head 级：HydraHead 用 7:1 LA\u002FFA 比实现 3:1 层混的长上下文性能","hydrahead-7-to-1-la-fa-head-mixed-attention","2026-06-20T16:14:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00"]