A new paper was posted on arXiv on July 20, with a one-line title: "Convolution for Large Language Models." The authors are from Huawei Noah's Ark Lab, Peking University, and other institutions. The core change is surprisingly small — they insert a kernel=3 depthwise convolution between the Q/K/V projection and the attention layer in Qwen3. Less than 0.01% extra parameters, but the average accuracy on 7 downstream benchmarks actually goes up.
Why is this worth talking about? Two reasons stand out.
First, self-attention does not encode locality. Transformer self-attention excels at global token interaction, but the relationship between neighboring tokens is essentially learned from data, which is inherently inefficient. This paper is like giving attention a CNN Nose before it does any global matching — sniff the local context first. 0.01% of parameters, stable gains across multiple data-budget scales — this means it is not an overfitting trick for any specific scale, but a general-purpose mechanism.
Second, it directly attacks the LLM repeater problem. A case study in the paper shows that after this convolutional pre-processing, when the same token appears twice in The cat sat on the mat, its embeddings are now significantly more differentiated than before — exactly the copy-paste issue that haunts long-context LLM output. Another concurrent arXiv paper, 2607.19345 (Copy Less, Ground More), tackles the same problem from the training-objective side using evidence-aware RL to suppress repetition. The two approaches are complementary — one is architecture, the other is training objective; one addresses the root cause, the other treats the symptom.
A deeper signal: over the past two years, LLM architectural innovation has quietly shifted from the scale-up route to squeezing efficiency out of structure. Exchanging 0.01% for an N% gain — this scalpel-style engineering aesthetic — is far more interesting than simply making the model 10x bigger. The next direction worth watching: Linear Attention / SSM + local convolution — let us see whether hybrid architectures can squeeze out a bit more performance.