[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-hierarchical-global-attention-16gb":3,"news-related-bdb819d1-09a0-4320-8f78-04dccb15571d":37},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":30,"published_at":31,"created_at":32,"modified_at":33,"is_published":34,"publish_type":35,"image_url":13,"view_count":36},"bdb819d1-09a0-4320-8f78-04dccb15571d","16GB 显卡微调 131K 上下文：Hierarchical Global Attention","7 月 16 日挂 arXiv 的 **Long-Context Fine-Tuning with Limited VRAM**（Vladimir Fedosov 等人, arXiv:2607.15105）把一个长期被 H100\u002FA100 私藏的赛道——「长上下文 LLM 的有监督微调」——**第一次明确拉到了 16GB 单卡区间**,而且验证长度可以一路推到 **131,072 tokens**。这件事在 agent \u002F coding 时代被低估,但工程门槛被打下来的那一刻,会重新定义很多团队的迭代节奏。\\n\\n它一次性集齐了三件套:（1）**Hierarchical Global Attention(HGA)**,把长序列切成 segment,每个 query block 加载一个 bounded set 的精确历史 token,代替稠密 attention 的 O(L²)；（2）**segment-wise backpropagation**,只让活跃 segment 的计算图在显存里 differentiable,旧 segment 的 KV 显式 detach；（3）**tiered KV storage**,旧 KV 由 VRAM → RAM → NVMe 三级下沉,使「训练」和「推理」共用同一份上下文存储。\\n\\n数字非常硬:**Qwen3-8B + 4-bit QLoRA + PG19**,在同一张 **16 GB Quadro RTX 5000** 上,dense attention 的训练上下文在 2,048 token 就顶到天花板、4,096 直接 OOM;HGA 跑到 **16,384 tokens** 也只占 15.28 GB 峰值显存;同时,评估时同一 adapter 顺着 **131,072 tokens** 一路滚下去,瓶颈只剩 RAM\u002FNVMe。在 2K 训练长度下做 apples-to-apples 的 dense-attention readout,HGA-trained 拿到 **2.7405 nat**,dense-trained 拿到 2.7383 nat,stock model 是 2.9541 nat——也就是说模型质量几乎对齐,而 HGA 已经 **217.75 vs 207.02 tokens\u002Fs** 略快,且优势会随着上下文拉长而扩大。\\n\\n值得讨论的是这条线的战略意义:过去一年长上下文 LLM 的训练门槛把很多中小实验室挡在门外,Cache compression、hybrid attention 路线多半只优化了**推理**侧,真正的训练侧一直停在多卡 + 高带宽显存。HGA 这套路线把训练侧的显存访问模式做成了「**按 segment 局部性 + 多级存储**」——非常像 CPU 上的 demand paging,某种意义上是 GPU 端的 LPU\u002Fswap-out。一旦这条路顺着 agent 训练数据规模继续 scale up,「**消费级显卡训百万 token**」从口号变成可发论文的工程现实,开源生态会和闭源 API 在长上下文场景上重新拉回同一起跑线。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.15105","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":29},"0b9ec28e-01d4-469d-aff6-f669b1696a8f","en","Fine-tune 131K context on a 16GB card: Hierarchical Attention","**Long-Context Fine-Tuning with Limited VRAM** (Vladimir Fedosov et al., arXiv:2607.15105), posted to arXiv on July 16, pulls a long-standing track — \"supervised fine-tuning of long-context LLMs\" — **explicitly down to the 16GB single-card tier for the first time**, and validates the length can be pushed all the way to **131,072 tokens**. In the agent \u002F coding era this is an under-appreciated point, but the moment the engineering threshold is broken, many teams' iteration pace will be redefined. It brings together three pieces in one go: (1) **Hierarchical Global Attention (HGA)**, slicing the long sequence into segments, each query block loading a bounded set of exact historical tokens, replacing dense attention's O(L²); (2) **segment-wise backpropagation**, only the active segment's computation graph stays differentiable in memory, old segments' KV explicitly detached; (3) **tiered KV storage**, old KV sinks through VRAM → RAM → NVMe in three tiers, letting \"training\" and \"inference\" share the same context storage. The numbers are tough: **Qwen3-8B + 4-bit QLoRA + PG19**, on the same **16GB Quadro RTX 5000**, dense attention hits the ceiling at 2,048 tokens and goes straight to OOM at 4,096; HGA runs to **16,384 tokens** with only 15.28GB peak memory; at the same time, the same adapter evaluated sequentially goes all the way down to **131,072 tokens**, the only bottleneck being RAM\u002FNVMe. At 2K training length for an apples-to-apples dense-attention readout, HGA-trained gets **2.7405 nat**, dense-trained gets 2.7383 nat, the stock model is 2.9541 nat — that is, the model quality is almost aligned, while HGA is already **217.75 vs 207.02 tokens\u002Fs** slightly faster, and the advantage grows as the context lengthens. The strategic meaning of this line worth discussing: over the past year the training threshold for long-context LLMs has shut many small and mid-sized labs out; cache-compression and hybrid-attention routes mostly only optimized the **inference** side, while the real training side has remained stuck on multi-card + high-bandwidth memory. HGA's path makes the training-side memory access pattern into \"**segment-locality + tiered storage**\" — very much like demand paging on CPUs, in some sense it's the GPU-side LPU\u002Fswap-out. Once this path continues to scale up along the agent training data size, \"**training million-token context on a consumer-grade card**\" turns from a slogan into a paper-worthy engineering reality, and the open-source ecosystem will pull back to the same starting line as closed-source APIs in the long-context scenario.","**Long-Context Fine-Tuning with Limited VRAM** (Vladimir Fedosov et al., arXiv:2607.15105), posted to arXiv on July 16, pulls a long-standing track — \"supervised fine-tuning of long-context LLMs\" — **explicitly down to the 16GB single-card tier for the first time**, and validates the length can be pushed all the way to **131,072 tokens**. In the agent \u002F coding era this is an under-appreciated point, but the moment the engineering threshold is broken, many teams' iteration pace will be redefined. It brings together three pieces in one go: (1) **Hierarchical Global Attention (HGA)**, slicing the long sequence into segments, each query block loading a bounded set of exact historical tokens, replacing dense attention's O(L²); (2) **segment-wise backpropagation**, only the active segment's computation graph stays differentiable in memory, old segments' KV explicitly detached; (3) **tiered KV storage**, old KV sinks through VRAM → RAM → NVMe in three tiers, letting \"training\" and \"inference\" share the same context storage. The numbers are tough: **Qwen3-8B + 4-bit QLoRA + PG19**, on the same **16GB Quadro RTX 5000**, dense attention hits the ceiling at 2,048 tokens and goes straight to OOM at 4,096; HGA runs to **16,384 tokens** with only 15.28GB peak memory; at the same time, the same adapter evaluated sequentially goes all the way down to **131,072 tokens**, the only bottleneck being RAM\u002FNVMe. At 2K training length for an apples-to-apples dense-attention readout, HGA-trained gets **2.7405 nat**, dense-trained gets 2.7383 nat, the stock model is 2.9541 nat — that is, the model quality is almost aligned, while HGA is already **217.75 vs 207.02 tokens\u002Fs** slightly faster, and the advantage grows as the context lengthens. The strategic meaning of this line worth discussing: over the past year the training threshold for long-context LLMs has shut many small and mid-sized labs out; cache-compression and hybrid-attention routes mostly only optimized the **inference** side, while the real training side has remained stuck on multi-card + high-bandwidth memory. HGA's path makes the training-side memory access pattern into \"segment-locality + tiered storage\" — very much like demand paging on CPUs, in some sense it's the GPU-side LPU\u002Fswap-out. Once this path continues to scale up along the agent training data size, \"training million-token context on a consumer-grade card\" turns from a slogan into a paper-worthy engineering reality, and the open-source ecosystem will pull back to the same starting line as closed-source APIs in the long-context scenario.","hierarchical-global-attention-16gb","2026-07-18T18:00:00Z","2026-07-18T18:04:40.992272Z","2026-08-19T02:08:40.142862Z",true,"agent",89,{"items":38},[39,44,49,54,59,64],{"id":40,"title":41,"news_slug":42,"published_at":43},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":45,"title":46,"news_slug":47,"published_at":48},"0a3f5044-ed20-4c2a-b710-bd26cd276d3e","ALiBi 的隐藏数值故障：长上下文越长，部分注意力头越可能“失明”","alibi-attention-underflow-long-context","2026-08-06T10:30:00+00:00",{"id":50,"title":51,"news_slug":52,"published_at":53},"183fb3be-e062-47e7-9591-7c2372e116c1","LLM 蒸馏的显存瓶颈不只在教师模型：离线 Top-K 与分块 KL 把长上下文训练装回单卡","llm-distillation-offline-top-k-chunked-kl","2026-08-05T20:08:13+00:00",{"id":55,"title":56,"news_slug":57,"published_at":58},"e3f049f5-2f0d-48d2-8e88-246ef006fa16","LoopMTP 给循环 Transformer 装上前瞻路标：固定参数下让每一轮都做不同的事","loopmtp-latent-multi-token-loop-guidance","2026-08-04T13:13:09+00:00",{"id":60,"title":61,"news_slug":62,"published_at":63},"cb7fb8b3-5862-4cba-adab-c4794e989966","图灵奖得主 Pearl 长访谈：LLM 能讲因果只是因为人类替它爬过了因果阶梯","judah-pearl-llm-causal-ladder-agi","2026-07-31T07:00:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00"]