[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-boostlora-gradient-boosting-lora-residual":3,"topics-all":36,"news-related-810251ee-b8bf-4fef-a9d8-e167c22ae4c5":55},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"810251ee-b8bf-4fef-a9d8-e167c22ae4c5","BoostLoRA：梯度增强让低秩适配器「自我进化」，小参数也能有大表达","参数高效微调（PEFT）方法（如 LoRA、QLoRA）已成为在消费级 GPU 上微调大模型的标准手段。其核心思路是保持预训练模型权重冻结，仅训练少量适配器参数，从而在有限显存下实现大规模模型的定制化。然而，PEFT 领域长期存在一个隐性瓶颈：适配器的表达能力与其参数量高度绑定——当参数量极低时，适配器被「锁死」在固定的低秩子空间内，即使延长训练时间也无法突破这一上限。\n\n4月29日，一篇发表于 arXiv（2604.27308）的论文提出了 BoostLoRA，从梯度增强（gradient boosting）的角度重新审视这一问题。BoostLoRA 并非试图为适配器分配更大的秩，而是采用迭代训练+合并的策略，让极小参数的适配器通过不断吸收梯度残差来逐步扩展可表达的子空间范围。\n\n其核心机制可理解为：每一轮迭代中，系统训练一个极小的「基础适配器」，将其输出残差交给下一个适配器修正，再将两者加权合并。新适配器的训练目标不是逼近真实标签，而是补足上轮残差。这一过程与梯度下降中的动量累积有相似逻辑，但作用在适配器层。最终效果是：原本只能表达「A」的固定低秩子空间，在3-4轮迭代后可以逐步逼近「A+B+C」等更复杂的函数映射。\n\n实验表明，BoostLoRA 在多个基准测试中超越传统固定秩 LoRA，尤其在小参数 regime（秩≤4）下提升显著。这意味着未来可以在极低成本下实现更高质量的模型定制——一块 RTX 3090 即可微调原本需要 A100 的70B模型。\n\n对行业而言，BoostLoRA 的贡献不仅在于数值提升，更在于它揭示了 PEFT 方法中「训练时间」与「表达能力」之间被忽视的全新关系：适配器的表达上限并非由初始化时分配的秩决定，而取决于我们愿意以何种方式累积它。对于资源受限的研究者和独立开发者，这是一个值得关注的开源方向——代码已随论文公开，可从 GitHub 获取。PEFT 领域正在从「静态子空间」向「动态累积」演进，BoostLoRA 是这一趋势的最新信号。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2604.27308","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":18,"name":19,"slug":19,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":21,"name":22,"slug":22,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"bdabdeee-0979-4e8e-8c31-36b9bfe52f7f","en","BoostLoRA: gradient boosting makes small adapters evolve","Parameter-efficient fine-tuning (PEFT) methods (like LoRA, QLoRA) have become the standard for fine-tuning large models on consumer-grade GPUs. The core idea is to keep pretrained model weights frozen and only train a small number of adapter parameters, achieving customization of large models under limited VRAM. However, the PEFT field has long had a hidden bottleneck: the adapter's expressiveness is highly bound to its parameter count — when the parameter count is very low, the adapter is \"locked\" within a fixed low-rank subspace, unable to break through this ceiling even with extended training.\n\nOn April 29, a paper published on arXiv (2604.27308) proposed BoostLoRA, re-examining this problem from the perspective of gradient boosting. BoostLoRA doesn't try to assign a larger rank to the adapter, but uses an iterative training + merging strategy, letting extremely small-parameter adapters gradually expand the expressible subspace range by continuously absorbing gradient residuals.\n\nIts core mechanism can be understood as: in each iteration, the system trains a very small \"base adapter,\" hands its output residual to the next adapter for correction, then weighted-merges the two. The new adapter's training goal isn't to approximate the true label, but to make up for the previous round's residual. This process has similar logic to momentum accumulation in gradient descent, but acts at the adapter layer. The final effect: a fixed low-rank subspace that could originally only express \"A\" can gradually approach more complex function mappings like \"A+B+C\" after 3-4 iterations.\n\nExperiments show BoostLoRA outperforms traditional fixed-rank LoRA across multiple benchmarks, especially with significant improvement in small-parameter regimes (rank ≤ 4). This means higher-quality model customization can be achieved at extremely low cost in the future — a single RTX 3090 can fine-tune a 70B model that originally required an A100.\n\nFor the industry, BoostLoRA's contribution goes beyond numerical improvement; it reveals an overlooked new relationship between \"training time\" and \"expressiveness\" in PEFT methods: the adapter's expressiveness ceiling isn't determined by the rank assigned at initialization, but by how we're willing to accumulate it. For resource-constrained researchers and independent developers, this is an open-source direction worth attention — code is publicly available alongside the paper on GitHub. The PEFT field is evolving from \"static subspace\" to \"dynamic accumulation,\" and BoostLoRA is the latest signal of this trend.","boostlora-gradient-boosting-lora-residual","2026-05-01T05:10:00Z","2026-05-01T13:10:13.302869Z","2026-08-19T02:08:40.142862Z",true,"agent",244,[37,46],{"slug":38,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":33,"created_at":44,"modified_at":45},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":47,"tag_slug":47,"title_zh":48,"title_en":49,"intro_zh":50,"intro_en":51,"id":52,"is_active":33,"created_at":53,"modified_at":54},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":56},[57,62,67,72,77,82],{"id":58,"title":59,"news_slug":60,"published_at":61},"748a4486-34e7-4215-b515-7eb56b3258c5","TwELL：Sakana AI与NVIDIA联合提出稀疏LLM推理加速20%，解决GPU批处理落地难题","sakana-nvidia-twell-20pct-sparse-batch-gemm","2026-05-30T08:20:00+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"2fa66657-afbb-4f03-849a-f420f42cf2ab","Prompt Caching：LLM推理成本削减90%的隐藏利器","prompt-caching-90pct-token-cost","2026-05-26T01:10:00+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"73f4d31e-745a-4bba-8a0f-38e7564966de","Sakana AI 提出 99% 稀疏性Transformer：在前馈层动刀革新LLM效率","sakana-99pct-sparse-ffn-transformer","2026-05-16T19:04:00+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"137ce22e-389d-47fb-8219-42ca53d6e916","Qwen 3.6 27B 重磅更新：MTP 技术让本地推理提速 2.5 倍","qwen-3-6-27b-mtp-local-2-5x","2026-05-16T01:01:00+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"4556940e-6456-43ce-b9b4-a0a7fa7a5865","MIT 新方法：自适应草稿模型将推理 LLM 训练速度提升 2-3 倍","mit-adaptive-draft-speculative-train-2-3x","2026-05-15T02:05:00+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"e4fd45e9-e0fd-4839-973e-909a442ce5ff","DeepSeek V3.2稀疏注意力：如何将长上下文推理成本砍半","deepseek-v3-2-dsa-sparse-attention-50pct-cost-cut","2026-05-01T10:15:00+00:00"]