[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-lora-effective-rank-capacity-utilization":3,"topics-all":35,"news-related-b602ac1b-9734-4b80-bb3c-16f19ae493a0":64},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":21,"news_slug":28,"published_at":29,"created_at":30,"modified_at":31,"is_published":32,"publish_type":33,"image_url":14,"view_count":34},"b602ac1b-9734-4b80-bb3c-16f19ae493a0","LoRA 的秩在虚标:配了 896,只用到 361","佐治亚理工团队实测:标准 LoRA 适配器的有效秩远低于配置值,r=896 只实现 361,配置越大利用率越低。论文提出免 SVD 的 Rényi-2 有效秩正则,λ=0.01 在三个模型所有不低于 16 的秩档位上降低 Wiki 困惑度,训练困惑度最多涨 0.02,把有效秩从诊断指标变成可训练目标,代码已开源。","调 LoRA 的人大多有个体感:rank 从 16 拉到 64 还有点用,再往上加就几乎不动了。11 月 12 日提交到 arXiv 的一篇论文给了一个不太舒服的解释:你配置的秩,和适配器实际用到的秩,根本不是一回事。佐治亚理工的 Tomohiro Sawada 与 Kartik Goyal 实测发现,标准 LoRA 训出来的更新矩阵,有效秩远低于配置秩——在 Qwen2.5-0.5B 上把 rank 配到 896,测出的有效秩只有 361,利用率约四成。\n\n## 秩配了,但没用上\n\n先交代实验设置:主体在 Qwen2.5-0.5B(base)上做 SFT,适配全部 MLP down_proj 层,固定 100 万 token 预算,数据是三个域的混合——45% GSM8K 数学题、10% WMT19 捷克语-英语翻译、45% Aya 豪萨语指令跟随,全程约 500 步。\n\n论文用的尺子是 Roy–Vetterli 有效秩:把更新矩阵的奇异值归一化成谱分布,对熵取指数。奇异值完全均匀时,有效秩才等于配置秩;能量越集中在少数方向,有效秩越低。按这个口径,利用率还会随配置秩增大而往下掉:同一套混合域微调里,r=256 时有效秩 145,约 57%;r=512 时 265,约 52%。这种「吃不饱」还不均匀——24 个被适配的 MLP down_proj 层里,靠后的层往往比前层、中层浪费得更狠。结论一句话:配置秩是「可用容量」的不可靠代理,加 r 不等于真拿到更多适配器。\n\n## 把有效秩从诊断变成训练目标\n\n更有意思的动作是把这个测量变成正则项。直接优化 Roy–Vetterli 有效秩绕不开 SVD,论文改用 Rényi-2 有效秩做代理:它不需要显式构造更新矩阵,从迹统计量就能算,可微、稳定、免 SVD,乘上系数 λ 加进语言建模损失即可。效果直接反映在数字上:r=896 时,λ 取 0.01、0.1、1.0 分别把有效秩从 361 抬到 668、888、896。另一条更工程化的路线是范数约束重参数化(NB-LoRA 式):仅约束范数就把有效秩从 341 提到 504,Wiki 困惑度从 21.1 降到 20.1,已经吃到大头收益。\n\n## 收益在哪,边界也在哪\n\n主实验在 Qwen2.5-0.5B、Qwen2.5-3B-Instruct、Llama-3.2-3B-Instruct 三个模型上扫秩:λ=0.01 在所有不低于 16 的秩档位上,把平均 Wiki 困惑度压到无正则 LoRA 以下;训练困惑度要么改善、要么最多涨 0.02(仅一例例外)。分档看,r=128、256、512 时 Wiki 困惑度分别降 0.34、0.53、1.04,训练困惑度变化不超过 0.07;r=896 的主对比里,λ=0.01 和 0.1 还把训练困惑度从 4.77 降到 4.38 和 4.53。\n\n两条边界同样值得记。其一,有效秩拉满不是最优:r=256 时 λ 从 0.01 加到 0.1,有效秩从 250.540 升到 255.777,Wiki 从 14.145 降到 13.975,但训练困惑度从 4.465 恶化到 4.559。其二,正则的时机很敏感:全程约 500 步的训练,从第 0 步就开罚,训练困惑度从 22.6 恶化到 24.7;拖到第 400 步才启动,则回到 22.4,遗忘上的收益还在。λ=1.0 的单次运行更极端:Wiki 困惑度比标准 LoRA 均值低 5.5,WMT 捷英困惑度从 15.02 降到 8.91,训练困惑度 4.69 仍在基线 4.77±0.09 的波动范围内——但只跑了这一次,参考价值要打折。\n\n对照组还说明「学得快」和「记得住」是两回事:PiSSA 与 LoRA-XS 把混合训练困惑度做到 4.29 和 4.58(标准 LoRA 是 4.77),但 Wiki 困惑度 21.6 和 21.4,比标准 LoRA 的 19.6 更遗忘——拟合适配得好,不等于旧知识保得住。论文的奇异方向分析再补一刀:凡是把混合域拟合好的方法(连稠密微调也算),大部分奇异值质量都落在所谓「入侵维度」上;只有欠拟合的 VeRA、TinyLoRA、SVF 避开了。所以问题从来不是「有没有入侵维度」,而是主导方向把容量用在了哪。\n\n## 所以呢\n\n对天天调 LoRA 的人,这篇的启示很直接:别把 rank 当容量旋钮盲拧,先体检利用率——有效秩用迹统计量就能廉价估算,大 rank 适配器尤其值得查。要上正则,λ 从 0.01 起步、罚项晚点开,比一味猛拉有效秩更稳。论文自己的打折也要落地:收益依赖指标与执行细节,λ 需要按任务调;实验主体是 0.5B 级模型、100 万 token 预算、单张 A40 的小尺度设置,而且是未经同行评审的预印本,更大模型上是否成立要看后续复现。\n\n论文:https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.17191\n代码:https:\u002F\u002Fgithub.com\u002Ftomohiro-sawada\u002Flora-capacity-utilization\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.17191","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[22],{"id":23,"lang":24,"title":25,"summary":26,"content":27},"3feee6e1-dbc2-4f28-888f-a80616a2894f","en","LoRA Rank Overstated: 896 Configured, 361 Realized","LoRA adapters use far less rank than configured: erank 361 at r=896. A SVD-free Renyi-2 penalty cuts Wiki perplexity at every rank of 16 or more.","Anyone who tunes LoRA has felt it: raising rank from 16 to 64 helps a little; beyond that, almost nothing moves. A paper submitted to arXiv on November 12 offers an uncomfortable explanation: the rank you configure and the rank your adapter actually uses are not the same thing. Tomohiro Sawada and Kartik Goyal at Georgia Tech measured that standard LoRA updates have an effective rank far below their configured rank — on Qwen2.5-0.5B with rank set to 896, the measured effective rank is only 361, roughly forty percent utilization.\n\n## Rank configured is not rank used\n\nFirst, the setup: the main experiments run SFT on Qwen2.5-0.5B (base), adapting all MLP down_proj layers under a fixed one-million-token budget over a mixture of three domains — 45 percent GSM8K math, 10 percent WMT19 Czech–English translation, and 45 percent Aya Hausa instruction following, for roughly 500 optimizer steps.\n\nThe yardstick is the Roy–Vetterli effective rank: normalize the singular values of the update matrix into a spectrum and exponentiate its entropy. Only when the singular values are perfectly uniform does effective rank equal configured rank; the more energy concentrates on a few directions, the lower it goes. Utilization also falls as configured rank grows: in the same mixed-domain fine-tuning sweep, effective rank is 145 at r=256 (about 57 percent) and 265 at r=512 (about 52 percent). The shortfall is not uniform either — across the 24 adapted MLP down_proj layers, late layers often fall below early and middle layers, wasting more of their budget. One-line conclusion: configured rank is an unreliable proxy for usable capacity; raising r does not reliably deliver more adapter.\n\n## Turning effective rank into a training objective\n\nThe more interesting move turns this measurement into a regularizer. Optimizing the Roy–Vetterli effective rank directly requires SVD, so the paper substitutes the Rényi-2 effective rank as a surrogate: it can be evaluated without materializing the update matrix, from trace statistics alone — differentiable, stable, and SVD-free — and is added to the language-modeling loss with a weight lambda. The effect shows up directly in the numbers: at r=896, lambda of 0.01, 0.1, and 1.0 raise effective rank from 361 to 668, 888, and 896. A more engineering-flavored route is norm-bounded reparameterization (NB-LoRA style): enforcing an operator-norm cap and maximizing the Frobenius norm within it already raises effective rank from 341 to 504 and lowers Wiki perplexity from 21.1 to 20.1, capturing much of the gain.\n\n## Where the gains are, and the limits\n\nThe main experiments sweep ranks on Qwen2.5-0.5B, Qwen2.5-3B-Instruct, and Llama-3.2-3B-Instruct: with lambda=0.01, mean Wiki perplexity drops below unpenalized LoRA at every tested rank of at least 16, while mean training perplexity improves or increases by at most 0.02 in all but one comparison. Per rank, at r=128, 256, and 512, Wiki perplexity falls by 0.34, 0.53, and 1.04 with training perplexity unchanged within 0.07; in the r=896 comparison, lambda=0.01 and 0.1 also improve training perplexity from 4.77 to 4.38 and 4.53.\n\nTwo limits are worth remembering. First, maximum effective rank is not a performance optimum: at r=256, raising lambda from 0.01 to 0.1 lifts effective rank from 250.540 to 255.777 and improves Wiki from 14.145 to 13.975, but training perplexity worsens from 4.465 to 4.559. Second, timing matters: in a roughly 500-step run, applying the penalty from step 0 worsens training perplexity from 22.6 to 24.7, while starting at step 400 recovers baseline-level 22.4 and the retention gains remain. The single run at lambda=1.0 is more extreme: Wiki perplexity lands 5.5 points below the standard-LoRA mean and WMT Czech–English drops from 15.02 to 8.91, with training perplexity 4.69 still inside the baseline 4.77±0.09 run-to-run variation — but it is one run, so discount accordingly.\n\nThe baselines show that fitting fast and remembering are different things: PiSSA and LoRA-XS reach training perplexity 4.29 and 4.58 on the mixture (standard LoRA: 4.77), yet their Wiki perplexities, 21.6 and 21.4, are worse than standard LoRA's 19.6 — fitting the mixture well does not preserve old knowledge. The paper's singular-direction analysis adds a twist: every method that fits the mixture well, dense fine-tuning included, places most of its singular-value mass on so-called intruder dimensions; only underfitting methods (VeRA, TinyLoRA, SVF) avoid them. The question is not whether intruder dimensions exist, but where the dominant directions spend their capacity.\n\n## So what\n\nFor anyone tuning LoRA daily, the lesson is direct: stop treating rank as a blind capacity knob and check utilization first. Effective rank can be estimated cheaply from trace statistics, and high-rank adapters are the most worth auditing. If you add the penalty, start at lambda=0.01 and activate it late rather than cranking effective rank to the ceiling. Keep the paper's own caveats in mind: benefits depend on the metric and execution, lambda still requires task-specific tuning, and the evidence is small-scale — mostly Qwen2.5-0.5B, a one-million-token budget, a single A40 GPU — and not yet peer-reviewed, so whether it holds on larger models awaits replication.\n\nPaper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.17191\nCode: https:\u002F\u002Fgithub.com\u002Ftomohiro-sawada\u002Flora-capacity-utilization\n","lora-effective-rank-capacity-utilization","2026-11-14T23:30:00Z","2026-11-14T23:10:12.140183Z","2026-11-14T23:10:12.140193Z",true,"agent",129,[36,46,55],{"slug":37,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":32,"created_at":44,"modified_at":45},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":47,"tag_slug":47,"title_zh":48,"title_en":49,"intro_zh":50,"intro_en":51,"id":52,"is_active":32,"created_at":53,"modified_at":54},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":56,"tag_slug":56,"title_zh":57,"title_en":58,"intro_zh":59,"intro_en":60,"id":61,"is_active":32,"created_at":62,"modified_at":63},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":65},[66,71,76,81,86,91],{"id":67,"title":68,"news_slug":69,"published_at":70},"c4623b85-37f5-435e-ae98-e67eb72eb1dc","别再逐篇问大模型:AI写程序当评委,每主题0.17美元起","execjudges-llm-written-program-judges-retrieval-eval","2026-11-20T01:40:00+00:00",{"id":72,"title":73,"news_slug":74,"published_at":75},"4544ed60-b5fb-41b6-9fee-dd2f59012547","隐状态转个方向:直答46到85,超CoT省九成token","cot-hidden-state-directions-direct-answers","2026-11-19T13:30:00+00:00",{"id":77,"title":78,"news_slug":79,"published_at":80},"0a4281e9-69b1-43f0-9e7b-78f0cd7ea4c2","医疗小模型回答砍72.8%,准确率反升3.2点","peca-conciseness-qlora-medical","2026-11-15T15:30:00+00:00",{"id":82,"title":83,"news_slug":84,"published_at":85},"79da91a4-039a-4014-9c4b-44d8dc1b5574","同一份语料两种结局:数学分从76.6崩到18.2","on-policy-mid-training-distillation-math-collapse","2026-11-12T11:40:00+00:00",{"id":87,"title":88,"news_slug":89,"published_at":90},"d30f7434-f5ad-487d-94b2-269c0dcdb382","LoRA反编译来了:读权重不行,得让模型跑起来","lora-skill-document-recovery","2026-11-09T19:11:39+00:00",{"id":92,"title":93,"news_slug":94,"published_at":95},"638eacdb-6796-40df-bfe1-c48bdf17efe5","剑桥拆解蒸馏:KL 方向才是主角,on-policy 不是","distillation-kl-direction-vs-rollout-policy","2026-10-04T19:15:00+00:00"]