Anyone who tunes LoRA has felt it: raising rank from 16 to 64 helps a little; beyond that, almost nothing moves. A paper submitted to arXiv on November 12 offers an uncomfortable explanation: the rank you configure and the rank your adapter actually uses are not the same thing. Tomohiro Sawada and Kartik Goyal at Georgia Tech measured that standard LoRA updates have an effective rank far below their configured rank — on Qwen2.5-0.5B with rank set to 896, the measured effective rank is only 361, roughly forty percent utilization.

Rank configured is not rank used

First, the setup: the main experiments run SFT on Qwen2.5-0.5B (base), adapting all MLP down_proj layers under a fixed one-million-token budget over a mixture of three domains — 45 percent GSM8K math, 10 percent WMT19 Czech–English translation, and 45 percent Aya Hausa instruction following, for roughly 500 optimizer steps.

The yardstick is the Roy–Vetterli effective rank: normalize the singular values of the update matrix into a spectrum and exponentiate its entropy. Only when the singular values are perfectly uniform does effective rank equal configured rank; the more energy concentrates on a few directions, the lower it goes. Utilization also falls as configured rank grows: in the same mixed-domain fine-tuning sweep, effective rank is 145 at r=256 (about 57 percent) and 265 at r=512 (about 52 percent). The shortfall is not uniform either — across the 24 adapted MLP down_proj layers, late layers often fall below early and middle layers, wasting more of their budget. One-line conclusion: configured rank is an unreliable proxy for usable capacity; raising r does not reliably deliver more adapter.

Turning effective rank into a training objective

The more interesting move turns this measurement into a regularizer. Optimizing the Roy–Vetterli effective rank directly requires SVD, so the paper substitutes the Rényi-2 effective rank as a surrogate: it can be evaluated without materializing the update matrix, from trace statistics alone — differentiable, stable, and SVD-free — and is added to the language-modeling loss with a weight lambda. The effect shows up directly in the numbers: at r=896, lambda of 0.01, 0.1, and 1.0 raise effective rank from 361 to 668, 888, and 896. A more engineering-flavored route is norm-bounded reparameterization (NB-LoRA style): enforcing an operator-norm cap and maximizing the Frobenius norm within it already raises effective rank from 341 to 504 and lowers Wiki perplexity from 21.1 to 20.1, capturing much of the gain.

Where the gains are, and the limits

The main experiments sweep ranks on Qwen2.5-0.5B, Qwen2.5-3B-Instruct, and Llama-3.2-3B-Instruct: with lambda=0.01, mean Wiki perplexity drops below unpenalized LoRA at every tested rank of at least 16, while mean training perplexity improves or increases by at most 0.02 in all but one comparison. Per rank, at r=128, 256, and 512, Wiki perplexity falls by 0.34, 0.53, and 1.04 with training perplexity unchanged within 0.07; in the r=896 comparison, lambda=0.01 and 0.1 also improve training perplexity from 4.77 to 4.38 and 4.53.

Two limits are worth remembering. First, maximum effective rank is not a performance optimum: at r=256, raising lambda from 0.01 to 0.1 lifts effective rank from 250.540 to 255.777 and improves Wiki from 14.145 to 13.975, but training perplexity worsens from 4.465 to 4.559. Second, timing matters: in a roughly 500-step run, applying the penalty from step 0 worsens training perplexity from 22.6 to 24.7, while starting at step 400 recovers baseline-level 22.4 and the retention gains remain. The single run at lambda=1.0 is more extreme: Wiki perplexity lands 5.5 points below the standard-LoRA mean and WMT Czech–English drops from 15.02 to 8.91, with training perplexity 4.69 still inside the baseline 4.77±0.09 run-to-run variation — but it is one run, so discount accordingly.

The baselines show that fitting fast and remembering are different things: PiSSA and LoRA-XS reach training perplexity 4.29 and 4.58 on the mixture (standard LoRA: 4.77), yet their Wiki perplexities, 21.6 and 21.4, are worse than standard LoRA's 19.6 — fitting the mixture well does not preserve old knowledge. The paper's singular-direction analysis adds a twist: every method that fits the mixture well, dense fine-tuning included, places most of its singular-value mass on so-called intruder dimensions; only underfitting methods (VeRA, TinyLoRA, SVF) avoid them. The question is not whether intruder dimensions exist, but where the dominant directions spend their capacity.

So what

For anyone tuning LoRA daily, the lesson is direct: stop treating rank as a blind capacity knob and check utilization first. Effective rank can be estimated cheaply from trace statistics, and high-rank adapters are the most worth auditing. If you add the penalty, start at lambda=0.01 and activate it late rather than cranking effective rank to the ceiling. Keep the paper's own caveats in mind: benefits depend on the metric and execution, lambda still requires task-specific tuning, and the evidence is small-scale — mostly Qwen2.5-0.5B, a one-million-token budget, a single A40 GPU — and not yet peer-reviewed, so whether it holds on larger models awaits replication.

Paper: https://arxiv.org/abs/2611.17191
Code: https://github.com/tomohiro-sawada/lora-capacity-utilization