arXiv 2606.06574 introduces PoLar (Polarity-based Layer Skipping), a method that makes LLM layers dynamically "skippable" or "repeatable" at inference time. The result: 3B models can match 70B+ models on math reasoning by selectively skipping and repeating layers, with a 60+ point improvement on the MATH benchmark.
The "tunable layers" insight: not all layers in a Transformer are equally important for a given task. For math reasoning, the "middle layers" are more important than the "end layers"; for creative writing, the opposite. PoLar makes this explicit: each layer has a "polarity" score (high = important, low = redundant), and at inference, low-polarity layers can be skipped, and high-polarity layers can be repeated.
The technical details: PoLar is trained with a "polarity loss" that explicitly teaches the model which layers can be skipped or repeated. The polarity scores are learned end-to-end, and they generalize across tasks (e.g., a layer learned to be "important for math" is also "important for code"). At inference, the model uses a "dynamic schedule" that skips/repeats layers based on the task and the polarity scores.
The benchmark: on the MATH benchmark, a 3B PoLar model hits 71.2, matching a 70B baseline (70.8). The inference cost is 1.5× of the 3B model (vs 23× for the 70B), giving a 15× cost-quality Pareto improvement. On code and reasoning benchmarks, the improvement is similar (40-60 points).
The bigger takeaway: "dynamic depth" is the right approach for efficient inference. The "fixed depth" assumption is wasteful, and the "dynamic depth" approach can dramatically improve efficiency. For the industry, this means "efficient inference" will move from "quantization" and "pruning" to "dynamic depth" — and the next round of inference optimization will focus on "which layers to run, in what order, how many times."