Every model launch ships with a list of claimed improvements: better attention, a leaner optimizer, smarter routing. Each item was measured against its own baseline, though. Stack them in a different setting and how much survives? Biswa Sengupta of the LLM Suite team at JPMorgan Chase turned that question into a disciplined experiment; the paper hit arXiv on November 18 (https://arxiv.org/abs/2611.24540), with a front-page disclaimer that it is informational and not a research-department product.

One model, five switches

The baseline is an eight-layer, 51.5M-parameter decoder trained on FineWeb under a nominal 300M-token budget. Five switches sit on top: E swaps layers 2-8 for 32-expert top-3 routing, with 161.3M total and 51.6M active parameters; M hands the hidden weight matrices to Muon while the rest stay on AdamW; H turns layers 1-3 and 5-7 into Gated DeltaNet linear memory, keeping only layers 4 and 8 on global attention (with no positional encoding at all); S adds normalisation to both the input and the output of each sub-layer; G inserts a sigmoid gate on attention outputs while narrowing the feed-forward layers to keep active parameters matched.

The design is the highlight: a classical fractional factorial layout, 16 combinations, each repeated with a second random seed - 32 core runs, about 9.6B tokens. Learning rates were swept per optimizer family: peak 0.002 for AdamW and 0.016 for Muon, with five Newton-Schulz iterations per update. Configurations sharing a seed receive identical batches, checked by first-batch hashes, and the final loss is measured on 10,485,760 held-out tokens.

Four help, one does nothing on average

By the two-seed mean loss contrast (nats, negative favors the switch): routing E -0.0983, Muon -0.0506, Gated DeltaNet hybrid -0.0365, extra normalisation -0.0194, all beyond the ±0.0043 reference margin. The attention gate G lands at +0.0018, inside the band - no average benefit. Single-switch contrasts differ by at most 0.004 nats between seeds. The all-five configuration posts the lowest loss in the selected set at both seeds (3.6792 and 3.6750). Across the 15 contrasts, the two seeds correlate at 0.996, so the replication is stable.

Two plot twists

First, G. Added alone to the all-off baseline, it cuts seed-0 loss by 0.019 nats and looks effective; inside the selected combinations its repeated mean is +0.0018. The paper is candid: this does not prove the gate useless, because G narrowed the feed-forward layers to match parameters, and some other change may have eaten its benefit. The original gated-attention paper measured steady gains at 15B MoE and 1.7B dense scale over 3.5 trillion tokens (https://arxiv.org/abs/2505.06708) - the same component flips conclusion when the scale, budget and parameter bookkeeping change.

Second, the noise estimate. Lenth's pseudo-standard error implies a run standard deviation near 0.020 nats, under which only E, M and H separate from zero. The repeated-run estimate is 0.0036, about six times smaller, and under it S plus several interactions emerge. Signals initially filed as noise turn out to be real. Interactions matter too: the largest pairwise contrast MH/GSE reaches +0.0122 nats, about 0.017 with its margin, brushing the paper's own 0.02-nat practical bound.

The cold water

  • Scale: 51.5M active parameters, one corpus, a 1,024-token context, a 300M-token budget; training time, energy and inference speed are not matched - no extrapolation to frontier models.
  • E's gain bundles roughly three times the total capacity (161.3M vs 51.5M); it is not pure routing skill.
  • The resolution-V design aliases: the G contrast mixes in a four-switch effect; unaliasing everything would need all 32 combinations, and only 16 were run.
  • The validation set served both tuning and final comparisons, so these are development results, not a fresh test set.
  • Two numerical bugs were caught during development: an RMSNorm GPU kernel leaked future-token information that smaller-width tests missed, and a sigmoid router underflowed gradients on extreme logits. Pre-fix losses were invalid.

So what

On the optimizer line the independent evidence points the same way: Moonshot's Moonlight scaled Muon to 16B and reports roughly 2x computational efficiency in scaling-law experiments (https://arxiv.org/abs/2502.16982), and a separate team that benchmarked ten optimizers warns that rankings hinge on tuning and evaluation setup (https://arxiv.org/abs/2509.02046). The paper's own abstract lands on the process lesson: test promising components together, repeat the comparisons, then decide what to keep. The methodology matters more than any single number here: stop ticking items off paper checklists. Put candidate components into one implementation under one budget and repeat the comparison across seeds - a single switch on a single seed can be noise, aliasing, or a bug nobody has found yet.