For years, the public record on large language model architecture has been a black box. Teams behind DeepSeek, Qwen, and Kimi ship new models with structural changes attached, yet their public reports rarely say which ideas paid off, which were abandoned, and why. Academia faces the mirror problem: limited compute and inconsistent baselines keep evaluation at small-scale, single-point comparisons, so it is hard to tell whether an improvement survives at larger scale.

A group of eleven authors from the Shanghai AI Laboratory, East China Normal University, Shanghai Jiao Tong University, and Fudan University decided to treat that record itself as infrastructure. Their paper introduces ArchSpace, a public platform for Recording, Reviewing, and Referencing LLM Architecture Research (arXiv:2611.17332, submitted 12 November 2026; only a v1 preprint so far, not yet peer reviewed).

A third of the papers, less than 10% of the citations

The paper opens with its own sampling survey. It covers language-model architecture studies from November 2023 to October 2026, including multimodal models, with no minimum parameter count; the authors searched titles and abstracts across all arXiv categories, then used AI-assisted screening plus targeted full-text checks. Papers are counted once by publication date: the paper-count plot spans 2,919 papers, and the citation plot covers 2,859 papers with 117,629 cumulative citations, retrieved from Semantic Scholar on 31 October 2026.

The finding: academic work accounts for about a third of the papers but receives less than 10% of the citations. The paper attributes this to a double concentration of attention and compute — attention to architectural innovation sits mainly with industry, while academic evaluation is constrained by scale, making it hard to show that an improvement holds up when scaled.

Screen with scaling laws first, then spend on training

ArchSpace is not about shipping a model. It is about a public validation framework for architecture proposals, and it runs in two phases.

Phase one is the scaling ladder. A candidate architecture is run across five compute budgets: 10^18, 3x10^18, 10^19, 3x10^19, and 10^20 FLOPs. The bar is not being lower at some single point, but whether the compute-optimal loss curve lands on the Pareto frontier. The loss is the equal-weight average over the eleven domains of the original OLMo 3 validation set, and Megatron-LM is the reference convention for counting FLOPs. Hyperparameter search comes in three settings: reuse the baseline's compute-optimal settings, sweep model size and token count in an IsoFLOP sweep while reusing the baseline's learning rate and batch size, and additionally tune learning rate and batch size.

Phase two trains 1B models: 150B tokens of pretraining at 8K context, followed by 10B tokens of long-context mid-training at 64K. The data comes from the Dolma 3 Pool 150B sample and the Dolma 3 Longmino Mix (50B), tokenized with the OLMo 3 tokenizer (identifier: allenai/dolma2-tokenizer). Evaluation uses the OLMES suites: after pretraining, the Base Main suite tests general capabilities, including mathematical reasoning and question answering; after mid-training, RULER is the primary signal and HELMET the complementary one.

Loss extrapolation is accurate; the optimal learning rate is not

The fitted results split into two halves with a wide gap. The scaling laws combine a coupled base surface with a fixed-slope cosh learning-rate/batch-size term, and they fit the measured losses from both baseline ladders with a MAPE of 0.25% to 0.27%. Frozen and extrapolated to roughly ten times the largest swept budget, using the largest swept models plus new 32-layer models, the MAPE is 0.32%, and the best measured loss at each swept budget sits within 0.5% of the compute-optimal curve.

The other half is hyperparameters. The paper says plainly that locating the optimal learning rate is harder than predicting the loss. In Phase II, the Qwen3.5 pretraining run reaches about 130 TPP, against about 10 TPP in the longest ladder run at this model scale; at batch size 4M the predicted optimal learning rate is about 8e-4, but the selected recipe uses learning rate 3e-3 with batch size 4M, roughly 3.6 times the predicted optimum. The paper's response is pragmatic: use the predicted optima to initialize direct recipe searches, rather than treating them as the answer.

Long context, and failures given a price

After mid-training at 64K context, Qwen3.5's RULER score rises from 17.5 to 57.0 and its HELMET score from 23.3 to 39.1; OLMo3's rise from 2.7 to 36.1 and from 7.4 to 29.6. The recipe tuned for Qwen3.5 also yields large gains for OLMo3 when reused without tuning.

The other side of the platform is its repository. The code lives at github.com/InternLM/archspace under an Apache-2.0 license, created in July 2026, currently at 89 stars and 19 open issues, with its homepage at archspace.live. The README aligns the validation pipeline with Olmo 3: model sizes of 1B, 3B, and 8B; training stages covering pre-training, mid-training, and instruction tuning; roughly 6.2T tokens in total across the complete workflow; training and evaluation logs published on Weights & Biases; and weights in the ArchSpace-Collection set on Hugging Face. The README states its position directly: the project values negative results, and an architecture attempt that does not deliver the expected gain is still valuable when it has been clearly and rigorously validated.

The Shanghai AI Laboratory's own news page (published 13 August 2026) puts the idea more bluntly: not just releasing the final model parameters, but progressively publishing the experiment records, architectural trade-offs, and validation results from pretraining, moving open source from results to process. The same page notes that the platform visualizes the iteration lineage of all derived architectures on top of the classic Decoder-only Transformer, and that more than ten proposals are under open validation, including the Next Concept Prediction model (NCP), the Depth-Attention model, and the knowledge-and-reasoning decoupled architecture (Mobius). Proposals that pass review get joint appointment and internship collaboration opportunities from the lab.

So what

The most interesting part of this is not the scaling laws. It is the premise being accepted: in architecture research, the most expensive asset is the record of what failed, and those records are precisely the ones least likely to circulate. The platform's approach is to bind proposal, review, and reproduction record together, so a validated architecture becomes the next round's baseline and negative results become citable. The repository offers open-source contribution recognition for ideas adopted into the next-generation InternLM architecture, and joint appointments and internships for proposals that pass review — it wants to trade participation for career pathways rather than moral appeals.

It is also worth staying clear-eyed. The repository was created in July 2026 and sits at 89 stars and 19 open issues; the paper is still a preprint; and that sampling survey uses a methodology the authors defined themselves, with the period and inclusion criteria as their own calls, so its conclusions should be read as this group's account. The real question is not whether a platform exists, but whether academic compute and attention can genuinely shift this way. Open-sourcing model parameters has been achieved; open-sourcing the experimental process is still a long way off.