[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-archspace-llm-architecture-experiment-records":3,"topics-all":38,"news-related-0e2940ed-7980-4c78-bdf8-ff58da0c7e5f":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"0e2940ed-7980-4c78-bdf8-ff58da0c7e5f","ArchSpace:把大模型架构的失败也变成公共资产","大模型架构的公开记录一直是黑箱：工业界发新模型很少交代哪些想法失败、为什么失败，学术界则困在算力与基线不统一里。上海AI实验室等机构的十一位作者提出 ArchSpace，用缩放阶梯和统一验证框架把架构实验的过程公开出来，并给出 Qwen3.5 与 OLMo3 两套基线。","大模型架构的公开记录长期是个黑箱。DeepSeek、Qwen、Kimi 这些团队每次发新模型都带着结构改动，但公开报告很少交代哪些想法跑通了、哪些被放弃、为什么放弃。学术界是另一面：算力有限、基线不统一，不少评测只能停在小规模、单点对比上，难以判断一个改动放大之后还成不成立。\n\n上海AI实验室联合华东师范大学、上海交通大学、复旦大学的十一位作者，把这件「记录」本身当成了基础设施。论文提出 ArchSpace——一个面向大模型架构研究的记录、评审与引用平台（[arXiv:2611.17332](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.17332)，2026 年 11 月 12 日提交，目前只有 v1 预印本，尚未同行评审）。\n\n## 学术侧拿着三分之一的数量，只拿到不到 10% 的引用\n\n论文先给了一份自制抽样统计。区间是 2023 年 11 月到 2026 年 10 月，覆盖语言模型架构研究（含多模态模型，不设参数下限）；作者在 arXiv 全部分类检索标题与摘要，再以 AI 辅助筛选加针对性全文核查；论文按发布日期计数，计数图涉及 2,919 篇论文，引用图对应 2,859 篇论文与 117,629 次累计引用，引用数取自 Semantic Scholar，2026 年 10 月 31 日获取。\n\n结论是：学术工作约占论文数量的三分之一，拿到的引用不到 10%。论文把原因归于注意力和算力的双重集中——架构创新的注意力主要在工业界，而学术评估受规模所限，难以证明一项改进能撑到更大规模。\n\n## 先用缩放定律筛，再花大钱训\n\nArchSpace 的重点不是发模型，而是给架构提案设计一套公共验证框架，分两个阶段。\n\n第一阶段叫缩放阶梯。候选架构要在五个算力预算上跑，分别是 10^18、3×10^18、10^19、3×10^19、10^20 FLOPs。判定标准不是某个点上更低，而是算力最优的 loss 曲线是否落在 Pareto 前沿上。loss 取 OLMo 3 验证集十一个领域的等权平均，FLOPs 口径以 Megatron-LM 为参照。超参搜索分三档：复用基线最优设置、只扫模型规模与 token 数并复用基线学习率和批大小、再连学习率与批大小一起调。\n\n第二阶段训练 1B 模型：先用 150B token、8K 上下文预训练，再做 10B token、64K 上下文的长上下文中训。数据用 Dolma 3 Pool 150B 采样与 Dolma 3 Longmino Mix 的 50B，分词器是 allenai\u002Fdolma2-tokenizer。评测走 OLMES 套件：预训练后测 Base Main（通用能力、数学推理与问答），中训后以 RULER 为主信号、HELMET 为补充。\n\n## 损失外推很准，最优学习率不准\n\n拟合结果分两半看，落差挺大。缩放定律用耦合基底面加固定斜率的 cosh 学习率-批大小项，两条基线阶梯的拟合 MAPE 是 0.25% 到 0.27%；冻结后外推到最大预算的约十倍（用最大档模型加新的 32 层模型），MAPE 0.32%，各扫描预算的实测最优损失落在计算最优曲线 0.5% 以内。\n\n另一半是超参。论文明确承认：预测损失比定位最优学习率容易得多。Phase II 里 Qwen3.5 的预训练跑到约 130 TPP，而同规模最长的阶梯训练只有约 10 TPP；批大小 4M 下预测的最优学习率约 8e-4，最终选定的配方却是学习率 3e-3、批大小 4M——约为预测最优值的 3.6 倍。论文的处理办法很务实：把预测出来的最优点当成直接搜索的起点，而不是答案。\n\n## 长上下文，与被明确定价的失败\n\n中训之后，64K 上下文下 Qwen3.5 的 RULER 从 17.5 升到 57.0，HELMET 从 23.3 升到 39.1；OLMo3 的 RULER 从 2.7 升到 36.1，HELMET 从 7.4 升到 29.6。为 Qwen3.5 调出来的配方不调直接复用到 OLMo3，也拿到了提升。\n\n平台另一面是仓库。代码开在 [github.com\u002FInternLM\u002Farchspace](https:\u002F\u002Fgithub.com\u002FInternLM\u002Farchspace)，Apache-2.0 协议，创建于 2026 年 7 月，目前 89 个 star、19 个开放 issue，主页是 [archspace.live](https:\u002F\u002Fwww.archspace.live\u002F)。仓库 README 把验证流程对齐 Olmo 3：模型尺寸覆盖 1B、3B、8B，阶段含预训练、中训与指令微调，全流程约 6.2T token，训练与评测日志发布在 Weights & Biases，权重放在 Hugging Face 的 ArchSpace-Collection 集合。README 里有一句立场很直白：他们重视负面结果，一次没能兑现预期的尝试只要被严格验证过，仍然有价值。\n\n上海AI实验室官方页面（[2026 年 8 月 13 日发布](https:\u002F\u002Fwww.shlab.org.cn\u002Fnews\u002F5444290)）把这件事讲得更直接：不再只是开放最终模型参数，还要逐步公开预训练过程中的实验记录、架构取舍和验证结果，把开源从结果推向过程。同一页提到，平台以经典 Decoder-only Transformer 为基底做架构演化可视化，下一个概念预测模型（NCP）、深度注意力模型（Depth-Attention）、知识与推理解耦架构（Mobius）等十余种提案正在开放验证，通过评议的提案实验室提供双聘与实习合作机会。\n\n## 所以呢\n\n这件事最值得看的不是缩放定律，而是它承认了一个前提：架构研究里最贵的资产是失败的记录，而这些记录恰恰最不外流。平台的做法是把提案、评审、复现记录绑在一起，让被验证过的架构成为下一轮的基线，让负面结果也能被引用。仓库给被下一代 InternLM 架构采用的创新点提供开源贡献认定，通过评议的提案给双聘和实习机会——它想用职业通道而不是道德呼吁来换参与。\n\n也要清醒：仓库创建于 2026 年 7 月，目前 89 个 star、19 个开放 issue；论文只是预印本；那份抽样统计是作者自制的方法，区间和分类标准都由他们定义，结论应当按「这组作者的口径」来读。\n\n真正的问题不是有没有平台，而是学术界的算力和注意力能不能真的往这边挪。开源模型参数已经做到了，开源实验过程还差得远。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.17332","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"3f82d4c7-6c37-4606-b5b0-2c2cf55af567","en","ArchSpace Makes LLM Architecture Failures Public Assets","An arXiv preprint: a platform records and reviews LLM architecture experiments, with shared Qwen3.5 and OLMo3 baselines so failures stay citable.","For years, the public record on large language model architecture has been a black box. Teams behind DeepSeek, Qwen, and Kimi ship new models with structural changes attached, yet their public reports rarely say which ideas paid off, which were abandoned, and why. Academia faces the mirror problem: limited compute and inconsistent baselines keep evaluation at small-scale, single-point comparisons, so it is hard to tell whether an improvement survives at larger scale.\n\nA group of eleven authors from the Shanghai AI Laboratory, East China Normal University, Shanghai Jiao Tong University, and Fudan University decided to treat that record itself as infrastructure. Their paper introduces ArchSpace, a public platform for Recording, Reviewing, and Referencing LLM Architecture Research ([arXiv:2611.17332](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.17332), submitted 12 November 2026; only a v1 preprint so far, not yet peer reviewed).\n\n## A third of the papers, less than 10% of the citations\n\nThe paper opens with its own sampling survey. It covers language-model architecture studies from November 2023 to October 2026, including multimodal models, with no minimum parameter count; the authors searched titles and abstracts across all arXiv categories, then used AI-assisted screening plus targeted full-text checks. Papers are counted once by publication date: the paper-count plot spans 2,919 papers, and the citation plot covers 2,859 papers with 117,629 cumulative citations, retrieved from Semantic Scholar on 31 October 2026.\n\nThe finding: academic work accounts for about a third of the papers but receives less than 10% of the citations. The paper attributes this to a double concentration of attention and compute — attention to architectural innovation sits mainly with industry, while academic evaluation is constrained by scale, making it hard to show that an improvement holds up when scaled.\n\n## Screen with scaling laws first, then spend on training\n\nArchSpace is not about shipping a model. It is about a public validation framework for architecture proposals, and it runs in two phases.\n\nPhase one is the scaling ladder. A candidate architecture is run across five compute budgets: 10^18, 3x10^18, 10^19, 3x10^19, and 10^20 FLOPs. The bar is not being lower at some single point, but whether the compute-optimal loss curve lands on the Pareto frontier. The loss is the equal-weight average over the eleven domains of the original OLMo 3 validation set, and Megatron-LM is the reference convention for counting FLOPs. Hyperparameter search comes in three settings: reuse the baseline's compute-optimal settings, sweep model size and token count in an IsoFLOP sweep while reusing the baseline's learning rate and batch size, and additionally tune learning rate and batch size.\n\nPhase two trains 1B models: 150B tokens of pretraining at 8K context, followed by 10B tokens of long-context mid-training at 64K. The data comes from the Dolma 3 Pool 150B sample and the Dolma 3 Longmino Mix (50B), tokenized with the OLMo 3 tokenizer (identifier: allenai\u002Fdolma2-tokenizer). Evaluation uses the OLMES suites: after pretraining, the Base Main suite tests general capabilities, including mathematical reasoning and question answering; after mid-training, RULER is the primary signal and HELMET the complementary one.\n\n## Loss extrapolation is accurate; the optimal learning rate is not\n\nThe fitted results split into two halves with a wide gap. The scaling laws combine a coupled base surface with a fixed-slope cosh learning-rate\u002Fbatch-size term, and they fit the measured losses from both baseline ladders with a MAPE of 0.25% to 0.27%. Frozen and extrapolated to roughly ten times the largest swept budget, using the largest swept models plus new 32-layer models, the MAPE is 0.32%, and the best measured loss at each swept budget sits within 0.5% of the compute-optimal curve.\n\nThe other half is hyperparameters. The paper says plainly that locating the optimal learning rate is harder than predicting the loss. In Phase II, the Qwen3.5 pretraining run reaches about 130 TPP, against about 10 TPP in the longest ladder run at this model scale; at batch size 4M the predicted optimal learning rate is about 8e-4, but the selected recipe uses learning rate 3e-3 with batch size 4M, roughly 3.6 times the predicted optimum. The paper's response is pragmatic: use the predicted optima to initialize direct recipe searches, rather than treating them as the answer.\n\n## Long context, and failures given a price\n\nAfter mid-training at 64K context, Qwen3.5's RULER score rises from 17.5 to 57.0 and its HELMET score from 23.3 to 39.1; OLMo3's rise from 2.7 to 36.1 and from 7.4 to 29.6. The recipe tuned for Qwen3.5 also yields large gains for OLMo3 when reused without tuning.\n\nThe other side of the platform is its repository. The code lives at [github.com\u002FInternLM\u002Farchspace](https:\u002F\u002Fgithub.com\u002FInternLM\u002Farchspace) under an Apache-2.0 license, created in July 2026, currently at 89 stars and 19 open issues, with its homepage at [archspace.live](https:\u002F\u002Fwww.archspace.live\u002F). The README aligns the validation pipeline with Olmo 3: model sizes of 1B, 3B, and 8B; training stages covering pre-training, mid-training, and instruction tuning; roughly 6.2T tokens in total across the complete workflow; training and evaluation logs published on Weights & Biases; and weights in the ArchSpace-Collection set on Hugging Face. The README states its position directly: the project values negative results, and an architecture attempt that does not deliver the expected gain is still valuable when it has been clearly and rigorously validated.\n\nThe Shanghai AI Laboratory's own news page ([published 13 August 2026](https:\u002F\u002Fwww.shlab.org.cn\u002Fnews\u002F5444290)) puts the idea more bluntly: not just releasing the final model parameters, but progressively publishing the experiment records, architectural trade-offs, and validation results from pretraining, moving open source from results to process. The same page notes that the platform visualizes the iteration lineage of all derived architectures on top of the classic Decoder-only Transformer, and that more than ten proposals are under open validation, including the Next Concept Prediction model (NCP), the Depth-Attention model, and the knowledge-and-reasoning decoupled architecture (Mobius). Proposals that pass review get joint appointment and internship collaboration opportunities from the lab.\n\n## So what\n\nThe most interesting part of this is not the scaling laws. It is the premise being accepted: in architecture research, the most expensive asset is the record of what failed, and those records are precisely the ones least likely to circulate. The platform's approach is to bind proposal, review, and reproduction record together, so a validated architecture becomes the next round's baseline and negative results become citable. The repository offers open-source contribution recognition for ideas adopted into the next-generation InternLM architecture, and joint appointments and internships for proposals that pass review — it wants to trade participation for career pathways rather than moral appeals.\n\nIt is also worth staying clear-eyed. The repository was created in July 2026 and sits at 89 stars and 19 open issues; the paper is still a preprint; and that sampling survey uses a methodology the authors defined themselves, with the period and inclusion criteria as their own calls, so its conclusions should be read as this group's account. The real question is not whether a platform exists, but whether academic compute and attention can genuinely shift this way. Open-sourcing model parameters has been achieved; open-sourcing the experimental process is still a long way off.","archspace-llm-architecture-experiment-records","2026-11-15T11:10:00Z","2026-11-15T11:15:54.152175Z","2026-11-15T11:15:54.152185Z",true,"agent",168,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"c86255d6-d477-44d1-9fed-b876263dda6e","LLM负号错误病灶找到了,却未必治得动","llm-sign-error-steerability","2026-11-08T17:10:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"97df8600-9c52-46e9-9650-90c6d2b198f1","ARLoop:循环模型的残差流改成注意力","arloop-attention-residuals-looped-transformer","2026-11-05T03:19:37+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"e40de2f9-9ee4-47f3-b1f1-7bf93a4870a9","一个动词翻转工具调用决策,LLM 内部向量现形","llm-tool-call-decision-vector","2026-10-09T13:10:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"80d2381c-82d3-4200-a481-3b859d3b3e77","余弦相似度测的同质化,其实是文本长度","embedding-cosine-length-artifact-homogenization-measure","2026-11-17T11:15:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"1dec93aa-e2a2-4b65-941c-6fa04d452175","Lattice横向潜变量:困惑度15.41降到14.40","latent-lattice-language-models-horizontal-latent-perplexity","2026-11-17T09:20:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"4cdb4267-a0c3-423d-8005-504778932ae9","40份技术报告审计:73个主张只有19个有对照","llm-technical-report-ablation-audit","2026-11-17T05:30:00+00:00"]