[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-intern-lumina-u2-multi-codebook-diffusion":3,"topics-all":38,"news-related-5a46f955-86f3-41b9-8e88-1b31e948469e":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"5a46f955-86f3-41b9-8e88-1b31e948469e","上海AI Lab开源统一视觉dLLM:激活参数仅1.4B","上海AI实验室开源 Intern Lumina U2:16.9B MoE 扩散模型仅激活 1.4B,统一图像、视频、3D 理解与图像生成。官方评测十个图像理解基准六个第一,PHV 采样让生图加速 4.45 倍,训练在 1024 颗昇腾 910B 完成,Apache 2.0 权重上架 Hugging Face。","统一模型的「两张皮」问题吵了两年:理解和生成各用一套表示与训练目标,共享骨干只是形式统一。上海 AI 实验室把 Intern Lumina U2 挂上 Hugging Face,交出一个彻底离散化的答案:图像、视频、3D 输入全部编成多码本离散 token,与文本在同一个优化空间里、用同一套掩码扩散目标训练(论文:https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.20818)。\n\n## 一个视觉位置,八个码本\n\n单码本离散 token 的老毛病是表示容量不够,细节和语义顾此失彼。U2 给每个视觉位置配 8 个互补码本(每本 4096 词表,FSQ 量化),视觉 tokenizer 用 Apple 开源的 AToken,把图像、视频、3D 编进同一个 4D 潜空间,训练全程冻结。骨干侧的处理很克制:输入端 8 个 code 的 embedding 拼接后投影成单个 token,序列长度不随码本数膨胀;输出端用一个轻量多码本 AR 头在位置内部按码本序依次预测,位置之间仍由扩散骨干并行精修。位置编码沿用骨干的 1D RoPE,空间结构靠分隔符标注。\n\n## 1.4B 激活的 MoE 底座\n\n骨干从 LLaDA2.0-mini 转来:总参 16.9B,每 token 只激活 1.4B;20 层里首层稠密,其余 19 层各带 256 个路由专家加 1 个共享专家,每 token 激活 8 个路由专家。为接视觉只加 0.65B 参数接口,文本能力直接继承。\n\n## 跑分:理解 6\u002F10 第一,生图逼近 Qwen-Image\n\n官方评测里,1.4B 激活的 U2 十个图像理解基准拿了六个第一,CharXiv 85.9、HallusionBench 64.6、MathVision 36.5、DynaMath 63.6 领先最明显;对比同骨干的 LLaDA2.0-Uni 十项里八项占优。视频理解三项基准全部第一(NVIDIA 版 Video-MME 56.7、MLVU 64.9,昇腾版 MVBench 61.8);3D MM-Vet 总分 50.5,超过 PointLLM、ShapeLLM 两个 3D 专用模型。生成侧,统一模型里 DPG-Bench 87.99、UniGenBench 86.11、TIIF-Bench 87.1 均为最佳,GenEval 89 追平最强——考虑到 Qwen-Image 每 token 激活参数量约是它的 14 倍,这个逼近相当能打。短板也直白:OneIG-Bench 文字渲染 20.11 落后 LLaDA2.0-Uni 的 31.23,样本多样性也偏低。\n\n## 生图 4.45 倍加速:置信度平台期提交\n\n扩散解码贵在反复精修。视觉置信度天然偏低——96 步里中位数只从 0.046 爬到 0.127,绝对阈值提交规则不适用,U2 改用时间信号:置信度进入平台期的位置提前提交,Halton 低差异序控制提交顺序,V-verify 漂移检测决定何时刷新缓存。叠加 CFG 合批、AR 头 KV 缓存、隐状态缓存,生产路径 159.9 秒的生图延迟压到 35.9 秒,端到端 4.45 倍加速,GenEval 还从 86.6 升到 89.3。\n\n## 1024 颗昇腾 910B 全栈,Apache 2.0 开源\n\n工程侧更值得记一笔:训练、推理、评测全栈在 1024 颗昇腾 910B 上完成适配,NVIDIA 侧只跑小预算参考实验做跨平台验证——昇腾是主训练平台而非备胎。代码以 Apache 2.0 开源在 GitHub,权重上架 Hugging Face。「首个全离散多码本扩散语言模型」还是团队自述,文字渲染短板也摆在那里,但这份把表示容量、任务统一性和国产算力栈一次交齐的开源,给统一模型路线提供了一个值得复现的参照系。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.20818","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7b67033c-19e6-4052-a626-e681bba64c7a","diffusion",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"7e89b5cc-57db-4f37-bc6d-28919a73931c","model-release",{"id":19,"name":20,"slug":20,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"53fc2af9-ca63-4b5e-b86c-168582bd10e9","en","Intern Lumina U2: A 1.4B-Activated Omni-Visual Diffusion LLM","Shanghai AI Lab's Intern Lumina U2: a 16.9B MoE diffusion LLM, 1.4B active params, unifying image, video, and 3D understanding with image generation.","The \"two skins\" problem of unified models has been debated for two years: visual understanding and image generation each use their own representation and training objective, so sharing a backbone is unification in name only. Shanghai AI Laboratory has put Intern Lumina U2 on Hugging Face with a fully discrete answer: image, video, and 3D inputs are all encoded into multi-codebook discrete tokens and trained with text in one optimization space under a single masked-diffusion objective (paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.20818).\n\n## One Visual Position, Eight Codebooks\n\nThe old weakness of single-codebook discrete tokens is limited representational capacity—each position compresses into one code, starving detail and semantics. U2 assigns each visual position 8 complementary codebooks (4096-entry vocab each, FSQ quantization). The visual tokenizer is Apple's open-source AToken, encoding images, videos, and 3D into a shared 4D latent space, kept frozen throughout training. The backbone handles this with restraint: at input, the 8 code embeddings are concatenated and projected into a single token, so sequence length does not grow with codebook count; at output, a lightweight multi-codebook AR head predicts codes sequentially within each position, while the diffusion backbone refines positions in parallel. Positional encoding stays with the backbone's native 1D RoPE, with spatial structure marked by structural delimiters.\n\n## A 1.4B-Activated MoE Base\n\nThe backbone converts from LLaDA2.0-mini: 16.9B total parameters, only 1.4B activated per token. Of its 20 Transformer layers, the first is dense; the remaining 19 are MoE layers with 256 routed experts plus one shared expert each, activating 8 routed experts per token. Only 0.65B parameters of visual interfaces were added, inheriting language capability without a second autoregressive-to-diffusion conversion.\n\n## Benchmarks: 6\u002F10 First in Understanding, Closing on Qwen-Image\n\nIn official evaluations, U2 with 1.4B activated ranks first on six of ten image-understanding benchmarks, with the clearest leads on CharXiv (85.9), HallusionBench (64.6), MathVision (36.5), and DynaMath (63.6); it beats same-backbone LLaDA2.0-Uni on eight of ten. It tops all three video benchmarks (NVIDIA run: Video-MME 56.7, MLVU 64.9; Ascend run: MVBench 61.8) and scores 50.5 total on 3D MM-Vet, beating the 3D specialists PointLLM and ShapeLLM. On generation, among unified models it posts the best DPG-Bench (87.99), UniGenBench (86.11), and TIIF-Bench (87.1), matching the best GenEval at 89—remarkable given Qwen-Image activates roughly 14x more parameters per token. The shortcoming is plain: OneIG-Bench text rendering at 20.11 trails LLaDA2.0-Uni's 31.23, and sample diversity runs low.\n\n## 4.45x Faster Image Generation via Confidence Plateaus\n\nDiffusion decoding is expensive because of iterative refinement. The team found visual confidence is naturally low—its median crawls only from 0.046 to 0.127 over 96 steps—so Fast-dLLM-style absolute-threshold commit rules do not apply. They switched to temporal signals: positions entering confidence plateaus commit early, Halton low-discrepancy ordering spreads commits evenly, and V-verify drift tests decide when to refresh the cache. Stacked with CFG batching, AR-head KV caching, and hidden-state caching, production-path image latency drops from 159.9s to 35.9s—a 4.45x end-to-end speedup—while GenEval actually rises from 86.6 to 89.3.\n\n## Full-Stack on 1,024 Ascend 910B, Apache 2.0\n\nThe engineering side deserves note: training, inference, and evaluation were fully adapted on 1,024 Ascend 910B NPUs, with a smaller-budget NVIDIA reference run for cross-platform validation—Ascend is the primary training platform, not a backup. Code is open-sourced on GitHub under Apache 2.0, weights on Hugging Face. \"First fully discrete multi-codebook diffusion language model\" remains the team's own claim, and the text-rendering gap is real—but this release, bundling representational capacity, task unification, and a domestic compute stack in one open package, gives the unified-model route a reference implementation worth reproducing.","intern-lumina-u2-multi-codebook-diffusion","2026-10-16T17:09:52Z","2026-10-16T17:09:57.956412Z","2026-10-16T17:09:57.956429Z",true,"agent",228,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"6f9e9f94-9dcc-4c6c-b254-6c5d0fe8ed37","京东开源 JoyAI-Video-Edit:16B 多模态扩散 Transformer 把视频编辑推进「边播边改」实时流时代","jd-joyai-video-edit-realtime-diffusion","2026-08-10T00:00:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"bcdc10bc-2f08-4c39-8ffa-e7e34041c112","京东开源 JoyAI-Video-Edit:用 16B 多模态扩散 Transformer 把视频编辑推进「边播边改」实时流时代","jd-joyai-video-edit-real-time-streaming","2026-08-05T03:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"35e9f682-7952-4f6a-a8e9-7f8bdd47e791","IBM 语音三件套更新:2B 旗舰 ASR 精度开源第二","granite-speech-5-collection-update","2026-10-22T21:08:48+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"d83c4910-6dce-4171-8a44-9f68c9f841af","Qwen-Audio-3.1-ASR开源:转写自带说话人","qwen-audio-3-1-asr-speaker-diarization","2026-10-22T17:13:29+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"bbbda7ff-b1c7-4142-b37f-2b3fd0547e02","Fibo 2.0 开源:生成编辑合一,登顶开源编辑榜","bria-fibo-2-0-unified-editing-generation","2026-10-21T21:08:11+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"bef7a773-36fb-406b-949d-8d728b97b242","Matrix-Game 4.0 开源:单卡 60FPS 的实时可玩游戏世界","matrix-game-4-0-open-source-60fps","2026-10-21T13:09:32+00:00"]