[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-pixelumm-encoder-free-unified-multimodal-8b":3,"topics-all":41,"news-related-03d88e2a-526c-4c39-b9b6-e90ac0ff65e0":60},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":27,"news_slug":34,"published_at":35,"created_at":36,"modified_at":37,"is_published":38,"publish_type":39,"image_url":14,"view_count":40},"03d88e2a-526c-4c39-b9b6-e90ac0ff65e0","PixelUMM 拆掉 VAE:NVIDIA 用 8B 单 Transformer 在像素空间统一图文视频","NVIDIA 9 月 29 日挂出的 PixelUMM(arXiv:2609.38597)拿掉 VAE 和视觉编码器,用 Qwen3-8B 单根 decoder-only Transformer 在像素空间统一图像和视频的理解\u002F生成,8B 比 1.7B 在生成 MSE 与文本 CE 上都用了约 1\u002F3 步数收敛。","事件背景\n\n统一多模态模型(UMM)长期有一个分叉——理解分支用 ViT 抽语义特征,生成分支用 VAE 编解码到潜空间,两条视觉接口并存,视觉上下文长度被双倍摊开,新工作每接进一个任务就得分别维护两条管线。pixel-space 直接建模这条思路在图像侧已经有人试过,但把视频的理解和生成也压到同一根像素管道上,过去统一还没有人先动手。\n\nNVIDIA 在 9 月 29 日挂出的论文 PixelUMM(arXiv:2609.38597,GitHub: nv-tlabs\u002FPixelUMM,模型权重同步开源在 Hugging Face nvidia\u002FPixelUMM)正面回应了这件事。论文作者包括 Cong Wei、Xuanchi Ren、Bryan Chu、Weiming Ren、Huan Ling、Jiahui Huang、Laura Leal-Taixé、Sanja Fidler、Wenhu Chen、Zian Wang、Jay Zhangjie Wu。\n\n## 核心思路\n\nPixelUMM 没有 VAE,也没有视觉编码器——一根 decoder-only Transformer 同时读写原始像素:图像按 16×16 patch 切,视频按 4 帧一组切成时空管(tubelet),二者都经单层线性投影接到共享的多模态主干,主干是 Qwen3-8B,理解专家与生成专家复用同一组 self-attention。模型内部用 Mixture-of-Transformers 拆任务参数,同时支持自回归文本预测与 pixel-space flow matching,既不出 VAE token 流,也不出单独的理解分支。\n\n更关键的是这套架构把 clean pixel、noisy pixel 和文本一起塞进同一个 self-attention 窗口,对应论文里的 F7 试验:15K 步多任务微调后,图像理解在 BLINK(+2.37)、CV-Bench(+2.27)、SEED-I(-5.41)双向变动并不剧烈,四项视频理解基准 0.89–3.49 分提高,说明干净像素条件从理解专家走不会系统性损伤理解本身——这是把 VAE 砍掉之后最容易被质疑的一点。\n\n## 成绩单\n\nPixelUMM 给出了 1.7B 和 8B 两档,都用 Qwen3 当底并按相同 recipe 训练(全局 batch 256)。横向:\n\n- 8B 比 1.7B 在生成侧(MSE)提前到约 14K 步达到 0.060,1.7B 模型要到 40K 步;文本交叉熵 0.50 也由 1.7B 的 31K 步提前到 8B 的 10K 步——8B 用了约 1\u002F3 步数达到相当水平。\n- 128 卡对照 8 卡:文本 CE 收敛到 1.0 用 2K 步(8 卡要 17.5K),提速约 9×;生成 MSE 收敛到 0.065 用 17K 步(8 卡要 23K),提速约 1.3×。更多 GPU 对文本收得远比对生成快。\n- 视频压缩方面,p32\u002Ft1(每 token 1024 像素)损失最低,p16\u002Ft4(每 token 4096 像素)最终模型采用这一档以与 Wan2.2 等常见 VAE 对齐;p32\u002Ft4(每 token 8192 像素)损失明显抬高,论文给到的结论是「压缩越强,视频生成越难学」。\n- patch 大小:32×32 拟合图像每步可塞 4 倍样本,但训练后期 T2I 损失明显高于 16×16,因此最终模型用 16×16 patch。\n- 卷积输出头对比:Wan-style 上采样+conv 边界最干净但 GFLOPs 17.1×;PixelShuffle 时序优先(F3-R03)训练 T2V 像素流损失收敛到约 0.02,边界残留比线性头低;但发布的 checkpoint 与所有线上 demo 仍然用线性输出头,因为切到卷积头要补训练。\n\n理解基准上 8B PixelUMM 在 BLINK、MMStar、SEED-I、DocVQA、CountBench、MMVP 等位置与 BAGEL-7B、TUNA-7B+5B、LLaVA-OV-1.5\u002F28B、Qwen3-VL-Inst-8B 等同期 UMM\u002FVLM 同台打分,数字有赢有输(论文也明确「不同训练数据不能直接判定架构高低」);视频侧 8B PixelUMM 用 sparse_mode 采样 96 帧,在 Video-MME(无字幕)拿到 57.33,落后于 LLaVA-OV-28B(71.90)和 Qwen3-VL-8B(71.40),但 MV-Generation\u002FF1 报告的 sparse_mode\u002F64FPS tubelet 与 1FPS 帧输入两项模式 MVBench\u002FVideo-MME\u002FLongVideoBench\u002FLVBench 几乎打平(Δ 在 ±0.7 以内)。\n\n## 为什么这件事值得关注\n\n砍掉 VAE 的代价不是免费的。PixelUMM 论文自己也列了局限性:多实体(几个动物或人物重叠时,身体会合并、消失或突然出现)、手指\u002F细粒解剖(指头数量不对、小肢体模糊)、物理与交互(运动不真实、刚性物体弯曲)——这些是潜空间视频扩散模型的常见毛病,在像素空间并没有被绕过。但 VAE 的代价同样真实:上下文长度、训练管线双重维护、跨视觉接口对齐。PixelUMM 给出的实验里,8B 模型在 1.7B 加速比约 3× 这个量级的步骤效率,128 卡 CB 128 GPU 对 CE 的 9× 加速(但 MSE 只 1.3×)说明「文本收得快、像素生成收得慢」是这套架构当前最值得关注的 scaling 现象——后续如果把卷积输出头真正训通(论文已经给到 PixelShuffle 时序优先方向),视频侧大概率能再吃到一个段收益。\n\n产品角度看,PixelUMM 把视觉上下文从「ViT tokens + VAE tokens」合并成单一 pixel token 流,对那些已经基于 Qwen3 搭多模态管线、又不想同时维护两条视觉接口的团队,是值得跟踪的方向——但 pixel-space flow matching 的训练稳定性、长序列视频的边界 artifact,都是上生产前需要先实测的硬指标。\n\n来源:\n- 论文页面:https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.38597\n- 项目主页:https:\u002F\u002Fnv-tlabs.github.io\u002FPixelUMM\u002F\n- 权重仓库:https:\u002F\u002Fhuggingface.co\u002Fnvidia\u002FPixelUMM","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.38597","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21,24],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"7b67033c-19e6-4052-a626-e681bba64c7a","diffusion",{"id":19,"name":20,"slug":20,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":22,"name":23,"slug":23,"description":14,"color":14},"8dac812d-3839-4abe-a855-5f56ec9515fd","nvidia",{"id":25,"name":26,"slug":26,"description":14,"color":14},"ebe5dcd1-46b1-4298-b8c2-8e0e2f456e56","video-generation",[28],{"id":29,"lang":30,"title":31,"summary":32,"content":33},"dc63a4b7-09d5-486d-a85d-91fa8f7e4f63","en","PixelUMM drops the VAE and unifies image and video","PixelUMM (arXiv:2609.38597): one Qwen3-8B Transformer with no VAE unifies image and video. 8B hits loss parity in ~1\u002F3 the steps of the 1.7B variant.","Background\n\nUnified multimodal models (UMMs) have long run two parallel visual interfaces — a ViT for understanding-side semantic features and a VAE for generation-side reconstruction latents. Visual context length is doubled, and every new task has to be wired into both pipelines. Pixel-space modeling has been tried on images, but extending that paradigm to a single interface for both image and video understanding and generation has not been pulled off by anyone — until now.\n\nNVIDIA's PixelUMM (arXiv:2609.38597, GitHub nv-tlabs\u002FPixelUMM, weights mirrored at Hugging Face nvidia\u002FPixelUMM), posted September 29, 2026, takes that on. Authors: Cong Wei, Xuanchi Ren, Bryan Chu, Weiming Ren, Huan Ling, Jiahui Huang, Laura Leal-Taixé, Sanja Fidler, Wenhu Chen, Zian Wang, Jay Zhangjie Wu.\n\n## Core idea\n\nPixelUMM has no VAE and no vision encoder — a single decoder-only Transformer reads and writes raw pixels. Images are split into 16×16 patches, videos into 4-frame spatio-temporal tubelets, and both go through single-layer linear projections into a shared multimodal backbone (Qwen3-8B). A Mixture-of-Transformers split separates task-specific parameters, with one set of self-attention shared across text, clean pixels, and noisy pixels. The model jointly supports autoregressive text prediction and pixel-space flow matching — no separate VAE-token stream, no separate understanding branch.\n\nThe crucial detail is that clean pixels, noisy pixels, and text all share one self-attention window. The paper's F7 experiment: after 15K steps of multi-task fine-tuning, image understanding moves both ways (BLINK +2.37, CV-Bench +2.27, SEED-I −5.41), while all four video benchmarks improve by 0.89–3.49 points. That tells you clean pixel conditions routing through the understanding expert do not systematically hurt understanding — the most obvious concern once you drop the VAE.\n\n## Results\n\nPixelUMM reports 1.7B and 8B models trained with the same recipe (global batch 256). Headlines:\n\n- 8B reaches MSE 0.060 at ~14K steps; 1.7B needs ~40K. Text CE 0.50 lands at ~10K for 8B vs ~31K for 1.7B — 8B reaches comparable generation and text losses in roughly one third of the training steps.\n- 128 GPUs vs 8 GPUs on the same 1.7B model: text CE reaches 1.0 at ~2K steps (vs ~17.5K at 8 GPUs), a ~9× speedup; generation MSE reaches 0.065 at ~17K (vs ~23K), a ~1.3× speedup. More GPUs accelerate text far more than generation.\n- Video tubelet compression: p32\u002Ft1 (1024 pixels\u002Ftoken) gives the lowest T2V loss; p16\u002Ft4 (4096 pixels\u002Ftoken) is used in the final model to align with common video VAEs like Wan2.2; p32\u002Ft4 (8192 pixels\u002Ftoken) raises it back up — \"stronger compression makes video generation harder to learn.\"\n- Patch size: 32×32 packs 4× more images per step but T2I loss stays above 16×16 in late training; final model uses 16×16.\n- Convolutional output heads: Wan-style upsample+conv has the cleanest boundaries but 17.1× GFLOPs; PixelShuffle temporal-first (F3-R03) converges T2V pixel-flow loss to ~0.02 with smaller boundary residual elevation. The released checkpoint and online demos still use linear heads — switching requires additional training. Understanding-benchmark tables put 8B PixelUMM on par with BAGEL-7B, TUNA-7B+5B, LLaVA-OV-1.5\u002F28B, Qwen3-VL-Inst-8B across BLINK, MMStar, SEED-I, DocVQA, CountBench, MMVP, etc. — wins and losses mixed (the paper explicitly notes that different training data preclude declaring an architecture winner). On video the same model in sparse_mode samples 96 frames and posts Video-MME (no subtitles) 57.33, behind LLaVA-OV-28B (71.90) and Qwen3-VL-8B (71.40), but sparse_mode\u002Fdense_mode at 4 FPS or 1 FPS are within ±0.7 on MVBench\u002FVideo-MME\u002FLongVideoBench\u002FLVBench.\n\n## Why it matters\n\nDropping the VAE is not free. The paper itself calls out the limitations: many-entity scenes where overlapping bodies drift in and out, hands with wrong finger counts, and physically implausible motion — PixelSPACE mode does not bypass the failure modes latent video diffusion models have. The price of keeping the VAE is also real: doubled context length, dual training pipelines, alignment across two visual interfaces. The paper's experiments show 8B reaches comparable loss in roughly one third the steps of 1.7B, and 128 GPUs speed up text by ~9× while speeding generation by only ~1.3× — \"text converges fast, pixels converge slowly\" is the scaling observation most worth watching on this architecture. If the convolutional head is properly trained through (the paper already points at PixelShuffle temporal-first as the better trade-off), video side should pick up another segment of gain.\n\nFor practitioners, PixelUMM collapses two visual interfaces (ViT tokens + VAE tokens) into one pixel token stream, which is worth tracking for teams that already build on Qwen3 multimodal pipelines and do not want to maintain two visual interfaces in production. Training stability of pixel-space flow matching and boundary artifacts in long video sequences are the production-blocking items to measure.\n\nSources:\n- Paper page: https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.38597\n- Project page: https:\u002F\u002Fnv-tlabs.github.io\u002FPixelUMM\u002F\n- Weights: https:\u002F\u002Fhuggingface.co\u002Fnvidia\u002FPixelUMM","pixelumm-encoder-free-unified-multimodal-8b","2026-09-29T21:59:00Z","2026-10-03T01:11:36.848675Z","2026-10-03T01:11:36.848686Z",true,"agent",653,[42,51],{"slug":43,"tag_slug":43,"title_zh":44,"title_en":45,"intro_zh":46,"intro_en":47,"id":48,"is_active":38,"created_at":49,"modified_at":50},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":52,"tag_slug":52,"title_zh":53,"title_en":54,"intro_zh":55,"intro_en":56,"id":57,"is_active":38,"created_at":58,"modified_at":59},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":61},[62,67,72,77,82,87],{"id":63,"title":64,"news_slug":65,"published_at":66},"62b2e6d9-4ac7-457c-a30c-548c713730ad","PhyCo：让视频生成模型「理解」物理世界","phyco-cvpr-2026-physics-video-controlnet-vlm-reward","2026-05-01T16:00:00+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"af21d26d-d9cd-4526-ac4a-66366a45848c","AV-GRPO:8张A800给22B音视频模型做RL后训练","av-grpo-audio-video-diffusion-rl","2026-09-27T17:08:13+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"9210c86b-3e1d-434b-bbfd-78e62c698aed","WorldCrafter 开源:给视频世界模型装上可查询的 3D 记忆,转一圈回来还是那个房间","worldcrafter-video-world-model-3d-memory","2026-09-22T19:08:48+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"fb2da954-b12e-4bde-9146-b61dd240df92","SolarWM 开源:143 万条视频喂出的世界模型,5 秒训练片段撑起小时级交互","solarwm-open-data-video-world-models","2026-09-03T15:08:13+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"bb473853-de1d-4d7e-ac50-a46188539a4a","FLUX 3 补齐交付最后一环:视频超分上探 4K,两种模式按需选","flux-3-video-upscale-4k","2026-08-28T15:35:00+00:00",{"id":88,"title":89,"news_slug":90,"published_at":91},"f4c705fd-47c9-481a-807f-8001820070f8","InfinityEdit:三注意力轻量适配器,把视频编辑推进无界流时代","infinityedit-infinite-video-editing-adapter","2026-08-25T13:00:00+00:00"]