[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-jd-joyai-video-edit-real-time-streaming":3,"news-related-bcdc10bc-2f08-4c39-8ffa-e7e34041c112":47},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":33,"news_slug":40,"published_at":41,"created_at":42,"modified_at":43,"is_published":44,"publish_type":45,"image_url":14,"view_count":46},"bcdc10bc-2f08-4c39-8ffa-e7e34041c112","京东开源 JoyAI-Video-Edit:用 16B 多模态扩散 Transformer 把视频编辑推进「边播边改」实时流时代","京东 8 月 5 日开源实时流式视频编辑模型 JoyAI-Video-Edit,基于自研 16B 多模态扩散 Transformer 和因果视频 VAE,在 720×1280 分辨率下达到 30.19 FPS 端到端推理速度,支持任意时长流式生成,用户在播放过程中即可实时修改人物、场景与风格。在 OpenVE-Bench 评测中全面超越 SANA Streaming、LiveEdit、Xmax-X2.0 等流式编辑基线。","## 技术背景:视频编辑为什么一直「卡」在离线批处理\n\n从 2023 年的 Runway Gen-2 到 2024-2025 年的 Sora、Kling、Wan,扩散视频生成在过去两年跑得飞快,但「视频编辑」这件事反而落后了 —— 主流方案都需要先把整段视频生成完,再做一次离线批量处理。这意味着创作者必须等模型把整段跑完才能动手调,几秒钟一个反馈循环是常态。\n\n进入 2026 年,LiveEdit、SANA Streaming 这批「流式视频编辑」工作登场,核心目标是把生成的时间粒度切到「每一帧到达时就处理」。但流式方案普遍受两个问题卡住:一是速度跟不上视频播放,导致编辑画面与原始音频节拍对不上;二是只能处理几秒到一分钟级别的短片段,长视频就直接 OOM 或时间一致性崩塌。\n\n京东这次开源的 JoyAI-Video-Edit,瞄准的恰好是这两个瓶颈。\n\n## JoyAI-Video-Edit 做了什么\n\n根据 Hugging Face 上的模型卡和 IT之家、新浪财经的同步报道,这个模型的核心架构由三块拼成:\n\n- **MLLM 条件编码器**:把用户的自然语言指令(「把这条街变成赛博朋克风」「给人物换一套红色外套」)和参考图编码成统一条件 token。\n- **因果视频 VAE**:对实时流入的视频帧做编解码,保证每一帧只依赖过去帧,不回头看未来帧。\n- **16B 参数的多模态扩散 Transformer (MMDiT)**:在隐空间里同时做条件注入和扩散去噪,带自回归式的滚动生成。\n\n为了让这套结构真的能跑在 30 FPS 上,训练和推理团队叠加了四类加速技术:**aligned autoregressive distribution matching distillation**(把多步蒸馏到少步,降低单帧去噪迭代次数)、**long-horizon optimization**(让模型在长视频上不漂移)、**bounded KV-state inference**(限制键值缓存的内存增长,避免长视频把显存吃光)、**deployment-oriented scheduling**(面向部署的算子调度优化)。\n\n最终的部署指标写得很硬:**720×1280 分辨率下端到端 30.19 FPS**,在常见视频播放节奏下画面处理无明显延迟,跟得上实时编辑的交互期望。\n\n## 模型在 OpenVE-Bench 上跑出了什么\n\n京东官方把 JoyAI-Video-Edit 拿去和当前流式视频编辑的代表 —— SANA Streaming、LiveEdit、Xmax-X2.0 —— 在 OpenVE-Bench 上做了横向对比。根据模型卡与 IT之家、新浪财经报道,模型在**全局风格转换、局部物体增减、字幕智能修改**这几类任务上明显领先,综合指标全面超越其他流式基线,实时出图的画质可以向离线专业剪辑工具看齐。\n\n更值得注意的是「任意时长稳定流式编辑」这一项 —— 之前的流式编辑基本只能处理几秒到分钟级的片段,推到直播级、影视前期拍摄级的长度就崩。JoyAI-Video-Edit 把这一限制取消了,创作者可以让视频中的人物连续更换服装、在直播中把普通街道变成动画世界、在家装设计中实时替换家具样式与整体装修风格。\n\n## 两条值得专门展开的场景线\n\n**一条是具身智能数据合成。** 机器人训练需要大量物体抓取、搬运、操作视频,但真机采集成本高,危险或少见工况更是难以反复采。JoyAI-Video-Edit 给出了一条自动化路径:把人手操作视频直接转成机械手\u002F机械臂作业素材,在保留物体位置、空间关系、动作轨迹的前提下替换场景、物体与机器人形态,一段视频能扩展出更多训练样本。\n\n**另一条是电商直播与影视前期。** 直播间里实时切换商品展示、模特穿搭、背景风格;家装设计里即时换家具、墙面、灯光;影视拍摄前把场景调整、人物造型改完再开机,降低实景重拍带来的资金与时间成本。\n\n开源后,这整套能力可以和京东既有的 **JoyAI-Image-Edit(图像编辑)**、**JoyAI-Echo(长视频生成)**、**JoyAI-VL-Interaction(实时交互)**、**JoyAI-Talker(语音)** 等模型形成完整的多模态工具链,覆盖图像创作、长视频、实时交互、语音合成、机器人动作控制全链路。\n\n## 我的判断\n\n京东在 JoyAI 系列上打的算盘非常清楚:不做「最大的视频生成模型」,而是把「实时流式」这条工程化最难的技术路线啃下来。流式视频编辑的核心难度在于 **speed-quality-consistency 三者的同时优化** —— 加速度往往意味着牺牲一致性,加一致性又得回头堆模型;JoyAI-Video-Edit 用 **16B 规模的 MMDiT + 蒸馏 + 长视野优化 + 有界 KV 缓存**这套组合,在 30 FPS 这个工程指标上把三者捏在一起。\n\n对于开发者来说,真正有吸引力的不只是「全球领先」的标签,而是它能让具身智能数据合成的成本曲线被改写。机器人训练数据长期是「高门槛-低产量-高单位成本」的循环;如果流式视频编辑能在保留物理轨迹的前提下做场景\u002F物体\u002F形态替换,等于把「一段真人视频 N 倍扩展成训练集」变成可工业化的流水线 —— 这比单纯的画质提升,对 AI 产业链下游的影响可能更深。\n\n京东把权重放在 Hugging Face 和 GitHub 双开源(Apache 2.0),有部署脚本 、本地端口 ,结合 MiMo-VL 与 ONNX 检测器做端到端运行。对想试水的开发者,门槛已经压到了一台带 GPU 的机器 + 几行 bash。\n\n最后留一个开放问题:当「实时流式 + 16B 规模」可以在消费级 GPU 上稳定工作,流式视频编辑是不是会从「专业剪辑工具的附属能力」升级为「创作者与 AI 协作的默认界面」?这件事 2026 下半年值得持续观察。\n\n## 参考\n\n- 京东开源原报道(IT之家):https:\u002F\u002Fwww.ithome.com\u002F0\u002F985\u002F793.htm\n- 京东开源原报道(新浪财经\u002F快科技):https:\u002F\u002Ffinance.sina.com.cn\u002Ftech\u002Froll\u002F2026-08-05\u002Fdoc-inimfshm5205260.shtml\n- JoyAI-Video-Edit Hugging Face 模型卡:https:\u002F\u002Fhuggingface.co\u002Fjdopensource\u002FJoyAI-Video-Edit\n- 论文(Hugging Face papers):https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2608.03974\n- GitHub 仓库:https:\u002F\u002Fgithub.com\u002Fjd-opensource\u002FJoyAI-Video-Edit\n- 京东黑板报官方公告(36氪快讯):https:\u002F\u002F36kr.com\u002Fnewsflashes\u002F3925977640777857","https:\u002F\u002Fhuggingface.co\u002Fjdopensource\u002FJoyAI-Video-Edit","24d5c6c5-6573-4180-a1fd-f1459842d1af",[11,15,18,21,24,27,30],{"id":12,"name":13,"slug":13,"description":14,"color":14},"fca9258a-9430-455a-b95d-b9fae5e373a8","ai-inference",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"7b67033c-19e6-4052-a626-e681bba64c7a","diffusion",{"id":19,"name":20,"slug":20,"description":14,"color":14},"7e89b5cc-57db-4f37-bc6d-28919a73931c","model-release",{"id":22,"name":23,"slug":23,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":25,"name":26,"slug":26,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":28,"name":29,"slug":29,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",{"id":31,"name":32,"slug":32,"description":14,"color":14},"ebe5dcd1-46b1-4298-b8c2-8e0e2f456e56","video-generation",[34],{"id":35,"lang":36,"title":37,"summary":38,"content":39},"73820b67-a033-4eb3-949d-d25b0f8d92f4","en","JD.com open-sources JoyAI-Video-Edit: 16B DiT for real-time editing","On August 5, JD.com open-sourced JoyAI-Video-Edit, a real-time streaming video editing model built on a self-developed 16B multimodal diffusion Transformer and a causal video VAE. It sustains end-to-end 30.19 FPS inference at 720×1280, supports streaming generation over arbitrary video lengths, and lets users edit subjects, scenes, and styles on the fly while playback continues. On OpenVE-Bench it comprehensively surpasses streaming baselines such as SANA Streaming, LiveEdit, and Xmax-X2.0.","## Why video editing was stuck in offline batch mode\n\nDiffusion video generation has moved fast from 2023's Runway Gen-2 through Sora, Kling, and Wan in 2024–2025 — but video *editing* lagged behind. Mainstream pipelines still demand a full video first, then a separate offline batch pass, so creators wait for the entire generation to finish before they can touch a single cut. Feedback loops of several seconds are the norm.\n\nIn 2026, a new wave of streaming video editing work — LiveEdit, SANA Streaming, and others — pushed generation down to a per-frame granularity. But streaming systems ran into two walls: inference speed that cannot keep up with playback, and length ceilings that collapse once a video exceeds a few seconds to one minute, with memory blow-ups and temporal consistency collapse.\n\nJD.com's newly open-sourced JoyAI-Video-Edit targets both walls head-on.\n\n## What JoyAI-Video-Edit actually does\n\nAccording to the model's Hugging Face card and parallel coverage from IT之家 and 新浪财经, the architecture is assembled from three pieces:\n\n- **An MLLM condition encoder** that ingests natural-language instructions (\"turn this street into cyberpunk\" \u002F \"swap the character's jacket to red\") together with optional reference imagery into a unified condition token stream.\n- **A causal video VAE** that encodes\u002Fdecodes frames as they arrive, with each frame depending only on past frames — never peeking at the future.\n- **A 16B-parameter multimodal diffusion Transformer (MMDiT)** that injects conditioning into the latent space and runs the denoising diffusion pass, running as an autoregressive-style rolling generator.\n\nTo actually push this structure to 30 FPS, the training and inference teams stacked four classes of acceleration: **aligned autoregressive distribution matching distillation** (compresses multi-step denoising into few-step), **long-horizon optimization** (so the model does not drift on long videos), **bounded KV-state inference** (caps KV-cache growth so memory does not blow up over long sequences), and **deployment-oriented scheduling** (operator-level optimizations for production serving).\n\nThe deployed numbers are stated very directly: **end-to-end 30.19 FPS at 720×1280**, no perceptible latency under typical video playback rates, sufficient for real-time interactive editing expectations.\n\n## What the model shows on OpenVE-Bench\n\nJD.com's team benchmarked JoyAI-Video-Edit against the current streaming representatives — SANA Streaming, LiveEdit, and Xmax-X2.0 — on OpenVE-Bench. Across the board, the model leads in **global style transfer, local object addition\u002Fremoval, and intelligent subtitle editing**, comprehensively surpassing other streaming baselines. The realtime render quality holds up next to offline professional editing tools.\n\nOne result that deserves attention is **stable streaming editing at arbitrary video length**. Earlier streaming editors topped out around a few seconds to a minute before failing; this work removes that ceiling. Creators can have a character swap outfits continuously across a video, transform an ordinary street into an animated world mid-stream, or try different furniture, wall, and lighting combinations in a real-time interior design session.\n\n## Two scenarios worth unpacking\n\n**Embodied AI data synthesis.** Robot training needs large volumes of object grasping, transport, and manipulation video. Real-world capture is expensive, and dangerous or rare conditions are hard to re-shoot. JoyAI-Video-Edit offers an automated path: convert a human-hand demonstration into a robotic-arm\u002Fhand clip while preserving object positions, spatial relations, and motion trajectories — then swap scenes, objects, and robot morphology. A single source video can fan out into many training clips.\n\n**E-commerce livestream and pre-production.** Real-time product, model, and background swaps during livestream sales; instant furniture, wall, and lighting swaps in interior design; in-place scene and character adjustments during pre-production planning — all of which shorten or eliminate expensive reshoots.\n\nAfter open-sourcing, this piece slots into JD.com's broader JoyAI family — **JoyAI-Image-Edit (image editing)**, **JoyAI-Echo (long video generation)**, **JoyAI-VL-Interaction (real-time interaction)**, **JoyAI-Talker (speech)** — forming a complete multimodal toolchain that covers image creation, long-form video, real-time interaction, speech synthesis, and robot motion control.\n\n## My take\n\nJD.com's play with the JoyAI line is clear: it is not chasing \"the largest video generation model\" — it is taking on the hardest *engineering* axis in video editing: real-time streaming. The core difficulty in streaming video editing is the simultaneous optimization of **speed, quality, and consistency**. Cranking speed usually sacrifices consistency; restoring consistency means stacking model size — and size hits back on latency. JoyAI-Video-Edit threads the needle using a **16B MMDiT + distillation + long-horizon optimization + bounded KV-cache** combo, and locks all three at 30 FPS.\n\nFor developers, the more interesting story is what this does to embodied-AI data pipelines. Robotics training data has been stuck in a \"high floor — low yield — high unit cost\" loop. If streaming video editing can replace scene, object, and morphology while preserving physical trajectories, then \"one human demo video, N-fan-out training set\" becomes an industrializable pipeline — and that shift is arguably more consequential for the downstream AI stack than any single quality bump in video synthesis.\n\nWeights are open-sourced under Apache 2.0 on both Hugging Face and GitHub, with a deployment script  and a local server endpoint at . End-to-end runtime needs MiMo-VL and an ONNX detector. For developers who want to play, the entry cost has been compressed to a single GPU box plus a few lines of bash.\n\nA closing question worth watching into the second half of 2026: when \"real-time streaming + 16B scale\" can run reliably on consumer-grade GPUs, does streaming video editing graduate from \"a feature inside pro editing tools\" to \"the default interface between creators and AI\"?\n\n## References\n\n- JD.com open-source coverage (IT之家, in Chinese): https:\u002F\u002Fwww.ithome.com\u002F0\u002F985\u002F793.htm\n- JD.com open-source coverage (新浪财经 \u002F 快科技, in Chinese): https:\u002F\u002Ffinance.sina.com.cn\u002Ftech\u002Froll\u002F2026-08-05\u002Fdoc-inimfshm5205260.shtml\n- JoyAI-Video-Edit Hugging Face model card: https:\u002F\u002Fhuggingface.co\u002Fjdopensource\u002FJoyAI-Video-Edit\n- Paper (Hugging Face papers page): https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2608.03974\n- GitHub repository: https:\u002F\u002Fgithub.com\u002Fjd-opensource\u002FJoyAI-Video-Edit\n- JD official announcement (36Kr flash, in Chinese): https:\u002F\u002F36kr.com\u002Fnewsflashes\u002F3925977640777857","jd-joyai-video-edit-real-time-streaming","2026-08-05T03:00:00Z","2026-08-05T04:02:59.217565Z","2026-08-05T04:02:59.217574Z",true,"agent",487,{"items":48},[49,54,59,64,69,74],{"id":50,"title":51,"news_slug":52,"published_at":53},"6f9e9f94-9dcc-4c6c-b254-6c5d0fe8ed37","京东开源 JoyAI-Video-Edit:16B 多模态扩散 Transformer 把视频编辑推进「边播边改」实时流时代","jd-joyai-video-edit-realtime-diffusion","2026-08-10T00:00:00+00:00",{"id":55,"title":56,"news_slug":57,"published_at":58},"2874a2e5-beae-4627-8f6f-a34cf2cc8d7a","一段随手拍视频直出4D人体:4DAnyone用RCP+TCR破解多视角一致性,代码权重全开源","4danyone-monocular-video-4d-human","2026-08-20T17:59:53+00:00",{"id":60,"title":61,"news_slug":62,"published_at":63},"5612d186-46ee-4509-9a93-94045ba004ae","LTX-2.5 开放权重视频模型:4K 反而在 Fast 端点,EXR 色彩管线也焊进去了","ltx-2-5-open-weights-video","2026-08-18T15:20:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"aad00b18-d354-48b5-ad21-62b53150b8c6","MiniMax H3 开源实测:你下载的权重,和 API 里跑的不是同一个模型","minimax-h3-local-vs-api-gap","2026-08-15T17:07:24+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"6e3002da-c1fd-4a6d-b903-4f65b976dd04","MiniMax H3 首个商用落点：美图 RoboNeo 接入背后,通用多模态模型的\"可编辑性\"才刚开始被检验","roboneo-minimax-h3-multimodal-editing","2026-08-03T18:02:02+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"6f375936-79af-4622-a75e-d802ade563e0","MiniMax H3 不只是 2K 视频：它想把生成、参考和编辑收回一个模型","minimax-h3-omnimodal-video-unified-generation-editing","2026-08-03T04:08:31+00:00"]