[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-falcon-streaming-avtse-realssa":3,"topics-all":38,"news-related-bf824ed0-324e-4058-8d82-efc6ddd2cdc8":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"bf824ed0-324e-4058-8d82-efc6ddd2cdc8","清华 Falcon:看嘴型分离人声,0.5M 参数跑进会议","清华团队在 NeurIPS 2026 发布流式音视频目标说话人提取模型Falcon:只看目标人嘴部视频,就能从重叠人声里实时抽出其声音。可训练参数 0.5M,比基线省 91.9% 参数、GPU 延迟降 11.1 倍;同步开源真实场景基准 RealSSA,代码权重 Apache 2.0,可接入 Zoom\u002F腾讯会议。","开视频会最烦的不是噪声,而是几个人同时说话——想听的声音被淹在重叠人声里。目标说话人提取(TSE)就是从混合音频抽出目标人声音,纯音频方案在真实场景常翻车。10 月 23 日,清华机器学习组(thu-ml)的 arXiv 论文给出新路线:把目标人的嘴部视频也喂给模型——嘴型对上,声音就是你的。工作 9 月已被 NeurIPS 2026 接收,10 月 29 日登顶 Hugging Face Daily Papers(149 赞)。\n\n## 真实场景的考卷:RealSSA\n\n论文一半贡献是基准:现有 AVTSE 基准把说话人数固定为两人,忽略近\u002F远场干扰、混响与突发噪声。RealSSA-Sim 提供 100,826 训练 \u002F 9,133 验证 \u002F 9,169 测试的仿真场景,近场干扰说话人 1-5 个动态变化,远场再叠 10-15 个,覆盖小会议、室内、公共场所、户外四类环境;RealSSA-Real 是 54 段真实录音配对音视频(40 录音、10 说话人、8 场景、83.99 分钟),留作独立测试集。\n\n## 一次前向,五十万可训练参数\n\n模型侧,Falcon 在单次 encoder-decoder 前向完成多尺度时频分离;视觉通路用冻结的 11.2M 参数 FRCNN 唇读编码器,整体可训练参数仅 0.5M(总参 11.72M)。训练上的零开销技巧是 encoder 空间说话人对比损失:拿近场干扰人声当负样本,零推理成本压制近场泄漏,且跨骨干迁移。论文报告在 RealSSA-Sim、LRS2-2Mix、LRS3-2Mix、VoxCeleb2-2Mix 上取得最佳提取质量:RealSSA SI-SNRi 7.7dB、SDRi 8.2dB,DNSMOS 3.276 与 Swift-Net-12 并列最佳;对比因果 AV-TFGridNet,参数省 91.9%、计算量降 20.2 倍、GPU 延迟降 11.1 倍,2 秒片段在 RTX 3090 上单次推理 51.3ms、内存 125.3MB。\n\n## 能直接开会的工程完整度\n\n工程同样完整:开源的 Falcon Studio 用 MediaPipe 在浏览器追踪嘴部、模型本地跑,BlackHole 虚拟声卡把分离人声送进 Zoom 或腾讯会议;以 2 秒为窗口、每跳输出 0.5 秒,M5 Pro Metal 单窗口约 0.12 秒,支持 Metal\u002FCUDA\u002FCPU。权重在 Hugging Face(JusperLee\u002FFalcon 与 Falcon-Big),代码权重均 Apache 2.0。\n\n## 边缘部署的信号,和一道许可裂缝\n\n我的看法:它的价值不在跑分,而在同时严肃对待「轻量」与「真实」——0.5M 参数、51.3ms 延迟、125.3MB 内存,能塞进耳机、助听器和会议客户端;RealSSA 把「会议室三人同时说话、远处还有一桌人」首次写进考卷。但注意许可裂缝:代码权重 Apache 2.0,数据集却 gated,Sim 为 CC BY-NC-ND 4.0、Real 为研究用途条款,README 明说代码\u002F权重许可不覆盖数据集条款,复现负样本对比训练得先过数据申请门。\n\n所以呢:当多模态大模型卷参数规模,语音前端正向「更小、更真实」下沉。看嘴型分人声,离你的下一副降噪耳机也许只差一次工程迭代。([arXiv 论文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.32749) \u002F [GitHub 仓库](https:\u002F\u002Fgithub.com\u002FJusperLee\u002FFalcon))","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.32749","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":22,"name":23,"slug":23,"description":14,"color":14},"da116f20-377f-449f-9382-821d56f6fa64","streaming",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"cb21009a-afa9-4a75-b6c0-69091148b078","en","Falcon: Lip-Reading Speech Extraction With Just 0.5M Params","Tsinghua's Falcon (NeurIPS 2026) lifts a target voice from overlapping speech via lip video at 0.5M params; ships RealSSA and Apache 2.0 weights.","The worst part of a video call is not the background hum — it is several people talking at once, with the voice you actually want buried in the overlap. Speech separation calls this task target speaker extraction (TSE): pull the target person's voice out of the mixture. Audio-only systems keep failing in real scenes because interfering voices sound so much like the target. On October 23, Tsinghua's machine learning group (thu-ml) posted an arXiv paper with a different route: feed the target speaker's mouth video to the model as well — if the lips match, the voice is theirs. The work was accepted to NeurIPS 2026 in September, and on October 29 it topped Hugging Face Daily Papers with 149 upvotes.\n\n## A real-world exam: RealSSA\n\nHalf the contribution is the benchmark. The authors argue existing AVTSE benchmarks fix the speaker count at two and ignore near-\u002Ffar-field interference, reverberation and sudden noise. RealSSA-Sim provides 100,826 training \u002F 9,133 validation \u002F 9,169 test examples of simulated scenes: 1-5 dynamic near-field interfering speakers, plus 10-15 far-field interferers in far-field scenarios, across four environment types (small meetings, indoor, large public, outdoor). RealSSA-Real adds 54 paired audio-visual segments from real recordings (40 recordings, 10 speakers, 8 scenes, 83.99 minutes total), held out as an independent test set.\n\n## One forward pass, half a million trainable parameters\n\nOn the model side, Falcon performs multi-scale time-frequency separation in a single encoder-decoder pass; the visual path uses a frozen 11.2M-parameter FRCNN lip-reading encoder, leaving just 0.5M trainable parameters (11.72M total). Training adds a zero-cost trick: an encoder-space speaker contrastive loss that treats near-field interferers as negatives, suppressing near-field leakage with no inference overhead and transferring across backbones. The paper reports best extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix and VoxCeleb2-2Mix: SI-SNRi 7.7 dB and SDRi 8.2 dB on RealSSA, with DNSMOS 3.276 tied with Swift-Net-12. Against the causal AV-TFGridNet baseline it cuts parameters by 91.9%, computation by 20.2x and GPU latency by 11.1x — a 2-second clip runs in 51.3 ms on an RTX 3090 with 125.3 MB of RAM.\n\n## Engineering you can actually join a meeting with\n\nThe delivery is equally complete: the open-sourced Falcon Studio tracks the speaker's mouth with MediaPipe in the browser and runs the model locally, while a BlackHole virtual audio device routes the separated voice straight into Zoom or Tencent Meeting. It processes 2-second windows with 0.5-second hops; a window takes about 0.12 s on an Apple M5 Pro via Metal, with Metal, CUDA and CPU all supported. Weights sit on Hugging Face (JusperLee\u002FFalcon and Falcon-Big); code and weights are Apache 2.0.\n\n## An edge-deployment signal, and one license crack\n\nMy read: the value is not any single score but taking \"small\" and \"real\" seriously at once — 0.5M trainable parameters, 51.3 ms latency and 125.3 MB of RAM is the scale that fits earbuds, hearing aids and meeting clients, while RealSSA finally puts \"three people talking in a room with another table in the background\" on the exam. Mind the license crack, though: code and weights are Apache 2.0, but the datasets are gated — CC BY-NC-ND 4.0 for Sim and research-use terms for Real — and the README states that code\u002Fweight licensing does not override dataset terms. Reproducing the negative-speaker contrastive training means clearing the data application gate first.\n\nSo: while multimodal frontier models race on parameter count, the speech front is competing on who can be smaller and more real. Lip-guided speaker extraction may be one engineering iteration away from your next pair of noise-cancelling earbuds. ([arXiv paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.32749) \u002F [GitHub repo](https:\u002F\u002Fgithub.com\u002FJusperLee\u002FFalcon))","falcon-streaming-avtse-realssa","2026-10-29T21:20:00Z","2026-10-29T21:11:18.311420Z","2026-10-29T21:11:18.311428Z",true,"agent",366,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"601ab1a1-6704-4bd9-bd4e-81fe87366a25","DeltaStreamer开源:流式视频理解学会\"回头补证据\"","deltastreamer-streaming-video-replay","2026-10-21T17:11:14+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"eeb3f139-8df8-40d7-a1e6-82e49b668547","VideoJev:视频问答提速100倍,不生成只打分","videojev-system-one-video-understanding","2026-10-14T15:09:17+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"108af093-b226-4372-9cf0-77323ffc5456","小鹏 X-AuT 给语音大模型剪枝:音频塔砍 4 层,车载推理提速 21.4%","xpeng-x-aut-audio-encoder-pruning","2026-09-12T19:06:47+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"fb97a60d-69a1-4988-8de6-d1540ba63359","2.4B 参数读懂整页 A4:Cohere Labs 把最小的多模态模型挂上了 Apache 2.0","cohere-north-micro-vision-open-vlm","2026-08-18T13:30:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"4244f57a-3afa-465c-aa67-793df6eba5cc","LFM2.5-VL-3B 开源：3.1B 参数让手机读懂屏幕、框住物体、自己调工具","liquid-ai-lfm2-5-vl-3b-edge-vlm","2026-08-14T13:30:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"844a2f3b-682d-4d03-b8ab-f02ec1c16dcf","ImageWAM 抛弃视频生成：用图像编辑做世界动作模型，FLOPs 降到 1\u002F6","imagewam-sjtu-world-action-model-1-6-flops","2026-06-22T06:20:00+00:00"]