The worst part of a video call is not the background hum — it is several people talking at once, with the voice you actually want buried in the overlap. Speech separation calls this task target speaker extraction (TSE): pull the target person's voice out of the mixture. Audio-only systems keep failing in real scenes because interfering voices sound so much like the target. On October 23, Tsinghua's machine learning group (thu-ml) posted an arXiv paper with a different route: feed the target speaker's mouth video to the model as well — if the lips match, the voice is theirs. The work was accepted to NeurIPS 2026 in September, and on October 29 it topped Hugging Face Daily Papers with 149 upvotes.

A real-world exam: RealSSA

Half the contribution is the benchmark. The authors argue existing AVTSE benchmarks fix the speaker count at two and ignore near-/far-field interference, reverberation and sudden noise. RealSSA-Sim provides 100,826 training / 9,133 validation / 9,169 test examples of simulated scenes: 1-5 dynamic near-field interfering speakers, plus 10-15 far-field interferers in far-field scenarios, across four environment types (small meetings, indoor, large public, outdoor). RealSSA-Real adds 54 paired audio-visual segments from real recordings (40 recordings, 10 speakers, 8 scenes, 83.99 minutes total), held out as an independent test set.

One forward pass, half a million trainable parameters

On the model side, Falcon performs multi-scale time-frequency separation in a single encoder-decoder pass; the visual path uses a frozen 11.2M-parameter FRCNN lip-reading encoder, leaving just 0.5M trainable parameters (11.72M total). Training adds a zero-cost trick: an encoder-space speaker contrastive loss that treats near-field interferers as negatives, suppressing near-field leakage with no inference overhead and transferring across backbones. The paper reports best extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix and VoxCeleb2-2Mix: SI-SNRi 7.7 dB and SDRi 8.2 dB on RealSSA, with DNSMOS 3.276 tied with Swift-Net-12. Against the causal AV-TFGridNet baseline it cuts parameters by 91.9%, computation by 20.2x and GPU latency by 11.1x — a 2-second clip runs in 51.3 ms on an RTX 3090 with 125.3 MB of RAM.

Engineering you can actually join a meeting with

The delivery is equally complete: the open-sourced Falcon Studio tracks the speaker's mouth with MediaPipe in the browser and runs the model locally, while a BlackHole virtual audio device routes the separated voice straight into Zoom or Tencent Meeting. It processes 2-second windows with 0.5-second hops; a window takes about 0.12 s on an Apple M5 Pro via Metal, with Metal, CUDA and CPU all supported. Weights sit on Hugging Face (JusperLee/Falcon and Falcon-Big); code and weights are Apache 2.0.

An edge-deployment signal, and one license crack

My read: the value is not any single score but taking "small" and "real" seriously at once — 0.5M trainable parameters, 51.3 ms latency and 125.3 MB of RAM is the scale that fits earbuds, hearing aids and meeting clients, while RealSSA finally puts "three people talking in a room with another table in the background" on the exam. Mind the license crack, though: code and weights are Apache 2.0, but the datasets are gated — CC BY-NC-ND 4.0 for Sim and research-use terms for Real — and the README states that code/weight licensing does not override dataset terms. Reproducing the negative-speaker contrastive training means clearing the data application gate first.

So: while multimodal frontier models race on parameter count, the speech front is competing on who can be smaller and more real. Lip-guided speaker extraction may be one engineering iteration away from your next pair of noise-cancelling earbuds. (arXiv paper / GitHub repo)