A 0.6B speech recognition model, with its backbone frozen and a few small readouts layered on top, scored higher than a 30B audio language model on questions like 'is this my turn to speak'. That is a number from the paper's own comparison table (https://arxiv.org/abs/2611.18330). What makes it interesting is not the score but the entirely different way of asking underneath it.

Decisions, not generation

The paper is called Marimba, authored by Solano Amadori, affiliated with charcot.ai, submitted to arXiv on 2026-11-13. The core mechanism is one sentence: for an 8-second audio window, a frozen Qwen3-ASR 0.6B encoder runs forward once, and the same features answer every question asked about that window.

The output is constrained to three types: choice, yes/no, and score, each carrying a probability, with no generated rationale attached.

The division of labour is explicit. Lexical questions, meaning 'was this particular word spoken', get no task-specific training at all: they read the frozen decoder's output directly, greedy decoding plus four beam searches for five hypotheses in total, normalized, matched as exact token sequences, then mapped to a probability through Platt scaling. Non-lexical questions go through a question-conditioned head that cross-attends to each candidate option. Turn-end detection takes a separate path, pooling the final two seconds of layer-18 features. Language identification is cheaper still, using the ASR model's own language token probabilities across 30 languages. The three speaker decisions sit on layer-12 features.

Where it wins, where it loses

One caveat first: in the paper's comparison over 1,000 fixed public questions, the turn-end row is in-distribution for Marimba, because its readout was trained on Smart Turn training data. Every other row is zero-shot for every system. So the wins below come with that discount.

Two rows of the comparison table deserve pulling out. On FLEURS binary term detection, its AUROC is 0.901 against 0.791 for Qwen3-Omni and 0.682 for Qwen2-Audio. On turn end the figures are 0.935, 0.786, and 0.476, with Qwen2-Audio sitting at chance: its 95% confidence interval runs from 0.40 to 0.55, and across all 200 clips it picked the affirmative option every time.

Two other rows go the other way, to Qwen3-Omni: Speech Commands binary AUROC of 0.991 against 0.926, and Common Voice language identification accuracy of 1.000 against 0.904. The official repository later extended the comparison table to four systems, adding FireRedAudio, which reached only 0.542 on turn end.

A tougher test is a purpose-built model. On Smart Turn v3.2's own 500-clip test set, the Marimba readout reaches AUROC 0.930 while Smart Turn reaches 0.982, a paired difference of -0.052 with a 95% interval from -0.075 to -0.032. Out of domain on ICSI, the two are 0.716 and 0.712, a difference of +0.004 with an interval from -0.046 to +0.054, and the paper states plainly that this is a statistical tie, evidence for neither advantage nor equivalence.

The speaker results are good. Speaker change reaches 0.966 and 0.980, against an ECAPA baseline of 0.924 and 0.962 and pyannote 3.1 at 0.865 and 0.812. Same speaker reaches 0.998 and 0.996 against ECAPA at 0.985 and 0.995. Speaker count accuracy is 0.778 and 0.859 against pyannote at 0.738 and 0.836. But three or more speakers get undercounted: on VoxConverse, 395 of 667 such windows were predicted as two people. Voice activity runs 0.986 to 0.995 on AMI and at least 0.999 on VoxConverse.

Lexical and privacy results need reading together. Language identification hits 100% on both open and closed set over 300 FLEURS clips, and 93.33% closed set and 84.33% open set on Common Voice. Personal-information detection used 200 synthetic Brazilian Portuguese clips and reached item-level F1 of 0.904, flagging 3 of 100 negatives; on reference transcripts the F1 is 1.000, which shows the misses come from transcription itself, with recall at 0.70 for phone numbers and 0.60 for CPF numbers.

The paper writes down where it loses

Unseen-task scene classification sits entirely at chance. Four-way choice accuracy is 0.24 to 0.29 where chance is 0.25, and binary AUROC is 0.48 to 0.54. These readouts are not a general-purpose semantic head.

Latency is uneven too. Turn end is 13 ms, voice activity 16 ms, language identification 161 ms, and term detection 2.35 s, because lexical questions must decode, and encoder reuse does not establish streaming readiness. Cross-domain calibration is no bargain either: five-fold cross-meeting recalibration on ICSI brings ECE down from 0.200 to 0.011, but that costs 200 labelled windows per fold, and the paper itself notes this shows in-domain calibration only, not cross-domain calibration.

On engineering details, the official repository (https://github.com/Charcot-AI/marimba-api) states that personal-information detection requires a caller-supplied text detector which the repository does not include, and that the audio scene classification lane is marked unavailable, returning None rather than guessing.

The ethics section is worth a separate look. The private data came from a commercial clinical documentation platform whose terms of use, accepted by the clinicians acting as data controllers under Brazil's LGPD, authorize the provider to de-identify the data and use it to train AI models. But the study was not reviewed by a research ethics committee, and patients did not give study-specific research consent. The private recordings are not released, and only aggregate results are reported.

Is this direction worth following

The mechanism is not one paper's lonely idea. A separate team's ASOM paper (https://arxiv.org/abs/2610.22401) takes the same route: frozen speech and text encoders feeding a 6.2M-parameter reader, answering 20 questions in 13.1 ms, 17 to 41 times faster than the audio language models. And the incumbent in turn detection, Smart Turn v3.2, is itself roughly 8M parameters on a Whisper Tiny backbone, running in as little as 10 ms on some CPUs (https://github.com/pipecat-ai/smart-turn).

The backbone is not the bottleneck either. Qwen3-ASR-0.6B is released under Apache-2.0, and the official repository documents language identification and ASR for 52 languages and dialects, reaching 2000 times throughput at a concurrency of 128 (https://github.com/QwenLM/Qwen3-ASR). Freezing a good enough backbone and hanging readouts off it has become cheap enough to stop looking like a research problem.

My read is that the hard part of a voice agent is not understanding speech but deciding whether to speak now and how confident that decision is. Writing the question as a typed yes/no and forcing a probability out gives the layer above something it can threshold and refuse on, and in a real-time voice stack that is worth more than a few points of average accuracy. The cost is equally concrete: because term questions have to decode, latency jumps from 13 ms to 2.35 s.

So the question I leave with you: when a system turns 'I am not sure' into a programmable return value, are you more willing to hand it the decision, or more worried that it is using a tidy probability to hide the uncertainty?