[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-sony-walrus-8b-speech-audio-foundation":3,"topics-all":38,"news-related-1208ec8f-77b0-476f-8309-720517d22d32":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"1208ec8f-77b0-476f-8309-720517d22d32","8B语音基座追平4倍大Omni:索尼把力气全花在数据上","索尼拿不变的Gemma 4 E4B骨架练出8B统一语音基座Walrus:语音token码率压到325bps,三阶段数据课程喂600M样本、覆盖20多个任务和100种语言。论文口径下FLEURS转写94.7%反超Qwen3-Omni-30B,多人混说话人错误率近半,纯商用数据版在音频理解上让掉20分。","多说话人互相抢话、一句话里中英混杂、嘈杂的娱乐音频——这些是语音基座模型真正的硬骨头,也是公开语料里最稀缺的东西。索尼团队的 Walrus 技术报告选了一条和当下主流相反的路:架构一行不改,把全部设计力气花在数据上。\n\n## 骨架照抄 Gemma 4,改动在词表和声码器\n\nWalrus 是一个统一语音+音频基座模型,总参数约 8B(含 embedding,有效参数约 4B),直接从 Gemma 4 E4B 初始化,backbone 结构保持不变,语音输入沿用 E4B 的 USM 编码器并在微调时解冻。单一权重服务所有任务,任务靠自然语言 prompt 切换:ASR、语码混杂 ASR、多说话人 ASR、机器翻译、语音翻译、语音到语音翻译、TTS、情感识别、音频字幕和通用音频理解,20 多个任务变体、100 种语言。语音生成侧只给词表扩展了 8192 个离散语音 token,每秒 25 个 token、每个 13 bit,码率压到 325 bps,再由一层 VQ tokenizer 加 GAN 声码器还原波形——token 预测和波形合成解耦,backbone 保持轻量。\n\n## 真正的贡献:三阶段数据课程加双权重\n\n训练用了超过 600M 样本。第一阶段打多语言地基,识别任务权重优先,各数据集按原始规模采样;第二阶段在第一阶段 checkpoint 上加时间戳预测、说话人分离、合成语码混杂 ASR 和对话上下文任务,并引入数据集权重给旧数据降采样;第三阶段专项补短板:合成娱乐语音、低资源语言,以及覆盖中英法德意日西 7 种语言、2 到 4 人混叠的合成多说话人数据。第三阶段还重排了任务权重——ASR 从 5.0 降到 4.0,TTS 从 0.25 升到 2.0,语音到语音翻译从 0.5 升到 2.0。论文对最难案例的处理方式很直白:语码混杂 ASR 数据完全合成;多说话人是真实加模拟混合;娱乐语音用 YODAS-Granary 这类宽松许可语料加合成增强;连合成多轮对话都用 Gemma 4 26B 与 Qwen3.5 9B 三种模型配对生成,再由 Gemma 4 26B 当裁判按 0 到 1 打分。\n\n## 战绩:4 倍大的 Qwen3-Omni 也没拉开\n\n论文评测口径下,FLEURS 转写词准确率 94.7%,高于 Qwen3-Omni-30B 的 93.8% 和 Whisper Large-v3 的 92.0%;CS-FLEURS 语码混杂识别 CER 7.23,而 Gemma 4 是 32.30;LibriMix 双人混叠 cpWER 19.32,约为 Qwen3-Omni-30B(37.04)的一半,远低于专用模型 VibeVoice-ASR 的 59.34——同量级的开源 SpeechLLM 在这个设定下干脆产不出有效输出。音频理解上,MMAU-Pro 71.92% 超过 Gemini-Flash-3.7 的 70.40% 和 GPT-4o-Audio 的 52.50%;MMSU 64.1% 对 Audio-Flamingo-3 的 61.4%;CVSS-C 语音到语音翻译 43.63 ASR-BLEU 也压过 30B 的 Qwen3-Omni(41.27)。但不是全赢:FLEURS 双向语音翻译 32.28\u002F22.94 仍低于 Qwen3-Omni 的 33.60\u002F25.67,论文自己也承认这一条,只在说话人相似度和韵律保持上占优。\n\n## 商用版的代价,写在许可协议里\n\nWalrus 出了两个版本:商用版 CR 只喂商用许可数据,停在第二阶段 checkpoint;研究版 R 用全量数据跑完三阶段。从 CR 到 R,语码混杂 CER 再降约 19 个点,内部娱乐 ASR 平均 WER(6 种语言)再降约 9 个点,MMAU-Pro 高出 20 分以上。换句话说,许可协议直接标出了能力差距——这是全文最值得产业界注意的一组数字。\n\n## 所以呢\n\n在 backbone 越来越像公共设施的时代,这份报告把「数据工程是多任务语音模型质量的主要驱动力」做成了一个受控实验:架构钉死,收益全部可归因到数据。对做语音的团队,启示有两条——最难的长尾场景(混叠、语码混杂、娱乐音频)要靠合成管线去补,而不是等公开语料;商用与研究的分野,未来大概率不取决于模型结构,而取决于你被允许用哪些数据。原文:https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13689","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13689","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"c4455801-0273-477d-9e92-00b4addfa439","asr",{"id":19,"name":20,"slug":20,"description":14,"color":14},"7e89b5cc-57db-4f37-bc6d-28919a73931c","model-release",{"id":22,"name":23,"slug":23,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"f939025c-eabc-4b54-b93a-c55630b2e696","en","Sony's 8B Walrus Speech Model Rivals 4x Larger Qwen3-Omni","Sony's 8B Walrus keeps Gemma 4's architecture and wins on data: 600M samples in three stages match the 4x larger Qwen3-Omni on most speech benchmarks.","Overlapping speakers, mid-sentence code-switching, noisy entertainment audio - the hardest cases for speech foundation models are exactly the ones public corpora lack. Sony's Walrus Technical Report takes a contrarian route: change nothing in the architecture, and spend all the design effort on data.\n\n## A borrowed Gemma 4 backbone, an extended speech vocabulary\n\nWalrus is a unified speech-and-audio foundation model with about 8B total parameters (roughly 4B effective, embeddings included), initialized from Gemma 4 E4B with the backbone architecture unchanged; the speech input keeps E4B's USM encoder, unfrozen during fine-tuning. A single set of weights serves every task, selected through natural-language prompts: ASR, code-switching ASR, multi-talker ASR, machine translation, speech translation, speech-to-speech translation, TTS, emotion recognition, audio captioning, and general audio understanding - more than 20 task variants across 100 languages. On the generation side the vocabulary gains 8,192 discrete speech tokens, predicted at 25 tokens per second and 13 bits each, a bit-rate of 325 bps; a single-layer VQ tokenizer plus a GAN vocoder turns them back into waveforms, decoupling token prediction from waveform synthesis.\n\n## The real contribution: a three-stage data curriculum with two weights\n\nTraining consumed over 600M samples. Stage 1 lays a broad multilingual foundation with recognition-priority task weights, datasets sampled at natural size. Stage 2 starts from the Stage-1 checkpoint and adds timestamp prediction, diarization, synthetic code-switching ASR and dialogue-context tasks, while introducing dataset weights that down-sample older corpora. Stage 3 closes specific gaps: synthetic entertainment speech, low-resource languages, and synthetic multi-talker mixtures of two to four speakers covering Chinese, English, French, German, Italian, Japanese and Spanish. Stage 3 also rebalances task weights - ASR drops from 5.0 to 4.0, TTS rises from 0.25 to 2.0, speech-to-speech translation from 0.5 to 2.0. The recipe for hard cases is blunt: code-switching ASR data is fully synthetic; multi-talker data combines real and simulated recordings; entertainment speech augments permissively licensed collections such as YODAS-Granary; and even the synthetic multi-turn dialogues are generated by three model pairings of Gemma 4 26B and Qwen3.5 9B, then scored 0 to 1 by a Gemma 4 26B judge.\n\n## Results: the 4x larger Qwen3-Omni does not pull away\n\nUnder the paper's evaluation, Walrus reaches 94.7% word accuracy on FLEURS, ahead of Qwen3-Omni-30B at 93.8% and Whisper Large-v3 at 92.0%. On CS-FLEURS its character error rate is 7.23 versus 32.30 for Gemma 4. On LibriMix clean 2-mix it reaches 19.32 cpWER - half of Qwen3-Omni-30B's 37.04 and far below the dedicated VibeVoice-ASR at 59.34; comparable open SpeechLLMs produce no valid output at all in this setting. On audio understanding, MMAU-Pro reaches 71.92%, above Gemini-Flash-3.7 at 70.40% and GPT-4o-Audio at 52.50%; MMSU reaches 64.1% versus 61.4% for Audio-Flamingo-3; CVSS-C speech-to-speech translation scores 43.63 ASR-BLEU versus 41.27 for the 30B Qwen3-Omni. It does not win everywhere: FLEURS bidirectional speech translation sits at 32.28\u002F22.94, below Qwen3-Omni's 33.60\u002F25.67 - the paper concedes this one, claiming only better speaker similarity and prosody preservation.\n\n## The commercial variant's price is written in the licenses\n\nWalrus ships in two variants: Commercial-Ready (CR), trained only on commercially usable data and frozen at the Stage-2 checkpoint, and Research (R), trained on the full curriculum. Moving from CR to R lowers code-switching CER by roughly 19 points, internal entertainment ASR WER by about 9 points across 6 languages, and lifts MMAU-Pro by more than 20 points. In other words, the license terms literally price the capability gap - the most industry-relevant set of numbers in the whole report.\n\n## So what\n\nIn an era where backbones are becoming public infrastructure, this report turns \"data engineering is a primary driver of quality in multi-task speech and audio modeling\" into a controlled experiment: fix the architecture, attribute every gain to data. Two lessons for speech teams - the hardest tail cases (overlap, code-switching, entertainment audio) are solved with synthetic pipelines, not by waiting for public corpora; and the line between commercial and research models will increasingly be drawn by which data you are allowed to use, not by architecture. Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13689","sony-walrus-8b-speech-audio-foundation","2026-11-11T21:06:18Z","2026-11-11T21:06:23.248002Z","2026-11-11T21:06:23.248012Z",true,"agent",257,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"503fc4f0-755d-43f1-89cc-8d86b4cdaed9","7.6M参数土耳其语TTS:WER 1.09%却没人做过盲听","antalia-2-mini-turkish-tts-synthetic-data","2026-11-12T01:40:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"291c06d9-3435-4644-af2a-43bf91b54024","日语多音字97.13%读对:注音扰动加纯文本蒸馏","japanese-tts-polyphony-ruby-opd","2026-11-05T21:07:29+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"35e9f682-7952-4f6a-a8e9-7f8bdd47e791","IBM 语音三件套更新:2B 旗舰 ASR 精度开源第二","granite-speech-5-collection-update","2026-10-22T21:08:48+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"d83c4910-6dce-4171-8a44-9f68c9f841af","Qwen-Audio-3.1-ASR开源:转写自带说话人","qwen-audio-3-1-asr-speaker-diarization","2026-10-22T17:13:29+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"bbedb317-c221-43e3-b9a4-61798ed17b55","Voxtral 3:Mistral 语音智能体全栈,一轮对话不到一秒","voxtral-3-voice-agent-sdk","2026-10-21T23:11:24+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"9ee90f89-5d06-40dc-aeed-e06ebf292b0a","DEFINE 给 TTS 加口音旋钮:换腔调不换嗓","define-accent-control-zero-shot-tts","2026-10-05T15:06:46+00:00"]