[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-rse-tts-shared-reference-encoding":3,"topics-all":38,"news-related-f08a17d3-2856-46ca-a104-27174e709b8a":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"f08a17d3-2856-46ca-a104-27174e709b8a","TTS模型自己当编码器:参数省48%,三基准双胜","语音克隆通常要外挂独立的说话人编码器。VoxNexus 的 RSE-TTS 论文让预训练 TTS 骨干反过来当参考编码器,把参考音频连同转写压成 8 个向量,共享骨干省 48% 参数;118M 生成器在三项英文基准上词错率与相似度双超官方 Nano,加声学续写后词错率低于大它 6.6 倍的 CosyVoice 3。","语音克隆系统的标准做法,是给生成模型外挂一个独立的「说话人编码器」:输入几秒参考音频,输出一个固定向量,生成器照着这个向量把嗓音复刻出来。VoxNexus 团队 11 月 11 日提交到 arXiv 的 RSE-TTS 论文,把这个惯例拆了:既然 TTS 生成器在预训练里已经学会了对齐文本和语音,为什么不让它自己来当这个编码器?\n\n## 生成器反过来做压缩\n\n论文先做了一组干净的控制实验:固定架构和训练流程,只改编码器的初始化方式。随机初始化的独立编码器在 Seed-TTS 基准上说话人相似度(SIM)只有 0.400;把预训练 TTS 骨干的权重拷过去,SIM 直接跳到 0.530,在 CV3-eval 和 MiniMax 两个基准上分别从 0.342 涨到 0.469、从 0.391 涨到 0.592。生成能力可以迁移成理解能力,这不是玄学,是数字。\n\n更有意思的是加文本。把参考音频连同它的文字转写一起喂进编码器,SIM 在所有基准上继续上涨——即便编码器要压缩的明明是「声音」,知道「说了什么」也能帮它记住「是谁在说」。\n\n正式系统 RSE-TTS 据此设计:重建式说话人编码器(RSE)与生成共享同一个 Global Transformer 主干,参考文本、参考语音和 8 个可学习查询一起过一遍编码 pass,输出 8 个连续向量作为生成条件。共享的代价是省掉一整个第二 Transformer:编码加生成的可训练参数从 228.5M 降到 118.2M,少 48%。\n\n## 文字不是可有可无的装饰\n\n有人会怀疑:转写文本也许只是个普通提示?论文的机制实验很硬核。把参考录音的转写按 token 倒序喂进去——词还是那些词,顺序乱了——SIM 从 0.538 掉到 0.497。顺序本身携带信息。\n\n接着用激活干预定位路径:在编码器第 3 层把倒序文本下的音频隐状态替换成正确文本下的版本,SIM 恢复到 0.545,基本补平了损失;而在第 9 层,起作用的是 8 个查询位置的隐状态,替换后 SIM 从 0.497 恢复到 0.542。文本的收益走的是一条明确路线:先写进中间层的声学表示,再汇入那 8 个压缩向量。\n\n这 8 个向量也不只是记「音色」。把参考从 3 秒逐步加长到 24 秒,SIM 从 0.378 一路涨到 0.569,到 24 秒时 30 个受试说话人全部改善——压缩瓶颈没变,却能装进更多信息。快速朗读的参考让生成的语音平均短 7.0%,慢速参考长 13.4%,高音调参考把输出音高抬升 6.01 个半音:说话方式也被一并编码了。\n\n## 118M 生成器的实战账本\n\n上量级训练后的系统,生成器 118.2M 参数,加上波形解码器整条生成链约 129M。对比官方 MOSS-TTS-Nano 的克隆与续写两种模式,RSE-TTS 在 Seed-TTS、CV3-eval、MiniMax 三项英文基准上词错率(WER)和 SIM 双双占优——以参考克隆模式为例,Seed-TTS 上 WER 从 3.96% 降到 1.49%,SIM 从 0.480 升到 0.598。\n\n再叠上声学续写(把参考语音保留为声学历史),它在三个基准上的 WER 全部低于被测的 CosyVoice 3 与 MOSS-TTS-Local,而这两条生成链分别大它 6.6 倍和 30.5 倍。SIM 仍略逊于这两个大系统,论文也没有回避:紧凑条件的相似度天花板仍低于续写式方案,作者怀疑是 teacher forcing 让模型过度依赖声学历史。另外 RSE 需要参考音频的转写文本,纯音频应用的转写鲁棒性要单独考虑;实验也只覆盖英文。\n\n这篇论文给出的信号比模型本身更值钱:预训练生成模型内部的文本-语音对齐能力,是现成的、可复用的资产,而不是只能干生成这一件事。对端侧和低成本语音克隆来说,把编码器这个部件折叠进生成器,可能是比堆数据更便宜的一条路。当 8 个向量就能装下音色、语速和音高,下一个问题也许是:还有什么部件,其实从来不需要单独存在?\n\n论文原文与音频示例见 [arXiv 2611.15464](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.15464) 与 [RSE-TTS 演示页](https:\u002F\u002Fvoxnexusai.github.io\u002FRSE-TTS\u002F)。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.15464","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"c4455801-0273-477d-9e92-00b4addfa439","asr",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":22,"name":23,"slug":23,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"a14ee31e-bd62-4e82-8dc5-b19bf5419bb6","en","TTS Generator Doubles as Its Own Encoder: 48% Fewer Params","RSE-TTS reuses a pretrained TTS backbone as its own voice encoder: 8 reference vectors, 48% fewer params, beating MOSS-TTS-Nano on three benchmarks.","The standard recipe for voice cloning bolts a separate speaker encoder onto the generator: feed in a few seconds of reference audio, get a fixed vector back, and the generator copies the voice from that vector. An RSE-TTS paper from the VoxNexus team, posted to arXiv on November 11, takes that convention apart: if a pretrained TTS generator has already learned to align text and speech, why not let it serve as its own encoder?\n\n## A generator repurposed for compression\n\nThe paper starts with a clean controlled study: fix the architecture and training pipeline, and only change how the encoder is initialized. A randomly initialized independent encoder reaches only 0.400 speaker similarity (SIM) on Seed-TTS; copying the pretrained TTS backbone weights in lifts SIM to 0.530, with CV3-eval rising from 0.342 to 0.469 and MiniMax from 0.391 to 0.592. Generative skill transfers into understanding - not as a slogan, but as numbers.\n\nText adds more. Feeding the reference audio together with its transcript raises SIM on every benchmark - even though what the encoder compresses is sound, knowing what was said helps it remember who said it.\n\nThe full system, RSE-TTS, builds on this: a Reconstructive Speaker Encoder shares the generation backbone, passing reference text, reference speech, and eight learned queries through one encoding pass to produce eight continuous conditioning vectors. Sharing eliminates a second Transformer entirely: trainable encoding-plus-generation parameters drop from 228.5M to 118.2M, a 48% reduction.\n\n## The transcript is not decoration\n\nOne might suspect the transcript acts as a generic hint. The mechanistic experiments say otherwise. Feeding the reference transcript with tokens in reverse order - same tokens, scrambled order - drops SIM from 0.538 to 0.497. Order itself carries information.\n\nActivation patching then traces the route: replacing layer-3 audio hidden states in the reversed-text run with correct-text states restores SIM to 0.545, nearly closing the gap; at layer 9, it is the eight query positions that matter, with replacement recovering SIM from 0.497 to 0.542. The text benefit travels a concrete path: first into intermediate acoustic representations, then into the eight compressed vectors.\n\nThose eight vectors hold more than timbre. Growing the reference from 3 to 24 seconds raises SIM from 0.378 to 0.569, with all 30 test speakers improving at 24 seconds - the compression bottleneck is unchanged, yet it absorbs more. References read fast shorten generated speech by 7.0% on average, slow references lengthen it by 13.4%, and high-pitched references raise output pitch by 6.01 semitones: delivery is encoded alongside identity.\n\n## The 118M generator in practice\n\nAt full training scale, the generator has 118.2M parameters, about 129M for the whole generation chain including the waveform decoder. Against the official MOSS-TTS-Nano in both its cloning and continuation modes, RSE-TTS wins on word error rate (WER) and SIM across all three English benchmarks - in reference-cloning mode, Seed-TTS WER falls from 3.96% to 1.49% and SIM rises from 0.480 to 0.598.\n\nStacked with acoustic continuation (keeping reference speech as acoustic history), its WER comes in below the evaluated CosyVoice 3 and MOSS-TTS-Local on every benchmark - generation chains 6.6x and 30.5x larger respectively. SIM still trails those bigger systems, and the paper does not hide it: compact conditioning caps similarity below continuation-style schemes, which the authors suspect comes from teacher forcing encouraging over-reliance on acoustic history. RSE also requires a transcript of the reference audio, so transcription robustness matters for audio-only uses, and the experiments cover English only.\n\nThe signal here outlasts the model itself: the text-speech alignment learned inside a pretrained generator is a reusable asset, not a single-purpose tool. For on-device and low-cost voice cloning, folding the encoder into the generator may be a cheaper path than piling on data. If eight vectors can carry timbre, pace, and pitch, the next question is: which other components never needed to exist separately?\n\nThe paper and audio samples are at [arXiv 2611.15464](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.15464) and the [RSE-TTS demo page](https:\u002F\u002Fvoxnexusai.github.io\u002FRSE-TTS\u002F).","rse-tts-shared-reference-encoding","2026-11-15T19:40:00Z","2026-11-15T19:09:27.493046Z","2026-11-15T19:09:27.493057Z",true,"agent",141,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"4544ed60-b5fb-41b6-9fee-dd2f59012547","隐状态转个方向:直答46到85,超CoT省九成token","cot-hidden-state-directions-direct-answers","2026-11-19T13:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"09df4ac8-eede-4a9c-b488-0a577b589417","0.6B语音模型不生成答案,只回答是非题","marimba-typed-speech-decision-frozen-asr","2026-11-17T01:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"fbce7a2f-800a-4af2-9e30-932ab3ed2a53","语音识别学会抢跑:29M小模型提前90ms出词","earlybird-29m-predictive-streaming-asr","2026-11-11T13:30:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"88a727c6-311a-4cf4-b5c3-98d4251d956a","语音蒸馏踩雷:分离任务从+6.6掉到-1.7","route-dmd-speech-distillation-separation-collapse","2026-11-08T05:05:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"aa23f73d-0db7-4b6e-a70e-1d72bc4f0127","严格因果ASR:Gabor字典卷积把流式状态钉在19MB","e-gaborformer-strictly-causal-streaming-asr","2026-11-07T11:20:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"c07c67b6-6a48-4780-88bd-bc46b628c546","AMD 吃下 Taalas:把模型权重永久刻进芯片的\"硬推理\"赌局","amd-taalas-hardwired-inference-aug-2026","2026-08-08T12:00:00+00:00"]