[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-e-gaborformer-strictly-causal-streaming-asr":3,"topics-all":38,"news-related-aa23f73d-0db7-4b6e-a70e-1d72bc4f0127":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"aa23f73d-0db7-4b6e-a70e-1d72bc4f0127","严格因果ASR:Gabor字典卷积把流式状态钉在19MB","arXiv论文提出E-Gaborformer:Gated DeltaNet-2递归层配滑窗注意力,卷积核锁在Gabor字典里,让编码器严格因果。每路通话状态恒定约19MB,同规模因果注意力十分钟累积超1GB。生产规模0.6B转写器WER比同尺寸Nemotron 3.5 ASR Streaming低28%到31%。","语音Agent要在用户说话时把字转出来,还得扛住成千上万通并发电话。让模型不看未来音频靠给注意力加掩码,但掩码后的自注意力仍要缓存过去的key和value。论文给了个具体的账:18层、d=1024的编码器在80毫秒帧上,每分钟音频、每个调用方吃掉约110MB的fp32缓存;同规模的无界因果注意力十分钟就能累积超过1GB。状态随通话时长线性上涨,服务器成本也跟着涨。\n\n论文(E-Gaborformer,[arXiv:2611.08092](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.08092),作者 Nir Krakowski、Tamir Cohen、Ohad Zohar、Chen Zarfati的解法是:不让注意力承担记忆。\n\n## 把过去压进固定大小的状态\n\nE-Gaborformer保留E-Branchformer的模块结构,换了三样东西。\n\n第一样是全局分支。层按GGGS模式排列——每四层里三层是Gated DeltaNet-2递归层(GDN-2),第四层才是128帧窗口的因果滑窗注意力。递归层每帧只更新一次固定大小的矩阵状态(论文里是64×64);注意力层只保留最近128帧(约10.24秒)的环形缓冲,提供压缩状态给不了的精确回忆。\n\n第二样是卷积核。论文没让网络自由学习卷积核,而是锁在一组固定因果Gabor原子张成的空间里:16个单位范数原子,由2个高斯和7个对数间隔的波包(每帧0.03到0.4个周期,各拆成余弦和正弦)组成。核长31、帧长80毫秒时,一个滤波器跨2.48秒,波包中心落在0.375到5Hz,覆盖短语和较慢音节调制的频段。每个通道只学16个系数而不是31个数,因果性自动成立——包络在当前帧达峰后向过去衰减。代价是字典病态(Gram条件数约8×10⁶)。\n\n## 一个反直觉的坑:递归分支一开始是哑的\n\n第一轮18个带GDN-2的因果训练中,有4个整整一个epoch都没走出初始CTC平台期——模型只吐blank,损失平躺。定位后发现,GDN-2分支的输出RMS只有0.005到0.006,同模型注意力分支是0.28到0.38。原因在输出路径:递归输出小到落在归一化epsilon以下,RMSNorm没法放大它,网络干脆学会绕过递归分支。\n\n修法很轻:把SiLU输出门换成sigmoid(起始值0.5),再把RMSNorm的epsilon从10⁻⁵降到10⁻⁶,都不加参数。GDN-2分支起始RMS回到约0.09,之后36个带GDN-2的因果训练全部走出平台期,改之前是18个里停4个。\n\n这个细节值得记住:混合架构的失败模式常常不在架构本身,而在某个分支被数值范围掐死。\n\n## 受控对比:和同体量E-Branchformer硬碰\n\n论文在LibriSpeech-100上做了严格因果的对照:所有模型共用同一套前端和训练配方,只报告开发集。研究版模型12层、d=256、27.6M参数,对阵26.8M参数的E-Branchformer(ESPnet移植版)。三个epoch时,E-Gaborformer在dev-clean\u002Fdev-other上是26.2\u002F44.5,对照组27.1\u002F45.1,三个种子在dev-clean上分别领先1.4、0.2和1.2个百分点;十个epoch后两边dev-other都是39.2%,dev-clean上还低0.3点。\n\n达到某个给定dev-clean WER比对照组早5%到18%。架构筛选跑了36种编码器配置、约160次训练。拆开看:两处Gabor卷积换成无约束深度卷积,dev-clean WER涨1.2点;所有注意力层去掉,涨1.1和1.4点。3:1混合和1:1精度相当但注意力层少一半。\n\n## 生产规模:19MB状态和Nemotron的对比\n\n放大到生产规模:18层、d=1024,14层GDN-2配4层滑窗注意力,编码器598M参数,配RNN-T解码器,整模626M,和Nemotron 3.5 ASR Streaming(0.6B)差不多大——后者是NVIDIA公开的600M参数流式转写模型,基于Cache-Aware FastConformer-RNNT架构。训练用了约1.2万小时希伯来语和9.6万小时英语,前10750个更新里Nemotron还当了老师。\n\n状态这边:GDN-2状态3.7MB、注意力窗口4.2MB、Gabor卷积缓存8.8MB,加上预测网络约2MB,共约19MB,和通话长度无关。\n\n对比结果来自英文语音客服真实流量(24段录音、规范化后1214个参考词):Nemotron 3.5 ASR Streaming的WER是14.0(因果)和13.3(480毫秒前瞻),E-Gaborformer两种模式都是9.6。因果模式下低31%,Nemotron开前瞻后差距是28%。前瞻对Nemotron有帮助(降0.7点),对E-Gaborformer毫无损失,因为18层里只有4层能看到未来帧。\n\n论文自己给了限定:这批流量正是E-Gaborformer的训练数据,Nemotron还当过早期老师。所以这更接近同一团队的工程自证,不是独立评测的中立排名。\n\n和上一版自家生产模型相比,希伯来语三组测试集的WER相对下降11%到20%(通用12.3→10.9,困难14.2→11.5,人名密集10.2→8.2),英语持平,只有地址、URL和邮箱那组涨了7%。\n\n## 训练侧:把RNN-T的损失算对\n\n论文还顺手治了RNN-T训练慢:每个格点只需要两个log概率(发下一个token或者发blank),却要在完整词表上跑log-softmax。做法是把前向-后向递归放到两类格上,梯度用闭式解写出来,损失和梯度与完整格完全等价;配套把输出层补齐到128的倍数、深度卷积改写成Toeplitz形式、单卡每次更新从7.55秒降到2.90秒(2.6倍)。\n\n## 所以呢\n\n这篇论文的价值不在「又一个更低WER的ASR」,而在于它把「流式状态」当成了一等公民。过去几年注意力越来越强,代价是「状态随音频长度增长」被默认接受了。E-Gaborformer的选择是:递归层扛长程记忆,注意力层补精确回忆,固定字典的卷积核压小特征提取的搜索空间,换来一个和通话长度无关的状态大小。\n\n对做Agent的人,这个换算很直接:长通话下状态不再涨,就是并发成本不再涨。19MB听起来不大,但乘上并发数乘上通话时长,就是一条完全不同的成本曲线。\n\n论文列出的下一步也很诚实:更长的受控训练、发射延迟测量、多分钟长流评测——恒定状态最大的价值场景,恰恰是它还没测的那个。\n\n那么问题来了:如果递归状态、注意力窗口、卷积历史都是固定的,「上下文」这个概念在流式语音模型里,到底还剩下什么?","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.08092","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"c4455801-0273-477d-9e92-00b4addfa439","asr",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"da116f20-377f-449f-9382-821d56f6fa64","streaming",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"f5987eaa-24a5-4975-807d-5fffe1c6c8e2","en","Strictly Causal ASR: Gabor Convolutions Pin State at 19MB","Strictly causal ASR with Gabor-dictionary convolutions keeps streaming state at 19MB and beats same-size Nemotron 3.5 on WER by 28-31%.","A voice agent has to transcribe people while they are still talking, and it has to do that across thousands of concurrent calls. Making a model ignore future audio usually means masking attention, but masked attention still caches every past key and value. The paper puts a number on that: an 18-layer, d=1024 encoder at 80ms frames accumulates roughly 110MB of fp32 cache per minute of audio per caller, and an unbounded causal attention encoder of the same size passes 1GB after ten minutes. State that grows with call length means server cost that grows with call length.\n\nE-Gaborformer (arXiv:2611.08092, https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.08092, by Nir Krakowski, Tamir Cohen, Ohad Zohar and Chen Zarfati) tries to solve both constraints at once, by refusing to let attention carry the memory.\n\n## Compressing the past into a fixed-size state\n\nE-Gaborformer keeps the E-Branchformer block but changes three things inside it.\n\nFirst, the global branch. Layers follow a GGGS pattern: three Gated DeltaNet-2 recurrent layers (GDN-2) for every fourth layer of causal sliding-window attention over a 128-frame window, about 10.24 seconds. The recurrent layer updates one fixed-size matrix state per frame, 64x64 in all models, while each attention layer keeps a ring buffer of its most recent 128 frames to provide the exact recall a compressed state cannot.\n\nSecond, the convolution kernels. Instead of letting the network learn them freely, the paper locks them into the span of a fixed dictionary of causal Gabor atoms: 16 unit-norm atoms made of two Gaussians and seven log-spaced wave packets, frequencies from 0.03 to 0.4 cycles per frame, each split into cosine and sine. With a kernel of 31 and 80ms frames, one filter spans 2.48 seconds and the wave packets center between 0.375 and 5Hz, the band of phrasal and slower syllabic modulation. Each channel learns 16 coefficients instead of 31 numbers, and causality is automatic because the envelope peaks at the current frame and decays into the past. The dictionary is ill-conditioned, with a Gram condition number around 8x10^6.\n\n## The counterintuitive bug: the recurrent branch starts mute\n\nIn the first round of 18 causal runs with GDN-2 layers, four never left the initial CTC plateau within an epoch: the model emitted only blanks and the loss stayed flat.\n\nThe cause was the output path. The GDN-2 branch output had an RMS of only 0.005 to 0.006, against 0.28 to 0.38 for the attention branch of the same model. The recurrent output fell below the normalization epsilon and RMSNorm could not rescale it, so the network learned to work without the recurrent branch.\n\nThe fix is cheap: replace the SiLU output gate with a sigmoid (starting at 0.5) and lower the RMSNorm epsilon from 1e-5 to 1e-6. Neither change adds parameters. The branch then starts at an RMS of about 0.09, and all 36 later causal runs with GDN-2 layers left the plateau, against 4 of 18 before.\n\nThat detail is worth keeping: in hybrid architectures the failure mode is often not the architecture, but one branch being quietly killed off by its numerical range.\n\n## A controlled comparison against E-Branchformer\n\nThe paper runs a strictly causal controlled study on LibriSpeech-100, where every model shares one front end and one recipe and only development sets are reported. The study model has 12 layers at d=256 and 27.6M parameters, against a 26.8M-parameter E-Branchformer ported from ESPnet. After three epochs, E-Gaborformer scores 26.2\u002F44.5 on dev-clean\u002Fdev-other versus 27.1\u002F45.1 for the baseline, leading by 1.4, 0.2 and 1.2 points on dev-clean across three seeds; after ten epochs both reach 39.2% on dev-other and E-Gaborformer stays 0.3 points lower on dev-clean.\n\nReaching a given dev-clean WER comes 5 to 18% sooner than the baseline, across an architecture screen of 36 encoder configurations and roughly 160 training runs. Replacing both Gabor convolutions with unconstrained depthwise kernels costs 1.2 points on dev-clean, and removing all attention layers costs 1.1 and 1.4. The 3:1 hybrid matches the 1:1 hybrid while using half the attention layers.\n\n## At production scale: 19MB and the Nemotron comparison\n\nScaled up, the production model has 18 layers at d=1024, 14 GDN-2 layers against 4 sliding-window attention layers, and a 598M-parameter encoder with an RNN-T decoder, for 626M parameters in total, roughly the size of Nemotron 3.5 ASR Streaming (0.6B), NVIDIA's public 600M-parameter streaming recognizer built on a Cache-Aware FastConformer-RNNT architecture. Training used about 12k hours of Hebrew and 96k hours of English, and during the first 10,750 updates Nemotron served as a teacher.\n\nOn the state side: 3.7MB of GDN-2 states, 4.2MB of attention key\u002Fvalue windows, 8.8MB of Gabor convolution caches, plus about 2MB for the prediction network, totaling roughly 19MB regardless of call length.\n\nThe comparison comes from real English voice-agent traffic, 24 recordings with 1,214 reference words after normalization. Nemotron 3.5 ASR Streaming scores 14.0 causally and 13.3 with 480ms lookahead, while E-Gaborformer scores 9.6 in both modes. That is 31% lower causally, and 28% lower once Nemotron gets its lookahead. Lookahead helps Nemotron by 0.7 points and costs E-Gaborformer nothing, because only 4 of its 18 layers can see future frames at all.\n\nThe paper qualifies that comparison itself: this traffic is what E-Gaborformer was trained on, with Nemotron having served as an early teacher. It is closer to an engineering self-demonstration in one setting than to an independent ranking.\n\nAgainst the authors' previous production model, Hebrew WER drops 11% to 20% relatively across three of four Hebrew sets (general 12.3 to 10.9, hard 14.2 to 11.5, name-heavy 10.2 to 8.2) while English holds steady. The one regression is the addresses, URLs and emails set, up 7%.\n\n## Fixing the RNN-T training math\n\nThe paper also treats a long-standing RNN-T training inefficiency: a standard implementation runs its dynamic program over the full-vocabulary log-softmax even though the loss needs only two log-probabilities per lattice node, either emitting the next token or emitting blank.\n\nThe fix moves the forward-backward recursion onto a two-class lattice and derives the logit gradient in closed form, leaving loss and gradients identical to the full lattice. Supporting optimizations include padding the output layer to a multiple of 128 rows so the 13,101 outputs return to tensor-core shapes, rewriting depthwise convolutions in Toeplitz form, and repacking the utterances of every micro-batch of one update by length, raising the fill rate from 61% to 80-83%. Per update time falls from 7.55 to 2.90 seconds on one GPU, a 2.6x speedup, and from 9.0 to 3.58 seconds, or 2.5x, across three GPUs with live data.\n\n## So what\n\nThe value of this paper is not another lower ASR WER. It is that it treats streaming state as a first-class design goal. Attention has grown stronger, and with it the default assumption that state grows with audio length. E-Gaborformer chooses differently: recurrence carries long-range memory, attention supplies exact recall, and a fixed dictionary of convolution kernels shrinks the feature-extraction space, for a state size that does not depend on call length.\n\nFor people building agents the trade is direct: in long conversations, state that stops growing means concurrency cost that stops growing. Nineteen megabytes sounds small, but multiply it by concurrency and by call duration and you get a completely different cost curve.\n\nThe paper's own next steps are honest: longer controlled training runs, emission-latency measurements, and evaluation on multi-minute streams, since the constant-state design matters most exactly in the scenario it has not yet tested.\n\nWhich raises the question: if the recurrent state, the attention window and the convolution history are all fixed, what does the idea of context even mean in a streaming speech model?","e-gaborformer-strictly-causal-streaming-asr","2026-11-07T11:20:00Z","2026-11-07T11:20:31.392087Z","2026-11-07T11:20:31.392101Z",true,"agent",136,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"09df4ac8-eede-4a9c-b488-0a577b589417","0.6B语音模型不生成答案,只回答是非题","marimba-typed-speech-decision-frozen-asr","2026-11-17T01:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"f08a17d3-2856-46ca-a104-27174e709b8a","TTS模型自己当编码器:参数省48%,三基准双胜","rse-tts-shared-reference-encoding","2026-11-15T19:40:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"fbce7a2f-800a-4af2-9e30-932ab3ed2a53","语音识别学会抢跑:29M小模型提前90ms出词","earlybird-29m-predictive-streaming-asr","2026-11-11T13:30:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"cb1044d5-dc15-4f72-b720-e4a110f939e9","论文实测:视频KV缓存2比特量化,主体一致性92.02","temporal-rotation-2bit-kv-cache-video-diffusion","2026-11-21T21:30:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"abd8eb9f-c106-4899-bf24-b6bbed3da5e3","EPFL论文实测:注意力砍维度而非砍token,只算三成点积","hisa-hidden-dimension-sparse-attention-rope","2026-11-20T09:00:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"4544ed60-b5fb-41b6-9fee-dd2f59012547","隐状态转个方向:直答46到85,超CoT省九成token","cot-hidden-state-directions-direct-answers","2026-11-19T13:30:00+00:00"]