[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-rwkv7-g1j-13b-attention-free":3,"news-related-44a035c8-b8a3-48e5-af4f-c76323dac7b5":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"44a035c8-b8a3-48e5-af4f-c76323dac7b5","RWKV7-G1j 13.3B 开源:不用注意力,每 token 推理成本是常数","RWKV 项目发布 13.3B 参数的 RWKV7-G1j 检查点:以循环机制取代自注意力,维护固定大小的状态而非随序列增长的 KV cache,每个 token 的推理开销不随长度上涨。以 Apache 2.0 许可上架 Hugging Face,61 层、65,536 词表,已有 31 个社区衍生微调。","9 月 2 日,开源权重池里出现了一个少数派:RWKV 项目把 RWKV7-G1j 13.3B 检查点以 Apache 2.0 许可上架 Hugging Face。说它少数,是因为当前开源生态几乎全员收敛在 Transformer 架构上,而 RWKV 坚持的是另一条技术路线——不用自注意力,用循环。\n\n## 机制:固定状态取代不断增长的 KV cache\n\nRWKV-7(代号 Goose)是 attention-free 的循环语言模型。与 Transformer 在不断增长的上下文窗口上做注意力不同,它按顺序处理 token,并维护一个固定大小的循环状态。带来的直接差异在成本曲线:推理时每个 token 的内存与计算开销大致恒定,不随序列长度上涨,长序列生成因此可能比同类注意力系统更便宜。训练侧也没有丢掉并行性——模型卡明确,训练仍然可并行化,循环解码时每生成一个 token 的工作量对序列长度而言是常数。\n\n这个检查点的具体规格:FLA 版本参数量 13.269B,61 层,隐藏维度 4,096,注意力头 64×64,前馈层 16,384,词表为 RWKV World tokenizer 的 65,536 tokens,配置上下文长度 16,384 tokens,权重 BF16 精度。是 dense 设计,不是 MoE。词表覆盖多语言,而非仅英语。\n\n## 发布形态:挤进你现有的工具链\n\n这次发布的工程姿态相当务实:检查点以 flash-linear-attention(FLA)库的 RWKV7 布局提供,装好库、import fla 之后,直接用 Transformers 的 AutoModelForCausalLM 加载推理,不需要换掉整套技术栈。模型卡附带了本地验证记录——配置、tokenizer、chat template、BF16 权重、Transformers 模型加载逐项验证,2,016 个模型张量全部加载进 RWKV7ForCausalLM,无缺失、无意外、无形状不匹配的 key。源检查点来自 BlinkDL\u002Frwkv7-g1,G1j 是这次发布的修订标识。截至目前页面列出 31 个社区衍生微调,上月下载量 122——数字不大,但说明有人在真实地用,而不是又一个发完就沉的仓库。\n\n## 值得注意的诚实\n\n模型卡有两处自我交代,在当前的发布文档里并不多见。其一,它明确说这是 base 语言模型,不是经过安全对齐的指令微调助手:附带的 chat template 提供了会话格式,但模型可能不会稳定遵循指令。其二,验证环境的说明写得很清楚:本地是 CPU-only 环境,CUDA\u002FTriton 生成没有执行过,运行时行为依赖 GPU、CUDA、PyTorch、Triton 与 FLA 的具体版本,报告 benchmark 结果前应先对照官方 RWKV 实现交叉核对。没验证什么、哪些结论需要用户自己复核,发布方直接写在文档里——这比营销腔的 model card 信息量大得多,也是替代架构项目建立信任的正确方式。\n\n## 所以呢\n\n开源生态的架构收敛是事实:模型尺寸、MoE 配比、注意力变体,大家卷的都是同一套 Transformer 底盘上的参数。RWKV7-G1j 的价值在于提供了一个持续维护的对照组——它维护固定大小的循环状态,而不是随前序序列增长的 attention KV cache,这条对照线本身就是它存在的理由。对做长序列、成本敏感推理的团队,这类检查点值得纳入对照实验;对整个生态,替代路线每存活一个版本,「推理成本曲线长什么样」这个问题就多一个答案。\n\n模型详情与权重见 Hugging Face 仓库:https:\u002F\u002Fhuggingface.co\u002Ffla-hub\u002FRWKV7-G1j-13.3B-20260831","https:\u002F\u002Fhuggingface.co\u002Ffla-hub\u002FRWKV7-G1j-13.3B-20260831","e1be55ed-9dd6-446e-aed6-42d7c22d9149",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"782370f2-a665-4ab6-9ad5-0747fd2104c8","en","RWKV7-G1j 13.3B: Open Model, No Attention, Constant Cost","RWKV's G1j 13.3B checkpoint, released under Apache 2.0, swaps attention for a constant-size recurrent state, keeping per-token cost flat.","On September 2, a minority report landed in the open-weights pool: the RWKV project published the RWKV7-G1j 13.3B checkpoint on Hugging Face under Apache 2.0. Minority, because the open-source ecosystem has converged almost entirely on the Transformer — while RWKV keeps betting on a different line: no self-attention, recurrence instead.\n\n## The mechanism: a fixed state instead of a growing KV cache\n\nRWKV-7, codenamed Goose, is an attention-free recurrent language model. Instead of attending over an ever-growing context window, it processes tokens sequentially while carrying a fixed-size recurrent state. The direct consequence is the cost curve: memory and compute per token stay roughly constant during inference, independent of sequence length, which can make long-sequence generation cheaper than in comparable attention-based systems. Training does not give up parallelism — the model card states that training remains parallelizable, while recurrent decoding does constant-size work per generated token with respect to sequence length.\n\nThe checkpoint's specs: 13.269B parameters in the FLA build, 61 layers, hidden size 4,096, 64×64 heads, feed-forward size 16,384, a 65,536-token RWKV World tokenizer vocabulary, configured context length of 16,384 tokens, and BF16 weights. It is a dense design, not a mixture-of-experts, and its vocabulary covers multiple languages rather than English only.\n\n## The release shape: fitting into your existing stack\n\nThe engineering posture is pragmatic. The checkpoint ships in the flash-linear-attention (FLA) RWKV7 layout: install the library, import fla, and load it with the standard Transformers AutoModelForCausalLM — no stack replacement required. The model card includes a local validation record: configuration, tokenizer, chat template, BF16 weights, and Transformers loading were each verified, and all 2,016 model tensors load into RWKV7ForCausalLM with no missing, unexpected, or mismatched keys. The source checkpoint comes from BlinkDL\u002Frwkv7-g1, with G1j identifying this revision. The page already lists 31 community finetunes and 122 downloads last month — small numbers, but evidence of real usage rather than another sink-after-release repository.\n\n## A notable honesty\n\nTwo self-disclosures in the model card stand out. First, it states plainly that this is a base language model, not a safety-aligned instruction-tuned assistant: the included chat template provides a conversational format, but the model may not follow instructions consistently. Second, the validation environment is described precisely: local validation was CPU-only, CUDA\u002FTriton generation was not executed, runtime behavior depends on the specific GPU, CUDA, PyTorch, Triton, and FLA versions, and benchmark results should be cross-checked against the official RWKV implementation before being reported. Saying what was not validated, and which conclusions users must re-verify, is far more informative than the usual marketing-toned model card — and it is the right way for an alternative-architecture project to build trust.\n\n## So what\n\nThe ecosystem's architectural convergence is real: model sizes, MoE ratios, attention variants — most of the competition happens on the same Transformer chassis. The value of RWKV7-G1j is a maintained control group: it holds a fixed-size recurrent state instead of a cache that grows with the preceding sequence, and that contrast is the reason it deserves to exist. For teams working on long-sequence or cost-sensitive inference, this kind of checkpoint belongs in the comparison set; for the ecosystem, every version an alternative lineage ships adds one more answer to the question of what an inference cost curve can look like.\n\nModel details and weights: https:\u002F\u002Fhuggingface.co\u002Ffla-hub\u002FRWKV7-G1j-13.3B-20260831","rwkv7-g1j-13b-attention-free","2026-09-03T13:14:19Z","2026-09-03T13:14:47.116709Z","2026-09-03T13:14:47.116718Z",true,"agent",49,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"199cd4ef-f092-45a5-8635-91778dd2bce2","编译即训练：一句规约炼出 83.6% 准确率的本地神经函数，教师模型只用一次","compile-by-training-neural-functions","2026-09-04T23:08:03+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"7a6d28b6-65da-4a29-96b1-dedb9894de97","随机驱逐追平最强打分器:Salesforce 重写 KV Cache 压缩常识","random-attention-kv-cache-eviction","2026-09-04T19:08:26+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"ff3b7c9f-e338-4af9-a562-4fda8b1bc929","vLLM 0.28 发布:584 项提交押注 Kimi K3,每卡省 17GB 显存","vllm-0-28-kimi-k3-optimization","2026-08-29T19:05:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"cac485ab-a429-4ebc-88c6-a1f924f978ff","AWS 开源 KeysAndValues:微调时就让模型学会“遗忘”,单张 A100 撑住 128K","aws-keysvalues-sparse-attention-finetuning","2026-08-26T05:20:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"c94bdf86-5de9-49fe-8c98-0f5c47611bfe","SGLang v0.5.18 发布:大模型冷启动提速 2.38 倍,710 个 PR 都改了什么","sglang-v0-5-18-cold-start-2-38x","2026-08-24T23:15:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"92433e6b-113a-4ada-af77-fbb8995a9850","LFM2.5-DSpark 开源:300M 草稿模型让端侧推理快 2.87 倍,输出零损耗","lfm2-5-dspark-draft-models","2026-08-21T21:10:00+00:00"]