On September 2, a minority report landed in the open-weights pool: the RWKV project published the RWKV7-G1j 13.3B checkpoint on Hugging Face under Apache 2.0. Minority, because the open-source ecosystem has converged almost entirely on the Transformer — while RWKV keeps betting on a different line: no self-attention, recurrence instead.
The mechanism: a fixed state instead of a growing KV cache
RWKV-7, codenamed Goose, is an attention-free recurrent language model. Instead of attending over an ever-growing context window, it processes tokens sequentially while carrying a fixed-size recurrent state. The direct consequence is the cost curve: memory and compute per token stay roughly constant during inference, independent of sequence length, which can make long-sequence generation cheaper than in comparable attention-based systems. Training does not give up parallelism — the model card states that training remains parallelizable, while recurrent decoding does constant-size work per generated token with respect to sequence length.
The checkpoint's specs: 13.269B parameters in the FLA build, 61 layers, hidden size 4,096, 64×64 heads, feed-forward size 16,384, a 65,536-token RWKV World tokenizer vocabulary, configured context length of 16,384 tokens, and BF16 weights. It is a dense design, not a mixture-of-experts, and its vocabulary covers multiple languages rather than English only.
The release shape: fitting into your existing stack
The engineering posture is pragmatic. The checkpoint ships in the flash-linear-attention (FLA) RWKV7 layout: install the library, import fla, and load it with the standard Transformers AutoModelForCausalLM — no stack replacement required. The model card includes a local validation record: configuration, tokenizer, chat template, BF16 weights, and Transformers loading were each verified, and all 2,016 model tensors load into RWKV7ForCausalLM with no missing, unexpected, or mismatched keys. The source checkpoint comes from BlinkDL/rwkv7-g1, with G1j identifying this revision. The page already lists 31 community finetunes and 122 downloads last month — small numbers, but evidence of real usage rather than another sink-after-release repository.
A notable honesty
Two self-disclosures in the model card stand out. First, it states plainly that this is a base language model, not a safety-aligned instruction-tuned assistant: the included chat template provides a conversational format, but the model may not follow instructions consistently. Second, the validation environment is described precisely: local validation was CPU-only, CUDA/Triton generation was not executed, runtime behavior depends on the specific GPU, CUDA, PyTorch, Triton, and FLA versions, and benchmark results should be cross-checked against the official RWKV implementation before being reported. Saying what was not validated, and which conclusions users must re-verify, is far more informative than the usual marketing-toned model card — and it is the right way for an alternative-architecture project to build trust.
So what
The ecosystem's architectural convergence is real: model sizes, MoE ratios, attention variants — most of the competition happens on the same Transformer chassis. The value of RWKV7-G1j is a maintained control group: it holds a fixed-size recurrent state instead of a cache that grows with the preceding sequence, and that contrast is the reason it deserves to exist. For teams working on long-sequence or cost-sensitive inference, this kind of checkpoint belongs in the comparison set; for the ecosystem, every version an alternative lineage ships adds one more answer to the question of what an inference cost curve can look like.
Model details and weights: https://huggingface.co/fla-hub/RWKV7-G1j-13.3B-20260831