What does it feel like to stuff an entire NES console into the weights of an LLM? Developer Drew Breunig's new open-source project nesformer answers that: a standard Qwen3.5 checkpoint that loads with Qwen3_5ForCausalLM.from_pretrained and needs no custom code — and in the GIF on the repo's front page, those weights are running Super Mario Bros. World 1-1. The NES's 6502 CPU, its 2 KB of work RAM, and 32 KB of cartridge program ROM all live inside the model's weights and internal state.
The most counterintuitive part: none of these weights were ever trained. A compiler computed every one of them. When a game runs, the model predicts the CPU's execution log one token at a time — most instructions take a single token — while a small host program plays the rest of the console around it: drawing the picture, reading the controller, firing the vblank interrupt.
The DeltaNet state matrix is the memory
The project stands on the shoulders of Qwenputer, in which Divyansh Agrawal first compiled a CHIP-8 and then a RISC-V CPU into Qwen3.5-4B. nesformer reuses that compiler and pushes the central trick to console scale: the model's Gated DeltaNet state is the machine's memory. Each DeltaNet head keeps a 128×128 state matrix that the delta rule updates at every token. With the gate held open and a one-hot key, an update overwrites exactly one row of the matrix, and a one-hot query reads it back — so every row is a memory cell, and the cartridge, RAM and registers all live in rows.
The checkpoint's spec sheet is fun reading: it reuses Qwen3.5-4B's head shapes (16 key heads, 32 value heads, dim 128) and MLP width of 9,216, with 7 Gated DeltaNet layers plus one all-zero attention layer, hidden size 768, and a 263,808-token vocabulary. Total parameters: 466.6M — 933 MB in bf16 — but only 9.3M are nonzero, so the whole weights package zips to 3.3 MB. The seven layers split the labor: layer 1's state holds the PC anchor, the A/X/Y registers and the flags; layer 3 fetches instructions; layer 5 reads and writes data; layer 7 emits the next token. Real game code averages just 1.1-1.3 tokens per instruction.
150-290x slower than the real thing — and exactly right
Speed is toy-grade, of course. The numba engine runs at about 1,650 tokens/s on one CPU core, the numpy engine about 210, and stock transformers about 10 — while a real NES would need roughly 700,000 tokens per second. The host also recognizes idle wait loops and cuts token consumption by 2.6-4.2x, but the whole thing still lands 150-290x slower than the real console: a frame takes 1-6 seconds, and 600 frames — ten seconds of NES time — take about 25 minutes.
Correctness, though, is maximal. In the free-running checks reported in the README, the traces generated by the weights are identical to a reference CPU: the test ROM Thwaite ran 1,916,527 tokens across 600 frames, and all 300 saved frames match a conventional 6502 emulator exactly; the ROM Full Game ran 2,645,885 tokens across 420 frames, with all 210 saved frames matching. The limitations are stated honestly, too: NROM cartridges only (mapper 0, 32 KB of program ROM), unofficial opcodes emit HALT, graphics are rendered scanline-by-scanline by the host rather than living in the weights, and there is no sound at all.
So what
The real takeaway is not "AI can play Mario" — the model never "learned" the game; its weights were simply compiled into a machine. The takeaway is an extreme demonstration of something deeper: the recurrent state of linear-attention architectures like Gated DeltaNet is genuinely addressable memory, and the delta rule is a write instruction. Weights don't have to be learned from data — a compiler can "write" a model too. For researchers asking how much structured state a model can carry inside, this is a beautiful upper-bound experiment; for everyone else, the project is MIT-licensed, currently at 229 stars, and runs on your own machine after a clone.
When weights can be compiled rather than trained, gradient descent stops being the only road to a model — which may be the most deliciously uncomfortable question nesformer leaves us with. Its predecessor Qwenputer is open-source too, and worth a bookmark.