No gradients, no training loops — just hand GPT-6 Astra a language model's weights as text and let it read and rewrite them. That is what Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab of MIT CSAIL call Weight-Space Programming, in a blog post published November 2. The headline result: Qwen3.5-0.8B's MATH-500 score climbs from 7.9% to 48.1%, with only 131,072 weights touched — 0.016% of the model's 0.8 billion parameters.

Not fine-tuning — diagnosis, then surgery

Astra's tool is a Python session with the weights loaded: it may read them and run diagnostic forward passes, but it gets no gradient information. Practice material is 7,469 GSM8K problems and 7,275 MATH problems (never the test sets), with a budget of 10, 25, or 50 forward passes. Astra homes in on the thinking-loop problem that Qwen's own model card warns about: the loop direction concentrates in layer 19's MLP, with neuron 11 of that layer as its strongest writer. Removing the direction lifts MATH-500 from 7.9% to 48.1%, and IFEval, which measures instruction following, rises from 21.0% to 62.3% — a spillover gain beyond math. Total edit size: 131,072 weights.

Reading weights: naming the game from 17,253 numbers

The reading experiments start with toy networks. Four grid games (Breakout, Pong, Pac-Man, Freeway) share the exact same MLP shape (500→32→32→5, 17,253 weights), each trained with PPO for 1 billion steps. Fed only the raw weights as text, with no list of candidate games, Astra names all four correctly; for Breakout it reports 99% confidence and its reasoning explicitly traces the bottom paddle to row 8 weights and the brick rows to rows 1–3. Diffusion models work too: three DDPMs trained to draw circles, squares, and triangles are read from the geometry of the output layer, and Astra draws each shape as an outline — a mistake the authors take as evidence that it reads weight structure rather than secretly computing forward passes. The harshest round: only the middle four transformer blocks of a 7.2M character-level model are shown, with input and output layers stripped away, so the network cannot run on any text. Astra still names the training language (Python, romaji Japanese, English) and, in both attempts, identifies every copying head — matching the measured positions exactly, with zero false positives.

Writing weights: zero training, 517 points in one shot

The writing side is more radical. In a Kinetix environment, a car trained with PPO for 1 billion steps is stuck at a ramp (a local optimum). In a single reply with no tools, Astra changes 6,073 of the network's 171,664 weights (3.5%), and the car then clears all 1,024 test episodes. Writing from scratch goes further: across three MinAtar games, given only the rules, the source code, and a network initialized to all zeros — no data, no loss, no gradients, no tools, not even a trial rollout — Astra writes the full weight tables as text, and the network goes straight to evaluation. Breakout scores 517.4, versus 15.8 for PPO trained 100 million steps and 25.3 at 500 million steps — a 20x gap against the latter; Seaquest reaches about 60% of 500M-step PPO, and Asterix comes close to parity.

The limits, as the authors state them

The post is honest about failures. Given two fine-tuned copies of Gemma 3 270M — one trained to be sycophantic, echoing whatever the user suggests — Astra cannot tell them apart from weights alone; only with an attention-pattern tool does it pick the sycophant in all three pairs. More telling is the reverse result: Gemma 3 270M rarely loops (8.5% of answers hit the token limit, versus 93% for Qwen3.5), and Astra cannot improve it — MATH-500 moves from 6.8% to at most 7.7%, a gain the authors themselves call not established. Their conclusion is blunt: weight editing can remove an obstacle that hid an ability the model already had; it cannot add an ability the model lacks.

So what?

The authors' forward-looking claim: deep learning has squeezed its inductive biases into tiny symbolic optimizers like Adam and Muon, and soon our weights will be natively optimized by LLMs that reason about them directly. This is an optimization story and an interpretability story at once — Astra reading weights is itself a form of white-box attribution. Artifacts are out: the edited model is on Hugging Face (RikiyaT/Qwen3.5-0.8B-weight-space-programming), experiment logs are on GitHub (RikiyaT/lm_understand-logs), and an online demo is linked from the post. When the optimizer is itself a reasoning model, where exactly does the boundary between training and programming sit? Source: Sparks of Weight-Space Programming