A 0.8B model doing addition, long division, square roots and greatest common divisors in your browser, every digit computed by the model itself — no calculator, no tool call — and this is not a benchmark score but a proof. That is what Sketchpad, posted on arXiv on 3 November, delivers.

The demo sounds like a cute trick: hand an open-weight vision-language model a sheet of paper and teach it long multiplication the way a person does it with a pencil. The real contribution lies elsewhere — it converts a "long-arithmetic accuracy" metric that can only be estimated by sampling into a finite, checkable, formally provable property.

The hard part is not length, it is that errors are silent

Long arithmetic is pushed forward by chain of thought, changed data formats and changed positional encodings, each round pushing further the length at which models still get it right. But they share one property — on an unseen input, correctness is only an estimate, and the errors are silent.

The paper's comparison is the footnote: ten models, 220 pre-registered problems (sums of 10 to 1,000 digits and products of 2- to 20-digit numbers, output token budget 32,768), run on 31 October. GPT-6.1 Sol answered 217 correctly, Claude Opus 5.5 answered 216, Gemini 4 Argon answered 207, with every error landing on 300-digit and 1,000-digit sums. The three wrong answers from GPT-6.1 Sol were complete answers, same format and length as the correct ones, with the token budget nowhere near exhausted — nothing in the output betrays them.

Tool use is the other route: tools are exact, but the computation moves outside the network. The paper looks for a third route — to guarantee correctness on every input, behaviour on unboundedly many inputs must be pinned down by finitely many facts.

One sheet, 2,446 cells

Three parts: a sheet holding all state, a model mapping a local view of the sheet to one step's outputs, and a runtime that moves the view and writes the outputs back. The key constraint comes first: the runtime writes no digit of a result. It only copies digits, moves the view, and decides when to stop. Every digit is produced by the model.

The model sees three cells at a time. Each cell is 32 pixels, rendered as a 32 × 96 strip read by the vision encoder as 3 image tokens. The glyph set is fixed at 11 (blank plus 0 to 9), so the set of frames is finite and enumerable. Each operation gets a table listing every frame it can encounter with its correct outputs: column addition (SUM) 242 rows, the five operations of long multiplication 1,553 rows, subtraction (SUB) 242 rows, estimation of a quotient digit (EST) 400 rows, decrement of an estimate (DEC) 9 rows. That is 2,446 rows — the entirety of the possible situations, written out as a table rather than a probability.

The base model is Qwen3.5-0.8B: 24 layers, hidden size 1,024, with 6 layers of full attention and 18 of Gated DeltaNet. What was trained is a rank-16 LoRA on 150 projection matrices, 10,223,616 parameters, 1.2% of the model's 873,438,784. Training ran 19.8 minutes on a single H100, passing every row at step 1,200 and stopping at step 1,600.

The certificate: measured on the real device, proved in Lean

The "margin" is the certificate's yardstick: the smallest lead of a correct token over any wrong token, computed over the full 248,320-token vocabulary, passing if above zero. On the deployment device — the GPU of an Apple M4 laptop, running WebGPU in a browser at 16-bit precision — all 2,446 rows pass, the smallest margin being 3.02 logits, so the certificate tolerates any change of the logits smaller than 1.51. In 32-bit PyTorch on the server it is 3.96. Why margins at all: a 4-bit quantised decoder once moved some margins of a preliminary adapter from +9.4 to −8.9, flipping sign outright.

Three Lean theorems pin this down. Theorem 1 says correctness on the tables is equivalent to correctness at every length. Theorem 3 says a margin large enough absorbs numerical perturbation. Theorem 2 is the harshest: if a step may read the steps before it — chain of thought — no finite test suffices.

The score: 220 against 217, but do not celebrate yet

Those 220 problems ran in PyTorch on the server, not in the browser: batched lockstep, using 64,700 views and 1,001 forward passes. Sketchpad answered all 220. Five open-weight models landed between 149 and 199, mostly exhausting the token budget on long sums and long products; Qwen3.5-27B answered 121; the base model answered 23 without chain of thought and 3 with it.

The paper supplies its own cold water: each model was run once per problem, so a difference of a few problems between two models is within sampling variation. By that standard, the three problems between 220 and 217 are not news. The number to read lies elsewhere: the bare base model scores 23, and the same base model with the sheet attached scores 220. What changed was not model scale. It was the sheet.

The boundary: the certificate covers the model, not the runtime

Theorem 1 splits the responsibility in two: the model's part is finite and already certified. The runtime's part — the layouts, the outer loops, the reference procedures, that is, the classical pencil-and-paper algorithms themselves — is ordinary code, and the paper states explicitly that this part is not formalised in Lean. "Provable" covers every step of the model; it does not mean the whole program has been proved correct.

Theorem 2 is the bill that must be paid: the paper constructs a two-state table and a family of decoders such that every member passes any finite set of tests with positive margins, yet errs on a longer run. The conclusion is blunt — this design must give up chain of thought.

The dialogue adapter is the compromise: it opens the sheet when a reply needs exact computation. It stays within 2 points of the base model on instruction following and MMLU, but scores 5.8 points below the base model on 1,069 grade-school word problems not used during development. The gate restores the base model bit for bit at zero, at the cost that the adapter's branches are still evaluated there, adding 1.2% of its parameters to every forward pass.

So what

The most counter-intuitive thing is not that a small model got the arithmetic right, but that the paper hands down a verdict on chain of thought along the way: letting a step read the steps before it is precisely what makes "getting it right" lose its provability. The industry's two-year bet on "let it think longer" runs straight into that wall.

The second thing is the price tag. Verifiability is not something a model comes with; it is bought with architectural constraints: give up chain of thought, give up 5.8 points of GSM8K, hand the outer loop to classical code, evaluate an extra 1.2% of adapter parameters on every forward pass. What comes back is concrete — a certificate bound to file hashes that can be recomputed on the user's own GPU; every number in the paper is listed in numbers.md, and the Lean proofs are checked by running lake build. A proof a third party can re-execute and a benchmark score are two kinds of credit.

The question is: once "getting it right" can move from a sampling estimate to an executable check, how many tasks still deserve a sampling estimate? And how much capability are you willing to pay for verifiability?

(Paper: Sketchpad: certified exact arithmetic with a small vision-language model)