arXiv 2606.16429 introduces Taylor-Calibrate, a method for initializing a Gated DeltaNet (GDN) — a recently proposed linear-attention architecture — from a pretrained Transformer. The result: a Transformer-to-GDN distillation that preserves 95% of the Transformer's quality while gaining GDN's inference efficiency.
The "initialization" challenge: when training a new architecture (like GDN) from scratch, the model needs a lot of data and compute to reach the quality of a pretrained Transformer. "Initialization from a pretrained model" is the standard solution, but for novel architectures like GDN, the initialization is non-trivial — the GDN's state-space representation is fundamentally different from the Transformer's KV cache.
The Taylor-Calibrate approach: a "Taylor expansion" of the GDN's state update, with the coefficients set to match the Transformer's attention output. The result is a GDN whose first-order behavior matches the Transformer's, and the higher-order terms are learned via standard training.
The benchmark: a Transformer-to-GDN distillation with Taylor-Calibrate initialization hits 95% of the original Transformer's quality with 1/3 the inference latency. The training requires 50× less compute than training a GDN from scratch to reach the same quality.
The bigger takeaway: "architecture transition" is becoming a real engineering discipline. As the LLM field experiments with new architectures (GDN, Mamba, RWKV), the ability to "port" a pretrained model to a new architecture is critical. Taylor-Calibrate is a clean, general technique for this, and it paves the way for "architecture migration" — i.e., the ability to switch architectures without losing model quality. For the industry, this means LLM vendors can experiment with new architectures without "starting from zero."