Speculative decoding has become the default configuration for inference optimization: a small model drafts a few tokens, then a large model verifies them in one pass. But that setup carries an implicit prerequisite you may not have thought about — the drafter either has to be learned alongside training, or finetuned separately afterwards. What if neither is available? A paper that Baidu's ERNIE team posted to arXiv on October 17 argues that the backbone model can be its own drafter (arXiv:2610.24766).

Auxiliary modules absorb the supervision

The paper targets Multi-Token Prediction (MTP). Parallel MTP predicts several future positions independently from the same hidden state, ignoring causal dependencies among them. Causal MTP, as in DeepSeek-V3, restores the autoregressive dependency by feeding prediction modules the previous hidden state together with shifted token embeddings, and prediction quality improves markedly. Both, however, share one structural problem: future-token prediction is carried out by dedicated modules attached to the backbone, rather than being integrated into the autoregressive generation process itself.

There are two consequences. The first is indirect supervision: MTP losses primarily supervise the auxiliary prediction modules, so the backbone receives multi-horizon learning signals only indirectly. The second is that long horizons are expensive — each additional horizon adds a prediction module in memory and another pass in compute. That cost pins large-scale causal MTP pretraining to a single draft token: DeepSeek-V3 trains with one MTP module, and Xiaomi's MiMo found that adding MTP layers brings no further improvement.

ECHO sounds similar to looped transformers, but runs the opposite direction. Looped transformers iterate repeatedly at the same token position; ECHO moves forward along the horizon, feeding each step the previous step's final-layer hidden state through a lightweight fusion layer and recursively invoking the backbone's final decoder layer and output head. In other words, apart from that lightweight fusion layer, every parameter used for future prediction is also used for next-token generation.

The numbers: +2.63 on reasoning and +2.23 on code at depth 3

Experiments use a 10B-A1B latent MoE backbone — 24 layers, 256 routed experts with eight active per token — pretrained on 600B tokens, roughly three days on 192 H800 GPUs. Evaluation covers fourteen few-shot benchmarks, grouped into mathematics, reasoning, multiple choice, code, and closed-book QA.

Compared against DeepSeek's MTP at matched depth, ECHO leads by +0.79, +0.57, and +1.35 on mathematics, reasoning, and code at depth 1. At depth 3, the gap widens to +2.63 on reasoning and +2.23 on code. Taking each method from depth 1 to depth 3, ECHO gains 0.63 points averaged over the fourteen benchmarks while DeepSeek's MTP gains only 0.16 — that is the quantified evidence for auxiliary-module saturation.

Two more numbers are worth noting. ECHO-1.5(3), with an expected depth of 1.5, matches ECHO-3's gain at half the MTP overhead. ECHO-2(7), with an expected depth of 2, trails during pretraining but overtakes by 0.49 after SFT, reaching a lead over DeepSeek's MTP-Instruct of +2.10. On stability, repeating the pretraining run leaves ECHO ahead by +0.52 and +0.78, with the reasoning and code gap shifting by only 0.11. That is not noise.

Train shallow, draft deep

The real ambition of the paper is inference acceleration. Sharing one layer means the trained model can draft directly, and it can draft beyond the depth it was trained for — auxiliary modules saturate past their trained depth, while ECHO keeps gaining. It also removes any need for post-hoc drafter finetuning: the paper notes that such drafters are finetuned on a dataset orders of magnitude smaller than the pretraining corpus, which risks overfitting to academic benchmarks.

The cost comparison is concrete. Pretraining takes about three days, followed by five hours of SFT. Post-hoc drafter training uses 24 H800 GPUs: roughly 22 hours for EAGLE-3, 19 hours for DFlash, and 55 hours for DSpark. ECHO's amortized MTP training overhead is only 1.5 depth-1 steps. On open-ended generation, split by the target model's predictive entropy, ECHO keeps draft-target alignment where post-hoc drafters lose theirs.

My read

What matters here is not the scores but the reframing. ECHO takes an assumption the field had grown comfortable with — that training a drafter needs an extra module — and asks instead whether the backbone's final layer can do double duty. The price for getting parameters for free is calling that layer repeatedly, and the authors acknowledge the supervision gets noisier at greater depth: ECHO-2(7) actually trailing during pretraining is that cost showing up in the data.

The paper is explicit about its limits. Pretraining was repeated once and finetuning ran with a single seed. The ernie-research/ECHO repository is already up on GitHub. For teams stuck paying to finetune a separate drafter, a backbone that drafts for itself is worth a look. (Source: https://arxiv.org/abs/2610.24766)