The longer an agent runs, the heavier its ledger. Task records, tool outputs and intermediate conclusions from many turns of interaction keep piling into the context, so compaction becomes survival gear: have the big model summarize the long history and swap the summary in for the original. The problem is that this step is itself a full generation pass — your most expensive model restating your longest history. The moment compaction triggers, the user just waits.
Why compaction is slow
Context compaction has always been a dilemma: summarize too aggressively and information is lost, hurting later tasks; summarize too carefully and latency spikes, wrecking the interaction first. The paper is blunt about the cost: summarizing long histories with large models introduces substantial delays, so compaction must balance information retention against latency. In other words, compaction is not a free butler — it is a big line item on the bill itself.
Slots in parallel, verified line by line
SpecCompact starts from one observation: structured summary slots can retain information even when generated independently. That single sentence dissolves the dependency chain inside a summary. The framework then becomes a pipeline: fine-tune a small model as the drafter and let it draft the slots in parallel; the target large model no longer writes the whole summary from scratch — it verifies each proposed line, and only repairs the rejected ones; subsequent drafting conditions on the updated slot prefix. A set of system-level optimizations further cuts repeated context processing and repeated model calls. The "small model drafts, big model verifies" structure shares its logic with speculative decoding at the token level: verification is cheaper than generation, so spend the most expensive compute only on gatekeeping.
How much faster, at what cost
On the AppWorld and tau-3 benchmarks across two target models, SpecCompact's compaction step runs 2.04x to 5.63x faster than Hermes-style summarization; in a user study within Hermes, the median compaction speedup reached 8.68x. The paper does not oversell: separate downstream evaluations reveal model- and task-dependent performance trade-offs — you get the speed, with no across-the-board guarantee on quality (paper).
So what
My take: the number worth remembering is not 8.68x but the asymmetry being generalized — verification is cheaper than generation, and that gap is now being exploited all the way from token-level speculative decoding up to semantic-level summary verification. For anyone building agent infrastructure, compaction latency is now a first-class line item on the interaction path: it is no longer a background batch job but a stall users can feel directly. Next time you debug "why is my agent slow", check how often it compacts its history.