Agentic reinforcement learning runs in an odd state: the policy weights change at every step, while the harness wrapped around the model — the scaffold deciding what the model sees of its own interaction and when it receives guidance — is typically hand-written once and never touched again. On November 13, Zelin He and seven co-authors from Penn State, LinkedIn and UC Santa Barbara posted a preprint, ModHarness-RL, that calls this out: the harness does not have to be dead. It can be written as programs, learned from training data, and evolved alongside the model.

Two problems with a hand-written harness

The paper lays out the tension first. An effective harness must be tailored to the task, the data, the model and the context budget; hand-tuning one for every such combination does not scale, so it should be derived from data. More importantly, the effective harness keeps changing during training: RL resolves some failure modes on its own, so new components should be added only for failures that persist. When to step in and when to hold back is itself a judgment that shifts as training proceeds.

The authors survey recent efforts to adapt the harness during RL — decaying fixed scaffolds on a preset schedule, alternating scaffold rewrites with weight updates, editing planning and memory modules from observed faults. In their framing, these update spaces are fixed in advance or tied to a task-specific scaffold; what is missing is a general way to derive harness operations from the typed states of the interaction and optimize them with the model.

Harness as programs over typed blocks

ModHarness-RL has two layers. Representation: the interaction is recorded as a structured State, a set of typed blocks — each turn's reasoning, tool call and tool result are blocks. Two core harness components are written as programs over these blocks. A Compression program reassembles the accumulated history into the model's context under a fixed budget; a Steering program detects when behavior deviates and appends guidance when a condition fires.

Programs carry three properties, in the paper's own terms: learnable, since an LLM can write them directly from training trajectories and their rewards; computationally inexpensive, so they run on every turn of every trajectory without calling another model; and testable, so each candidate harness can be replayed over stored trajectories and scored before it is accepted or rejected.

The evolution layer is a pipeline alternating with RL: summarize experience from recent trajectories, propose candidate programs, verify them by execution, and gate them on held-out training tasks. The Steering side has a Replay mechanism worth singling out: each candidate guidance program is rerun over trajectories from past training steps, and only failures that persist as the model trains get an intervention installed. The two judge cases in the paper are telling. In one, a failure keeps recurring, the judge installs the guidance, and the failure is cut down. In the other, the failure is already rare and falling, the judge holds off, and continued training erases it on its own.

The numbers: how much, at what cost

The setup is held constant: GRPO, 200 steps, one node of eight H100s. CodeSearch uses Qwen3-4B and Qwen3-8B, BIRD SQL uses Qwen3.5-2B and Qwen3.5-4B, ALFWorld uses Qwen2.5-1.5B and Qwen2.5-7B, each under a context-limited and a context-sufficient budget. Every method that learns from rollouts uses GPT-5-mini as its creator. CodeSearch asks the agent to localize a bug in a real repository, training on SWE-Smith and evaluating on SWE-Bench Lite, Verified and Pro.

Per the paper's own measurements: on CodeSearch, ModHarness-RL improves over the fixed-harness baseline by 14 to 24%, and leads the four families of learnable baselines (memory, skills, prompts, harness evolution) by 5 to 12%. With Qwen3-8B under a limited budget it reports an overall F1 of 0.412, up 24% over the base harness; under a sufficient budget, 0.460, up 14%. It also beats zero-shot GPT-5-mini — the very model that writes its harness programs — in every CodeSearch setting, which the authors take as evidence the gain is not distillation from the creator. On ALFWorld and BIRD SQL, each program applied alone improves over the base harness in seven of eight settings.

The efficiency side is sharper. Under a sufficient budget, Compression cuts prefill tokens by 74% on ALFWorld and 10% on CodeSearch, and because the program is deterministic, earlier turns keep an identical rendering and most of the prefix cache stays valid. Against online compaction by GPT-5-mini, Compression scores higher in 2 of 4 settings and at most 5% lower in the other two, while teacher-model calls and tokens drop by roughly two orders of magnitude over a training run, and the per-turn wall-clock overhead in the context-limited settings falls by more than a thousandfold in the paper's estimates.

Two ablations cut against intuition. Running the two stages sequentially — policy first then harness, or the reverse — reduces performance in every setting, so co-evolution itself carries value. And Compression, fit once against the base model and then frozen, beats refitting it every ten steps as the policy trains; with Qwen3-8B under a limited budget the refit variant drops to 0.279. A stable context matters more to policy training than a snugly updated one.

Where the edges are

The paper's own limitations are plainly stated: the comparison is most thorough on CodeSearch, while the other two environments mainly test whether each program generalizes; the gate scores candidates on a limited set of held-out tasks and can occasionally accept a slightly worse program; the programs are written by a proprietary model, GPT-5-mini — the policy sees only program output, that text is never a loss target, and the programs are built once and reused; and deterministic programs capture only patterns an explicit rule can detect, which suits structured, verifiable interaction like tool calls far better than open-ended conversation where the signal is semantic. The scale is also modest: models from 1.5B to 8B, a single node, some distance from frontier training runs.

So what

The genuinely interesting move here is turning harness engineering from craft into an optimizable object, with a cheap optimization loop — the programs do not run a model online, and replaying stored trajectories is enough to score them. As model capability grows daily during training, a harness frozen at the hand-written version from three months ago only accumulates mismatch. And once even 'what context to feed the model' becomes a learned object, how many purely manual decisions are left in an agent training system?

Reference: ModHarness-RL: Recasting Harnesses as Learnable and Evolving Programs for Agentic RL(https://arxiv.org/abs/2611.18533, arXiv preprint, submitted November 13, 2026, not yet peer-reviewed)