The real hidden cost of agent self-improvement research is usually not the algorithm, it is the evaluation harness. Every paper on self-improving agents rebuilds its own execution environment, scoring protocol, failure recovery and experiment logging, which means the "how many points did we gain" figures across different papers are rarely measured on the same axis. RSIHub, submitted to arXiv on October 23, sidesteps algorithmic novelty and targets that infrastructure layer instead. The paper lists the NeurIPS 2026 Meta-Agents Workshop as its venue, and the Hugging Face paper page attributes the team to ByteDance (arXiv:2610.32253).
Freezing the evaluator is the actual design
RSIHub abstracts the problem as recipes plus operators. A recipe declares which resources are allowed to be rewritten, while operators are reusable lifecycle primitives; the framework itself owns execution, evaluation, recovery and lineage. The whole loop is cut into seven phases -- select, rollout, analyze, mutate, guardrail, evaluate, absorb -- mapped onto nine named operator slots, five of them required (select, rollout, mutate, gate, record) and four optional (analyze, validate, novelty, reflect).
The critical constraint sits on the repository homepage: evaluation is never a selectable operator, it stays framework-owned. A candidate agent may rewrite prompts, skills and harnesses, and may even rewrite selected evolution operators, but it cannot touch the code that scores it. The paper frames this as an answer to attribution risk: if the party being optimized can casually modify the measurement conditions, the gain number itself stops meaning anything. The evaluator, the archive stamps and the framework code all stay outside the declared mutable surface.
Four methods on one shared substrate
The framework re-expresses four existing methods -- AHE, Hyperagents, A-Evolve and GEPA -- as recipes, keeping method-specific logic inside recipes and operators while the framework core is left untouched. The experiments use MiniSWE and Codex as seed agents on Terminal Bench 2 and tau-cubed Banking, running one trial per task and reporting pass@1. The target model is GPT-5.4 at high reasoning effort throughout, and every recipe uses a Codex mutation agent backed by GPT-5.4 at xhigh reasoning effort.
On Terminal Bench 2 with MiniSWE, the seed scores 56.2 and all four methods improve on it by 7.8 to 13.5 percentage points, with Hyperagents and A-Evolve tied at the top on 69.7, AHE on 65.2 and GEPA on 64.0. Switch to a harder seed and a different benchmark and the story changes immediately. Codex scores 69.7 on Terminal Bench 2, where AHE only reaches 71.9, Hyperagents and A-Evolve each add 1.1, and GEPA does not move at all. On tau-cubed Banking, MiniSWE seeds at 27.8 and Hyperagents climbs to 33.0 while A-Evolve stays flat at 27.8; Codex seeds at 24.7, AHE reaches 34.0, and GEPA also lands on 33.0.
Rankings flip, and that matters more than the best score
Breaking the results down by train, gate and sealed splits is more revealing. Hyperagents takes the highest overall score for MiniSWE on Terminal Bench 2, ties A-Evolve on the sealed split, and is the only method that actually moves the gate split; A-Evolve gains 7.9 points on train while leaving gate untouched. On tau-cubed Banking, Hyperagents produces the largest overall gain for MiniSWE while regressing 2.1 points on sealed, AHE regresses 4.1 points, and GEPA regresses 2.1 points as well. The paper's own conclusion is that no method dominates across benchmarks, seeds and splits.
This does not contradict the headline of a 13.5 percentage point overall pass@1 gain, but the two should be read together. The headline is the paper's own measurement, with no third-party reproduction yet; while frequently flipping rankings, gate regressions and sealed regressions are exactly where such results deserve scrutiny.
Execution fidelity and fault injection
The paper invests heavily in two control experiments designed to keep environment noise from being counted as gain. The first compares direct execution of an unchanged seed agent against execution through the framework, averaging three repetitions: on Terminal Bench 2 with Codex, direct-to-framework agreement is only 82.0 percent, below the 88.0 percent agreement between two direct runs, an excess of 6.0 points, while two direct runs already disagree on 12.0 to 18.9 percent of tasks. The paper states plainly that these controls only characterize execution variability and do not establish the statistical significance of single-run evolution gains.
The second set is fault injection and boundary enforcement, seven scenarios per benchmark at 30 independent trials each, 420 trials in total, all reaching the expected outcome with no corrupted state ever admitted into a result: termination during mutation or evaluation, writes outside the mutable surface, modification of evaluator files, access to gate, sealed and anchor evidence during mutation, and forged archive outcomes are all rejected.
Where I land
The honesty of this paper is worth more than its scores. It does not claim that any particular self-evolution method is stronger; it turns the tedious job of comparing methods fairly into reusable infrastructure. The code is released under Apache-2.0, requires Python 3.12 or newer, and the repository page shows 191 stars and 8 forks, still an active prototype.
The repository also carries a skill evolution demo: with the same Codex model and the same paper prompt, the Gen 0 twelve-line minimal Skill overflowed the SVG viewBox with 14 text elements, while the evolved Gen 2 editorial Skill passed the deterministic renderability and geometry gates, moving the deterministic completion pass rate from 1/4 to 4/4 across a four-paper showcase. The team labels it a representative evolution run rather than a broad benchmark.
The real limitation sits at the boundary: the paper concedes that these guarantees rely on trusted framework services, host and filesystem protections, and that runtime fingerprints do not replace host or service attestation, so stronger deployments need independent attestation, isolation and containment. The responsible use statement is blunter still, higher pass@1 does not establish safe deployment, and success on a banking benchmark does not establish suitability for financial decisions; what this framework offers is auditing and rollback, not containment.
So the more interesting question may not be which self-evolution method is stronger, but rather: when the thing being optimized starts editing its own experiment bench, who guarantees the ruler does not move?