Self-Improvement via Fast Tree-search

Summary

Harness engineering has shown to be an effective way to realize self-improvement in coding agents. These coding agents, when given their own harness design as an engineering task, can iteratively improve their own implementation to create a loop of recursive self-improvement.

However, the improvement signal is often driven by expensive environment feedback. To measure the strength of the new harness usually involves evaluating the new harness on a full set of benchmark tasks, becoming prohibitively expensive and extremely time-consuming. How then can we search over the space of possible harness designs efficiently?

In this work, we introduce SIFT - Self-Improvement via Fast Tree-search. SIFT improves tree search efficiency from two directions: (1) with a disaggregated improvement-evaluation pipeline that allows self-improvement to run without being bottlenecked by evaluation signals, (2) cheap LLM-as-a-judge signals that guide the search process prioritizing the most promising nodes. Furthermore, the best discovered agents by SIFT show stronger generalization across tasks and LLMs, providing a method to perform tree-search-based self-improvement without overfitting to the specific set up.

1. Motivation

A self-improving coding agent alternates between two roles. A coding model solves downstream tasks using an agent harness, while a self-improvement model examines the harness and its failure traces and proposes an evolved child agent, which we can tend use in the next iteration of self-improvement. Recursing on this process generates an archive tree of agent versions in a continual process of expansion and evolution to discover stronger coding agent implementations.

Often, the bottleneck in this evolutionary process is evaluation. This process typically relies on evaluation across a full benchmark (think 500 task in SWE-bench) to provide a relatively reliable signal. Not only is this expensive and time-consuming, benchmark scores can also be extremely noisy, possibly also result in overfitting on the specific benchmark the harness was supposed to hill-climb on. To provide a generalizable and cheap signal to guide the expansion process without relying on full benchmark scores, SIFT uses a code review judge as an intermediate ranking signal before committing to a full evaluation.

A judging round compares a new node against at most ten existing nodes. Even ten comparisons are substantially cheaper and faster than a 50-task evaluation.

2. Method

SIFT proceeds in the following stages. During the tree search process, we maintain a priority queue of nodes to evaluate next and a win-loss matrix between nodes i and j in the tree. We initialize the priority queue to contain just the root node.

1. A self-improvement model modifies a selected parent agent and produces a new child.
2. The judge compares the child against up to ten strong agents already in the archive.
3. A regularized Bradley–Terry model converts the accumulated pairwise results into a global strength ranking.
4. The child enters a priority queue for downstream evaluation.
5. Expansion and evaluation continue asynchronously, so the search does not wait for each benchmark run.

The probability of selecting node i as the next parent is

Here, r_b is judge rank, r_a is benchmark-accuracy rank, and v is the number of previous expansions from that node. The experiments use α = β = η = 1. The visit-count term encourages exploration; setting η = 0 reduces the o3-mini Polyglot result from 35.1% to 30.1%.

3. Polyglot results

Aider Polyglot contains 225 coding tasks across C++, Go, Rust, Java, JavaScript, and Python. During search, SIFT evaluates on a fixed 50-task subset. The complete 225-task benchmark is held out for final evaluation. Searches are limited to 30 expansion steps.

Qwen search

Over 30 steps, the best measured Polyglot-50 accuracy rises from 16% to 38%, and the average accuracy of the archive also increases. The best descendant scores 31.1% on the held-out full benchmark. This run costs $34.30, takes just 6.71 hours of wall-clock time, completing within 224 CPU hours, which is roughly one tenth of DGM’s CPU use, also exceeding the HGM result of 30.5% at 347 CPU hours.

o3-mini search and repeatability

SIFT reaches 35.1% after 30 expansions, compared with DGM’s 30.7% after 80 nodes. The run finishes in under five hours, uses 42 CPU hours, and costs $150 in API credits. Three additional runs produce full-benchmark results between 32.0% and 35.6%.

Transfer across coding models

We run the best harnesses found with o3-mini and Qwen3-30B using gpt-5-mini and gpt-5.4-mini as the coding model. The modified harnesses consistently outperform the corresponding base harnesses. This suggests that at least some of the changes are improvements to agent design, rather than optimizations that only work for the model used during search.

(a) o3-mini to other LLMs

(b) Qwen3 transfer to other LLMs

4. Judge analysis

How candidates are shown to the judge

The judge can receive either the sequence of patches or the resulting full source files. The ranking is compared with realized Polyglot accuracy across 50 non-root nodes. Full files provide a substantially stronger rank signal.

SIFT therefore uses full-file comparisons without swap-ordering. Most of the improvement comes from giving the judge the final program context, rather than asking it to reconstruct behavior from a sequence of patches.

A qualitative example

In one search, node 17 had lower measured accuracy than several alternatives but the highest Bradley–Terry judge score. SIFT continued expanding it, producing node 25, which later reached 42% on Polyglot-50. The judge ranked the lineage 2 → 17 → 25 first or second before its final downstream performance was known.

Where the speedup comes from

The asynchronous pipeline accounts for most of the time reduction. Judge-guided speculative expansion provides an additional gain by allowing promising nodes to be expanded before their full evaluations finish.

5. Terminal-Bench 2.1

Terminal-Bench 2.1 contains 87 heterogeneous, long-horizon tasks executed and graded in task-specific containers. The coding model is gpt-5-mini, the diagnosis and self-improvement model is gpt-5, and the pairwise judge is gpt-5.4-high. Search evaluations use a fixed 50-task subset and cap each agent at the smaller of the task timeout or 30 minutes.

One judge-guided search and one no-judge search are initialized from the same 14/50 agent and capped at 30 expansions. Three agents are then selected: the judge’s top node, the highest-scoring node within the judged search, and the highest-scoring node from the no-judge search. Each is evaluated three times on the full 87-task set.

The judge-selected node scores slightly worse on the search subset than the accuracy-selected nodes, but performs substantially better under repeated full-set evaluation. Across all 20 fully evaluated candidates from the judged search, mean full-set performance rises monotonically by judge-rank group, from 20.2/87 for ranks 16–20 to 28.7/87 for ranks 1–5. The node-level Spearman correlation between judge rank and full-set score is 0.72.

What the judge identified

Node 11 added a local verifier, but the verifier was controlled by an environment flag that defaulted to off. Its bash tool also started a fresh interactive shell on every call while claiming to maintain persistent state. The judge identified both problems from the implementation. It preferred node 19, whose changes were active on the execution path: a persistent shell, explicit timeouts, process-group cleanup, output truncation, and a bounded number of validation passes. Node 19 solves seven more tasks on average than node 11 in the repeated full-set runs.

The 50-task search scores and 87-task reruns use different task sets and should not be pooled. The initial agent’s full-set scores were collected earlier under a more generous time budget, so they are a reference band rather than a matched experimental arm.

6. Limitations and safety

• The judge is a noisy ranking signal, not a substitute for downstream benchmark evaluation.
• The strongest reported setup often uses a judge that is more capable than the coding model.
• A single Bradley–Terry score can hide task-specific trade-offs between agent designs.
• Judge quality depends on how candidate implementations are represented and prompted.
• Self-improvement creates evaluation-integrity risks. We observed proposals to relax timeouts or retry counts. The experiments restrict writable files and reject changes to benchmark or harness code.

Future directions include dynamically deciding which nodes deserve full evaluation, pruning consistently dominated nodes, using judges that can run lightweight targeted probes, and jointly modeling task difficulty so that evaluation compute is spent on the most informative tasks.