ScholarEvolve Lets AI Agents Rewrite Their Own Code From Research Papers, Lifting Task Completion 12.4 Points

Improving an AI agent's performance does not necessarily require retraining the AI model itself. Even without changing the model's weights, improving the software that controls how the agent calls tools, manages its execution history, and carries out tasks may substantially raise its task completion rate. On September 30, 2026, researchers at the University of California, Santa Barbara (UCSB) and Microsoft posted a non-peer-reviewed preprint on arXiv, "Learning from Research: Toward Lifelong Agent Harness Evolution," proposing a framework called "ScholarEvolve" based on this idea.

ScholarEvolve analyzes an AI agent's failure history, searches relevant academic papers for improvements, and automatically rewrites the software the agent runs on. The targets for improvement include tool-call handling, conversation history and context management, and skill accumulation, which together are called the "harness." In the research team's experiments, the weights of the model executing the tasks were held fixed, yet the task completion rate of the open-weight model "Qwen3.5-27B" on AppWorld Normal rose 12.4 points, from 69.0% to 81.4%.

This does not mean, however, that Qwen3.5-27B itself learned new knowledge or capabilities. In practice, the result came from using external high-performance AI models and coding agents to reflect insights from research papers in the harness code. The research team reports performance gains across multiple benchmarks, but evaluating practical viability also requires considering the computational cost of the optimization, reproducibility, and the safety of incorporating external code.

{
  "type": "bar",
  "title": "Task completion rate comparison on major benchmarks",
  "unit": "%",
  "categories": ["AppWorld Normal (Qwen3.5-27B)", "AppWorld Challenge (Qwen3.5-27B)", "tau2-Bench Telecom (GPT-5.4-mini)"],
  "series": [
    { "name": "Initial harness", "values": [69.0, 49.6, 72.7] },
    { "name": "Meta Harness", "values": [null, 54.6, null] },
    { "name": "ScholarEvolve", "values": [81.4, 63.6, 81.9] }
  ],
  "caption": "Values reported by the authors for each experimental setting. The weights of the task-executing model were fixed during evaluation",
  "source": "arXiv:2609.40169"
}

In every evaluation environment, the harness improved by ScholarEvolve outperformed the initial configuration. On AppWorld Challenge, which includes unseen APIs, it beat "Meta Harness," an existing method that improves harnesses based on execution history, by 9.0 points. The research team sees this result as evidence that improving designs by consulting external research, not just past failures, is effective.

The Limits of Relying Only on Failure History

Prior research on autonomously improving AI agents has mainly analyzed the execution history (trajectories) from task runs, identified the causes of failures, and fixed them. For example, when an agent fails an API call, the log is fed to another AI agent that revises the prompt or retry handling. Some studies focus on improving how skills and memory are managed, such as Google's WikiSkill and Microsoft's SkillOpt, while others, such as Salesforce's DarwinX and Stanford University's Meta-Harness, rework the configuration of the entire execution framework.

However, the research team points out that methods relying solely on execution-history analysis plateau early. One reason is that the meta-agent proposing improvements basically chooses fixes based on its own pretrained knowledge. A failure log may reveal where a problem occurred, but it does not necessarily lead to new design techniques that solve it. Especially when facing unseen APIs or complex environments, existing knowledge alone may not be enough to devise suitable countermeasures.

In one example cited by the team, when errors in the search tool and Python code execution were frequent, the meta-agent added a module that validated every action in advance. The mechanism could detect invalid calls, but it was so strict that it also blocked correct API calls needed to complete tasks. A fix meant to reduce immediate errors ended up undermining the agent's overall flexibility. It shows that solving individual problems does not necessarily improve the performance of the whole system.

Another problem is that history-based improvement is fundamentally passive: it waits for failures to occur. Until an agent encounters a new problem, no improvement addressing it begins. If this approach is adopted in production, users may continue to be affected by the same defects until enough failure data accumulates. The research team concluded that, just as human developers read papers to learn new design techniques, AI agents need a mechanism for drawing on external academic literature to improve their execution framework before they experience failures.

A Four-Stage Process That Turns Academic Papers Into Executable Code

ScholarEvolve manages an AI agent's harness as five modules: a "tool interface" that handles tool calls, "context management" that organizes prompts and execution history, "skills" that hold reusable operating procedures, "memory" that accumulates past experience, and an "agent workflow" that controls task decomposition and execution order. Each module has clear input/output interfaces, so that changing one part is less likely to affect other functions. With this design, problematic functions can be improved individually rather than rewriting the whole harness.

Optimization proceeds in four stages. First, an auditor AI model examines execution histories, such as those from development tasks, and identifies the agent's weaknesses. For example, it extracts problems such as failing to properly reference previously retrieved information or being unable to recover from Python syntax errors. Based on this analysis, it creates search queries and retrieves relevant academic papers and summaries from arXiv.

In the second stage, the retrieved papers are classified by design technique. ScholarEvolve uses a method called "TopicGPT" to organize which improvements to which modules each paper could help with. Simply picking papers from the top of the search results could yield many similar techniques. Keeping different approaches as candidates ensures diversity in the design ideas explored.

In the third stage, the design techniques described in the papers are converted into actual code. A research agent examines the paper's main text, appendices, and the code in public repositories, then creates a change design document (a "mutation blueprint") summarizing the mechanisms and prerequisites needed for implementation. A coding agent then rewrites the Python code of the target module based on that document. In this stage, changes are limited to a single module while the other four are held fixed, making it easier to understand the effects of each modification.

Before being evaluated on benchmarks, the generated code undergoes basic operational checks. These verify whether it can be imported as a Python module, whether it adheres to the specified input/output interfaces, and whether it returns actions in the proper format. This allows code with syntax errors or interface inconsistencies to be screened out early. However, these are functional checks and do not comprehensively guarantee the code's safety.

In the final fourth stage, the individually improved modules are combined and the performance of the whole harness is evaluated. Using a method similar to a genetic algorithm, multiple module candidates are combined and their results on validation tasks are compared. The performance differences are then tested statistically using a paired bootstrap method, and only candidates with confirmed improvements are adopted into the best configuration so far, called the "champion." This mechanism prevents replacement with candidates that happened to score high in a single evaluation, and it helps suppress performance regressions.

A concrete example is the improvement made to Qwen3.5-27B's tool interface. The initial agent sometimes failed by generating Python code with invalid syntax or by guessing arguments without checking the API specifications. Drawing inspiration from the research paper "ToolACE-R," ScholarEvolve added a feature that checks syntax using an abstract syntax tree (AST) before code is executed, and a feature that displays API specifications as needed. Rather than merely instructing the model to write correct code, the approach is notable for having the execution framework itself detect erroneous operations and make necessary information available.

In this way, ScholarEvolve does not simply add the contents of papers to prompts; it implements them as functions of the execution framework. Splitting improvement targets by module makes it possible to compare multiple design techniques and ultimately combine them to optimize the whole system. What the research team is aiming for is a mechanism that continuously reflects new insights from papers in concrete software that changes the AI agent's behavior.

Performance Gains Confirmed on Benchmarks

The research team evaluated ScholarEvolve on two benchmarks that require complex operations close to real-world work. One is "AppWorld," in which the agent operates multiple virtual applications and calls APIs through Python code. The other is "tau2-Bench Telecom," which involves customer support and troubleshooting for telecom services through dialogue with a simulated user. The task-executing models were the open-weight Qwen3.5-27B and the proprietary GPT-5.4-mini. In all experiments, the weights of the task-executing model were fixed, and performance differences due to harness changes were measured.

In the AppWorld experiments, 90 tasks were used to improve the harness and 57 for development-stage evaluation. Final performance was then measured on the official test sets, Normal (168 tasks) and Challenge (417 tasks). The evaluation metric TGC (Task Goal Completion) represents the task completion rate, while SGC (Scenario Goal Completion) indicates the proportion of scenarios in which all three variants were completed. SGC is a stricter metric that shows whether the agent can complete tasks stably even when conditions change somewhat.

On AppWorld Normal with Qwen3.5-27B, TGC rose 12.4 points, from 69.0% for the initial harness to 81.4%. SGC improved 20.2 points, from 48.8% to 69.0%, suggesting that not only did more tasks succeed, but the agent also became better at handling tasks under different conditions. On AppWorld Challenge, which deals with Amazon and Gmail APIs that were not included in the optimization data, TGC also rose 14.0 points, from 49.6% to 63.6%.

This Challenge test also included a comparison with the existing method Meta Harness. Under the same search budget and using GPT-5.4 via Codex, Meta Harness's TGC was 54.6%, while ScholarEvolve recorded 63.6%. The 9.0-point gap suggests that incorporating new design techniques from external research papers, not just fixes based on execution history, may also help with handling unseen APIs.

The paper also explains that ScholarEvolve's 81.4% score for Qwen3.5-27B closed about 74% of the gap with the 85.7% recorded by GPT-5.4 in its initial state. However, the task-executing models differ in this comparison, and the initial harness configurations are not identical. It therefore does not mean that Qwen3.5-27B acquired capabilities equivalent to GPT-5.4. The figures are averages of three evaluations by the research team, and no independent replication by third parties has yet been shown.

Module Combinations Shape Performance

The ScholarEvolve experiments also produced an interesting result: the performance ranking reversed between evaluating individual modules alone and combining multiple modules. The research team verified this difference through ablation experiments that examine the influence of specific components.

For example, when the skill management module was evaluated alone, one candidate recorded a 75.4% task completion rate. Another candidate, inspired by the paper "Skill-as-Pseudocode," scored slightly lower alone at 73.7%. However, when combined with the improved tool interface and context management modules, the harness using the latter reached 84.8%, exceeding the 80.7% of the configuration using the former. In other words, a module with high standalone performance is not necessarily the best choice for the system as a whole.

This result shows that interactions among the functions making up the harness strongly influence agent performance. For example, in an environment where the tool interface properly detects syntax errors and context management retains necessary history, a skill module that manages procedures as typed pseudocode is likely to work more effectively. It is important not only to improve each module's performance individually but also to evaluate them in combination. This is why ScholarEvolve searches over combinations of candidate modules.

Improvements were also seen in the ability to handle runtime errors. In AppWorld Challenge, when the team analyzed the histories of tasks in which a Python runtime error occurred midway, the initial harness's task completion rate was 44.9%, which rose to 59.9% after the improvements. Rather than completely preventing errors, it appears the ability to correct course after an error, using information such as tracebacks, and continue the task improved. Since API call failures and unexpected responses are unavoidable in real business environments, this recovery ability is an important factor.

On the other hand, not every type of failure decreased substantially. According to the team's failure analysis, errors caused by invalid response formats fell from 492 to 206, less than half. However, business-constraint violations went from 91 to 89 and omissions in retrieving necessary information from 77 to 70, showing only small improvements. While syntax flaws and API call errors can be reduced through harness-side measures, problems such as understanding complex business rules and judging what information is needed appear harder to solve with the same approach.

Will Performance Keep Improving as New Papers Are Incorporated?

The research team also tested whether the harness's performance continues to improve as the pool of academic papers available for reference grows over time. In this experiment, the weights of the task-executing model and the accumulated execution histories were held fixed, and only the range of available papers was expanded step by step. The periods were literature through December 2025 as Period 1, January–April 2026 as Period 2, and May–August 2026 as Period 3.

As a result, Qwen3.5-27B's TGC on AppWorld Normal rose in stages from 69.0% in the initial state to 77.4% in Period 1, 79.8% in Period 2, and 81.5% in Period 3. SGC in Period 3 was 66.7%. The result suggests that the harness can be improved further by using new research findings, even without adding model weights or execution history.

Based on this experiment, the authors propose an operating method in which the latest papers are collected monthly and used to continuously update the agent's execution framework. However, this verification split past papers by publication date; it is not a track record of continued autonomous updating over months or years in a real business environment. Moreover, adding more new papers does not always lead to better performance, and improvements may plateau or the system may overfit to particular tasks. How much improvement can be sustained over long-term operation remains a question for future verification.

Compute Cost, Reproducibility, and Security Challenges Remain for Practical Use

ScholarEvolve is a promising way to raise performance without retraining the task-executing model, but several challenges remain before companies can actually adopt it. Notably, not needing to retrain the model is not the same as the optimization cost of the whole system being low.

In these experiments, GPT-5.4 was used to search for and select papers, and Codex and GPT-5.4 were also used to generate module code. In the AppWorld optimization, 20 paper-derived candidates were generated, and 15 for tau2-Bench Telecom, with task execution and performance evaluation repeated for each. Comparing candidates and statistical verification also require additional computation. Fixing the task-executing model's weights avoids the burden of retraining, but paper analysis, code generation, and benchmark runs by external models still demand considerable computing resources.

The execution environment for Qwen3.5-27B is also relatively large in scale. In the setup described in the paper, vLLM 0.27.1 was used for the inference server, with tensor parallelism of 4, a sampling temperature of 0, and thinking mode enabled. The maximum token length per model call was set to 65,536. This does not mean 65,536 tokens were consumed on every call, but it is an environment designed to handle long contexts.

To decide whether to adopt it, companies need to know not only the final task completion rate but also the number of API calls, total token consumption, GPU hours, and similar figures needed for optimization. For example, if improving a harness once requires a large number of external model calls, the everyday task-execution costs saved by the improvement must be weighed against the cost of optimization. The reported figures alone make it difficult to judge such cost-effectiveness adequately.

Reproducibility is another important issue. Although the paper includes a link to a GitHub repository, at the time of writing the official code release and detailed execution instructions could not be confirmed. For outside developers to reproduce the experiments under the same conditions, they would need not only the optimization pipeline code but also the evaluation scripts, model settings, candidate selection methods, and execution environment. Until the repository's availability, license, and required dependencies are clear, independent verification by third parties will remain limited.

The Risks of Automatically Implementing External Research

Security is another major challenge. ScholarEvolve retrieves design techniques from external academic papers and public repositories and rewrites the agent's execution framework based on them. While this allows new research findings to be incorporated quickly, it also brings untrusted information into the code-generation process.

The sanity probes described in the paper focus on functional checks: whether the generated Python code can be imported, whether it follows the interface conventions, and whether it returns correctly formatted actions. These are effective for confirming basic program behavior, but they are not a mechanism for reliably excluding code that carries out malicious processing or leads to information leakage. Even code that runs correctly may contain security problems, such as unauthorized external communication or excessive use of privileges.

In particular, in a mechanism where an AI agent reads text from external papers and repositories and generates code by following it, countermeasures against prompt injection are essential. If an attacker embeds malicious instructions in public materials, the code-generating agent could mistake them for legitimate design requirements. Referencing third-party public code could also bring in vulnerable implementations or unintended dependencies. At this point, such vulnerabilities have not been demonstrated in ScholarEvolve itself, but mechanisms that ingest external information and automatically update executable code generally need to be designed with these threats in mind.

If companies adapt this approach, one option would be to limit references to pre-vetted design documents and internal libraries rather than using papers and code on the internet without restriction. Generated code would be run in an isolated environment, and only code that passes static analysis and security testing would be adopted. Before harness changes are reflected in production, it would be desirable to insert human review and to apply policy controls, independent of the model's judgment, to external API access and the handling of confidential information. It is important to design functional checks and safety reviews separately.

What ScholarEvolve has shown is that improving AI agent performance does not depend solely on scaling up or retraining models. Even with an existing model's weights fixed, appropriately improving the surrounding software can substantially raise task completion rates. It also suggests that incorporating design techniques from academic papers may overcome the limits of improvement methods that rely only on past failure histories.

On the other hand, these results are based on a non-peer-reviewed research report, and long-term autonomous updating, safety in real-world operation, and cost-effectiveness have not been demonstrated. Going forward, alongside the release of official code and replication experiments by third parties, quantitative evaluation of search costs and safety verification when incorporating external information will be important. How far the direction of continuously improving the execution framework that runs the model, in addition to improving the model itself, will translate into practical performance gains is worth watching.