Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development

🤗 Upvotes: 43 | cs.AI

Authors:
Yiwei Li, Wanli Yang, Hexiang Tan, Xiangzhou Huang, Zhengyu Chen, Ziran Li, Borun Chen, Shanglin Lei, Huaisheng Zhu, Hao Tian, Fei Sun, Xunliang Cai, Jingang Wang

Title:
Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development

Arxiv:
http://arxiv.org/abs/2608.13417v1

Abstract:
Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast, your go-to podcast for the latest in AI research.

Ashley: Today's paper is from the Hugging Face daily paper list of August 18, 2026, and it's garnered 43 upvotes.

The title of the paper is 'Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development.'

Evan: The first two authors are Yiwei Li and Wanli Yang, with the corresponding author being Yiwei Li.

Ashley: They are from Meituan and the University of Chinese Academy of Sciences.

Evan: Alright, let's dive into the Introduction.

Ashley, why don't you start us off with some background?

Ashley: Sure, Evan.

Autonomous agents are becoming increasingly proficient at improving models, systems, and other technical artifacts through long-horizon experimentation.

This means that these agents can run experiments, interpret feedback, and refine their methods over extended periods.

Evan: That's fascinating.

But to understand their true capabilities, why do the authors argue that we need to go beyond final scores?

Ashley: Good question.

Final scores alone don't tell us where progress is made or lost during the process.

They also fail to show whether the agents' accumulated experience leads to improved decisions later on.

Basically, we need more granular insights into their within-run behavior and how well they reuse previous experiences.

Evan: Right, that makes sense.

So, what do the authors propose for this deeper understanding?

Ashley: To address these gaps, the authors present a systematic evaluation framework for seven frontier models across 36 long-horizon tasks.

Their framework evaluates three key aspects of agent behavior: Solution Framing, Execution, and Feedback Control.

Beyond these, they also assess experience reuse within and across tasks.

Evan: Can you break down these three aspects a bit more?

Ashley: Of course.

Solution Framing evaluates how well an agent formulates and pursues practical directions.

Execution measures whether the agent reliably implements these directions into executable and correct results.

Feedback Control checks how effectively the agent uses feedback to maintain progress and recover from setbacks.

Evan: Got it.

So, in simpler terms, they are examining how good these agents are at planning, doing, and learning from their actions?

Ashley: Exactly.

And to dig even deeper, they create rule-based metrics using verifier outcomes and recorded trajectory signals to measure these capabilities objectively, without relying on Large Language Model judgments.

Evan: Interesting.

What about their findings on experience reuse?

Ashley: They treat experience reuse as a meta-capability, conducting controlled comparisons to measure its effect on subsequent decisions within a single task, known as intra-task, and across different tasks, known as inter-task.

Evan: And how does all this affect the performance of current agents?

Ashley: The results indicate that while agents can formulate and implement practical solutions, their performance can vary significantly across different runs.

Most importantly, they usually adapt or combine established techniques rather than coming up with genuinely novel methods.

Evan: So, we see some variability in performance.

Do the authors say why that is?

Ashley: They suggest that this variability and the current limitations are due to several factors, including process bottlenecks unique to each task, the dual nature of experience reuse, and design choices in the surrounding harnesses that support the agents.

Evan: This framework and detailed analysis seem crucial for understanding and improving autonomous research agents.

So to recap, the paper introduces a comprehensive evaluation framework that moves beyond final scores, focusing on the processes and experience reuse in long-horizon AI R&D.

It highlights the strengths and weaknesses in current agents' capabilities and suggests pathways for improvement.

Ashley: That's right, Evan.

And that wraps up our discussion on the Introduction section of the paper.

Evan: Alright, let's move on to the Method section of this paper.

Ashley, can you give us a breakdown of the systematic framework the authors proposed?

Ashley: Of course.

The authors' evaluation framework is structured around four main questions.

First, how strong are the final results produced by current agents?

Second, where is progress gained or lost within the research loop?

Third, can accumulated experience improve subsequent decisions?

And fourth, how does harness choice affect agent performance?

Evan: Interesting questions.

How do they go about answering these?

Ashley: To measure final performance, they directly evaluate task scores.

But to diagnose within-run behavior, they decompose the research process into three capabilities: Solution Framing, Execution, and Feedback Control.

They also introduce rule-based metrics derived from verifier outcomes and recorded trajectory signals to score these capabilities.

Evan: What about experience reuse?

How do they assess that?

Ashley: Great question.

Experience reuse is treated as a meta-capability.

They conduct controlled comparisons to measure its effect on subsequent decisions in both intra-task and inter-task settings.

Essentially, they look at how past experiences within the same task and across different tasks influence the agent's decisions.

Evan: And how is the overall evaluation set up?

Ashley: The evaluation involves seven frontier models tested on a comprehensive suite of 36 long-horizon tasks.

These tasks are organized into four workload families that capture distinct demands of automated research: Model Development, System Optimization, Puzzle & Challenge, and CUDA.

Each task comes with an objective, a starting artifact, an expert reference solution, a time budget, and an automated verifier.

Evan: Can you explain these workload families a bit more?

Ashley: Model Development tasks involve optimizing machine learning models, System Optimization tasks focus on improving system performance, Puzzle & Challenge tasks are designed to test problem-solving capabilities, and CUDA tasks involve optimizing low-level GPU operations.

Evan: Alright, that makes sense.

So, how do they ensure a fair comparison across models?

Ashley: For cross-model comparison, all seven models use Claude Code as a practical shared harness, keeping the tool interface and iteration policy fixed.

They also compare how different harness designs impact performance by comparing Claude Code with native and open-source alternatives.

Evan: What exactly do they measure in their evaluation protocol?

Ashley: They report two main metrics for each model-task pair: avg@3, which represents the average performance across three independent rollouts, and best@3, the highest performance observed across those rollouts.

Each task keeps its original time budget, typically between 2 to 12 hours.

Evan: Can you break down the processes they evaluate within a single run?

Ashley: Certainly.

Automated research involves repeated rounds where an agent proposes a direction, implements it, observes the result, and decides how to proceed.

The process is split into three metrics: C1 for Solution Framing, C2 for Execution, and C3 for Feedback Control.

These are computed deterministically from evaluation signals rather than subjective judgments.

Evan: How are these metrics determined?

Ashley: C1, or Solution Framing, measures the quality of directions the agent pursues, based on the verifier score of those directions.

Shorter runs carry their last running best score forward, and longer runs are cut off at 20 checkpoints.

C2, or Execution, checks whether changes are correctly implemented by validating whether the artifact runs and meets correctness criteria.

C3, or Feedback Control, assesses how well the agent preserves successful discoveries and responds to any regressions.

Evan: What do they mean by preserving successful discoveries and responding to regressions?

Ashley: For C3, peak retention measures the final score relative to the highest score reached during the run.

If the agent scores lower at the end than at some point earlier, it indicates a regression.

Recovery credit measures how well the agent recovers from these regressions, considering how quickly and effectively it can regain lost ground.

Evan: So, how do they handle the accumulation and reuse of experience?

Ashley: They use a counterfactual design for intra-task experience reuse, comparing the first commit after a specific point in the trajectory with and without prior experience.

For inter-task experience, they compare performance on target tasks with and without lessons extracted from a source task.

Models extract lessons from their best baseline trajectory and then attempt a new task, with and without those lessons.

Evan: That's a comprehensive setup.

It seems like they're covering every aspect of the agents' performance and learning capabilities.

Ashley: Exactly.

This detailed evaluation framework helps them identify specific strengths and weaknesses within and across tasks, uncovering areas where agents perform robustly and where they require more refinement.

Evan: It's thorough, no doubt.

Anything else regarding their methodology?

Ashley: They also incorporate cost and resource analysis, recording token consumption and wall-clock time for the main evaluation runs.

Public API prices are used to estimate each model’s mean inference cost per task, combining performance insights with efficiency considerations.

Evan: Efficiency in terms of computational resources and cost is a critical aspect, especially for large-scale research.

Ashley: Indeed.

And this dual focus on performance and cost-effectiveness allows the authors to provide practical recommendations for model training, inference-time strategies, and harness design.

Evan: To sum it up, their methodological approach offers a multi-dimensional evaluation of long-horizon automated R&D agents.

Covering everything from proposal quality and execution reliability to feedback control and experience reuse, while also measuring costs and resources, provides a holistic view of current capabilities and areas for improvement.

Ashley: And that concludes the detailed look at the Method section of the paper.

Next, we’ll discuss their findings and the implications for future research.

Evan: Let's move on to the Experiment and Results section of the paper.

Ashley, can you walk us through the experiments they conducted?

Ashley: Sure, Evan.

The authors conducted a comprehensive set of experiments to evaluate the performance of seven frontier models across 36 long-horizon tasks.

These models include Claude-Opus-4.7, GPT-5.5, Gemini-3.1-Pro, GLM-5.2, Kimi-K2.7-Code, DeepSeek-V4-Pro, and LongCat-2.0.

All models were tested using a shared harness as well as their own native or open-source harness alternatives.

Evan: Interesting mix of models.

How did they perform overall?

Ashley: Overall, Opus-4.7 ranked first in terms of average performance, achieving the highest avg@3 score of 0.739 and best@3 score of 0.790.

GPT-5.5, GLM-5.2, and Gemini-3.1-Pro formed a close second tier.

Evan: And how did they perform across different task categories?

Ashley: In Model Development, Opus-4.7 was the leader, but interestingly, the best individual run came from Gemini-3.1-Pro.

System Optimization showcased a different trend where Opus-4.7 led in average performance, but the best individual run results were nearly identical for Opus, GPT-5.5, and GLM-5.2.

Puzzle & Challenge proved the most accessible with high scores across all models, and Opus-4.7 narrowly led here.

For CUDA tasks, which proved to be the most challenging, Opus-4.7 led avg@3, but GPT-5.5 achieved the highest best@3.

Evan: So, it looks like no single model dominated across all task categories.

What about the within-run performances?

Ashley: Exactly.

While models like Opus-4.7 showed strong average performance, they weren't universally the best in every scenario.

When examining the within-run performances, they found interesting insights.

For instance, Execution scores were broadly reliable across all models, but significant variations existed within the Solution Framing and Feedback Control metrics.

For example, GPT-5.5 displayed better execution with a score of 0.958 compared to Gemini-3.1-Pro's 0.889, but Gemini excelled in Feedback Control, scoring 0.920 over GPT-5.5’s 0.858.

Evan: And what about process bottlenecks?

Any particular tasks where they observed significant issues?

Ashley: CUDA tasks had the lowest scores in Solution Framing (C1) and Execution (C2), but maintained high Feedback Control (C3).

This indicates that discovering and implementing effective optimizations in CUDA tasks was particularly challenging.

On the other hand, Puzzle & Challenge tasks ranked high across all dimensions, showing that the agents could handle them quite well.

Evan: And what about the role of experience reuse?

Ashley: Intra-task experience reuse showed positive gains for most models.

For instance, Opus-4.7 recorded a small gain of +0.0362, while LongCat-2.0 showed a substantial gain of +0.1454.

This indicates that weaker models tend to rely more on accumulated experience to improve subsequent solutions.

However, experience can sometimes mislead, as seen with Kimi-K2.7-Code, which experienced a small negative gain in a few instances.

Evan: And what about inter-task experience reuse?

Any significant findings there?

Ashley: Inter-task experience reuse showed a strong dual effect.

For example, DeepSeek-V4-Pro displayed the largest gains with +0.093 on avg@3 and +0.071 on best@3, benefitting from extracted lessons.

In contrast, Gemini-3.1-Pro exhibited a decline in avg@3, showing that transferred experience can sometimes carry forward misleading conclusions or inappropriate tactics.

This highlights the importance of filtering and adapting reusable experiences.

Evan: It's interesting how the same experience reuse can both help and hurt, depending on the context.

How about the resource usage and efficiency aspects?

Ashley: Yes, resource usage was another critical aspect of their analysis.

They found that Opus-4.7, while being the best performer, was also the most expensive in terms of mean cost per task at $89.9.

Conversely, GPT-5.5 and GLM-5.2 offered close performance for significantly less, at $16.5 and $33.0 per task respectively.

On the budget-friendly end, LongCat-2.0 and DeepSeek-V4-Pro provided competitive performance at very low costs, $3.9 and $4.3 per task.

Evan: So, depending on the budget, different models might be more appropriate despite the performance gaps.

Ashley: Exactly.

The cost-performance trade-offs provide valuable guidance for various deployment scenarios, helping to match the right model to specific needs and budget constraints.

Evan: A lot of valuable insights there.

That concludes our deep dive into the Experiment and Results section.

Evan: Moving forward, let's discuss the Related Work section of the paper.

Ashley, how do the authors position their work within the broader research landscape?

Ashley: The authors cite a range of studies and benchmarks that have evaluated autonomous agents on executable research and engineering tasks.

They reference early benchmarks like MLAgentBench, MLE-bench, and RE-Bench that assess an agent's ability to modify code, run experiments, and improve machine-learning systems under real-world constraints.

Evan: So these early benchmarks laid the groundwork.

How does this paper differentiate from them?

Ashley: Great question.

Recent benchmarks such as PostTrainBench, MLS-Bench, and AutoLab extend these evaluations to longer time horizons and resource-bounded settings.

Unlike these benchmarks that primarily focus on final performance metrics or global search behaviors, this paper moves beyond by jointly evaluating the process competence and experience-driven self-improvement of agents.

Evan: Beyond final performance.

How exactly do they delve deeper into the agent's process?

Ashley: Several related studies have focused on process-level evaluation.

For instance, AgentBoard introduced progress rate metrics to measure advancement toward intermediate goals, and TRAJECT-Bench evaluates the correctness of tool selection and order of execution.

Similarly, WebStep uses semantic environment states to distinguish exploration reach from execution accuracy.

Evan: These sound quite comprehensive.

How does the current paper's approach build upon or differ from these evaluations?

Ashley: What sets this paper apart is its focus on evaluating the end-to-end research workflow of agents, particularly how they manage their iterative research tasks.

Unlike previous works that evaluate isolated steps or predefined subgoals, this paper uses objective, judge-free metrics that do not assume canonical solution paths.

Evan: That's a nuanced way of looking at agent capabilities.

And how about experience reuse?

Any prior work they build upon?

Ashley: Prior works like Reflexion have explored how agents can improve by retaining and reusing past experiences.

For instance, Reflexion converts task feedback into verbal reflections to guide future attempts, while ExpeL extracts reusable insights from previous task trajectories for cross-task transfer.

Evan: So the concept of learning from experience isn't new.

What makes their approach unique?

Ashley: Their approach stands out in its systematic evaluation of experience-driven self-improvement both within tasks (intra-task) and across different tasks (inter-task).

This builds on the foundation laid by evaluations like LifelongAgentBench and SEA-Eval that extend beyond episodic assessment to sequential task streams, measuring skill transfer and longer-term evolution.

Evan: Interesting.

And does this framework capture the essence of agent harnesses as well?

Ashley: Yes, harnesses play a critical role.

Studies like SWE-agent have shown how the agent-computer interface design can affect performance.

The Holistic Agent Leaderboard further analyzes models, scaffolds, and benchmarks collectively.

The current paper follows this tradition by controlling the harnesses to isolate the model's true capabilities, but also explores how automated harness optimization can provide additional headroom for improvements.

Evan: It’s impressive how they tie in so many aspects of prior research to provide a holistic evaluation framework.

Ashley: Indeed.

The synthesis of these diverse streams of research gives us a thorough understanding of both the current state and future potentials of autonomous research agents.

By moving beyond traditional benchmarks to evaluate end-to-end processes and experience reuse, this paper contributes significantly to guiding the development and optimization of autonomous research systems.

Evan: And that wraps up our exploration of the Related Work section.

Evan: Let's wrap things up by summarizing the key contributions and takeaways from this paper.

Ashley: Sure.

This paper, 'Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development,' presents a comprehensive framework to evaluate autonomous research agents beyond just their final outputs.

By dissecting their performance into Solution Framing, Execution, and Feedback Control, and by examining intra- and inter-task experience reuse, the authors provide a detailed picture of current agent capabilities and limitations.

Evan: Right.

One key takeaway is that while current agents can formulate and implement practical solutions, their performance is inconsistent, and genuine methodological innovation remains rare.

They often rely on established techniques, and their ability to reuse experience can both help and sometimes mislead.

Ashley: Exactly.

The paper highlights that process bottlenecks vary depending on the task, showing that different tasks demand different improvements in model training, inference-time strategies, and harness design.

CUDA tasks, in particular, proved challenging in both Solution Framing and Execution.

Evan: Additionally, the resource analysis showed the importance of cost-efficiency, indicating that while Opus-4.7 performs exceptionally well, other models like GPT-5.5 and GLM-5.2 offer competitive performance at a fraction of the cost.

Ashley: In summary, this paper provides a multi-dimensional evaluation framework that not only helps in understanding where current autonomous research agents stand but also suggests concrete directions for future improvements.

These include better model training, experience management, optimized harness designs, and more informative evaluation metrics.

Evan: That brings us to the end of this episode.

We hope you enjoyed our deep dive into the systematic evaluation of autonomous research agents.

Ashley: Thank you for listening to Daily Paper Cast.

Join us again next time for more insights and discussions on cutting-edge AI research.

Evan: Don't forget to follow us and stay updated with the latest in AI research.

See you in the next episode!

Goodbye!

Ashley: Goodbye!