NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness

🤗 Upvotes: 134 | cs.CL

Authors:
NeoHorse Team, Guoliang Cao, Guohao Dai, Tianyu Guo, Kai Han, Hailin Hu, Zihan Jiang, Xiang Kuang, Boxun Li, Yulong Li, Zehua Pei, Yuchuan Tian, Jiamin Wang, Yu Wang, Yunhe Wang, Yihong Wu, Haiyang Xu, Shuo Zhang, Hang Zhou, Siyang Cheng, Jiayu Fan, Wei He, Qingrui Jiao, Hongguang Li, Zhiyuan Li, Runke Liu, Xi Liu, Xinchen Liu, Sinno Jialin Pan, Yi Ren, Liuyang Song, Chenyu Wang, Bei Yu, Quanlu Zhang, Xiangyu Zhang, Mengyu Zheng, Yingjie Zong

Title:
NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness

Arxiv:
http://arxiv.org/abs/2609.08183v1

Abstract:
Recursive self-improvement (RSI) requires a concrete mechanism through which an AI system observes its capabilities and converts that evidence into the next round of learning. We present NeoHorse-1, a family of agent-native models developed to explore this path through agentic post-training. Our system combines a heterogeneous model pool with intelligent routing, recording the predicted capability demand, selected service tier, and subsequent interaction for each user turn. These records are converted into training examples that preserve interleaved reasoning, tool calls, and harness context, and are admitted through structural validation, six-dimensional semantic evaluation, and subscene-level labeling. Routing signals organize supervised fine-tuning into a three-stage curriculum and extend to routing-guided on-policy distillation, where a teacher supervises student-generated responses under the same progression. Capability-guided allocation then converts evaluation feedback into the next training mixture, closing an evaluation-selection-update loop in which what the system learns to do shapes what it learns from next. Across eleven benchmarks covering harness-based agents, tool use, coding, and instruction following, post-training raises the macro-average from 58.94 to 64.87 at 4B and from 65.60 to 69.04 at 9B, substantially narrowing the aggregate gap between the post-trained 4B model and the 9B base model. NeoHorse-1 provides an initial prototype of this feedback-driven process and a path toward harness-mediated RSI across successive iterations.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast!

Ashley: Today we dive into a paper selected from Hugging Face's daily paper list for September 9, 2026, which has garnered 134 upvotes.

Evan: The title of the paper we're discussing is 'NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness'.

Ashley: It's authored by the NeoHorse Team, with the first two authors being Guoliang Cao and Guohao Dai, and the corresponding author Yu Wang from Tsinghua University.

Evan: So Ashley, can you give us a bit of context about Recursive Self-Improvement or RSI that this paper is exploring?

Ashley: Sure, Evan.

Recursive Self-Improvement, or RSI, is fundamentally about how AI systems can progressively enhance their own capabilities.

This involves refining their responses, learning from their own experiences, and even reshaping parts of their operational frameworks.

Evan: Got it.

And how does NeoHorse-1 fit into this concept of RSI?

Ashley: NeoHorse-1 is essentially an initial step towards making RSI practical.

It leverages what's known as a routing harness—a kind of execution layer that manages an agent's context and interactions.

This harness collects and utilizes interaction data to train models in an innovative way, essentially serving both as a training guide and a feedback mechanism.

Evan: Interesting.

So, what are the primary objectives the NeoHorse-1 paper aims to address?

Ashley: The paper seeks to demonstrate how a routing harness can support RSI by providing a mechanism that collects interaction trajectories—records of decisions, tool uses, and task outcomes.

These trajectories offer tangible evidence of a model's capabilities and areas for improvement, which are crucial for effective self-improvement.

Evan: So the harness acts both as a recorder of what the model does and a mentor by providing feedback based on those records?

Ashley: Exactly.

By continually collecting and analyzing these trajectories, the system can identify capability gaps and adjust its training data and processes accordingly.

This turns the model's everyday interactions into a continuous learning loop.

Evan: What specific contributions does NeoHorse-1 bring to the field of AI and RSI?

Ashley: NeoHorse-1 introduces a hybrid model pool combined with intelligent routing.

This system uses a structured Curriculum Learning approach that progressively introduces higher-capability demands in training stages, helping the model adapt to more complex tasks over time.

It also employs on-policy distillation, where a teacher model supervises the student model's responses, ensuring that the student adapts to real interaction demands.

Evan: And what kind of improvements or outcomes did NeoHorse-1 demonstrate in their evaluations?

Ashley: The paper reports substantial improvements.

The macro-average score of their 4 billion parameter model improved from 58.94 to 64.87, and the 9 billion parameter model from 65.60 to 69.04 across ten different benchmarks.

Notably, the gains were significant in areas requiring intensive execution and multi-step reasoning.

Evan: That's impressive.

It sounds like NeoHorse-1 is effectively narrowing the performance gap between smaller and larger models by using this recursive self-improvement loop.

Ashley: Indeed.

The idea of using a harness as both a training guide and a feedback provider is a significant innovation in the field.

It lays the groundwork for more advanced self-improving systems.

Evan: Thanks for that detailed overview, Ashley.

That wraps up the Introduction section of the paper.

Evan: Alright Ashley, let's dive into the Method section of the NeoHorse-1 paper.

What are the core components of their proposed method?

Ashley: The methods in NeoHorse-1 are centered on transforming interaction experience into useful training data.

This involves three main components: agentic supervision, routing-guided curriculum learning, and routing-guided on-policy distillation.

Evan: Great!

Let's start with agentic supervision.

What does that entail?

Ashley: Agentic supervision leverages the concept of interleaved thinking, where a model's interaction with its environment is recorded in full detail.

This includes user requests, the model's reasoning, tool calls, and tool results.

Essentially, these interaction records become rich training examples for supervised fine-tuning, or SFT.

Evan: So each user turn, which captures everything from the user request to the final response, becomes a training unit?

Ashley: Exactly.

Each user turn is serialized into a training sequence using a predefined chat template.

The assistant's reasoning within the current turn is retained, while earlier reasoning is omitted.

This ensures the learning context stays relevant and focused.

Evan: Got it.

Now, how about the routing-guided curriculum learning?

Ashley: Routing-guided curriculum learning organizes training examples based on their difficulty, using a capability demand score.

These scores come from the routing harness, which evaluates the complexity of each request and interactions.

The examples are introduced in a three-stage curriculum, progressively moving from simpler to more complex tasks.

Evan: How does the system determine these capability demand scores?

Ashley: The harness assigns each user turn to one of four service tiers based on the estimated complexity: C0 for low-risk requests, C1 for general-purpose tasks, C2 for multi-step reasoning and execution, and C3 for maximum capability or reliability.

These tiers are then used to organize the curriculum.

Evan: So the training progresses through these tiers, gradually increasing the complexity of examples?

Ashley: Yes, the curriculum is split into three stages that incorporate examples with increasing routing scores.

This prevents the end of training from being dominated by only high-demand interactions and ensures a balanced learning experience.

Evan: Interesting.

What about routing-guided on-policy distillation?

How does that work?

Ashley: Routing-guided on-policy distillation, or OPD, extends the concept of SFT by involving a teacher model that supervises the student model's responses generated from its own prefixes.

Essentially, it combines the student’s evolving behavior with teacher supervision to align the learning process with the actual interaction contexts that the student encounters.

Evan: Can you walk us through the actual steps involved in this distillation process?

Ashley: Sure.

The process starts by taking the recorded contexts right before the assistant's responses as starting points.

These contexts are scored for their complexity, and allocated into the same three-stage curriculum used in SFT.

The student model then generates responses for these contexts, and the fixed teacher model provides next-token distributions for each position of the response.

Evan: So the teacher model is providing supervision based on the student's generated prefixes?

Ashley: Precisely.

The teacher supervises by providing tighter supervision through what’s known as reverse Kullback-Leibler divergence, or reverse-KL.

This measures how well the student's generated token probabilities align with the teacher's token probabilities.

Evan: That sounds complex.

How does this method enhance the student model's learning?

Ashley: By aligning the student's responses with those of the teacher, the student model effectively learns to produce higher quality responses based on real interaction contexts and feedback loops.

This helps in refining the model's behavior and improving its performance over time.

Evan: So combining these techniques—agentic supervision, routing-guided curriculum learning, and on-policy distillation—enables recursive self-improvement by continually refining the model based on its own interactions?

Ashley: Yes, that's the essence of it.

The harness-driven approach captures diverse interaction experiences, aligns training with real demands, and iteratively supervises and refines the model's performance, closing the feedback loop required for RSI.

Evan: This comprehensive method seems to provide a robust blueprint for agentic training and self-improvement.

Wrapping up, these methods collectively allow NeoHorse-1 to better understand its own shortfalls and strengths, right?

Ashley: Correct.

By continually analyzing and learning from its interaction trajectories, NeoHorse-1 can adjust its training focus, evolve its understanding, and enhance its capabilities in a structured manner, showing notable performance improvements.

Evan: That concludes the Method section of the paper.

Evan: Alright Ashley, let's move on to the Experiment and Results section of the NeoHorse-1 paper.

What benchmarks and evaluations did the researchers employ to test their methods?

Ashley: The researchers evaluated NeoHorse-1 across ten benchmarks that span harness-based agents, tool use, coding, and instruction following.

These include QwenClawBench, WorkBuddy Bench, PinchBench, VitaBench, BFCL V4, and τ²-Bench.

Evan: Can you give us a bit more detail on what each of these benchmarks entails?

Ashley: Certainly.

QwenClawBench targets realistic OpenClaw tasks, while WorkBuddy Bench covers multi-domain workplace scenarios.

PinchBench focuses on standardized OpenClaw workflows, VitaBench examines multi-turn interactions in daily-life service scenarios, BFCL V4 measures function-calling and agentic tool-use capabilities, and τ²-Bench evaluates multi-turn task completion involving user–agent–tool interactions across the airline, retail, and telecom domains.

Evan: They compared different model scales, right?

What were the main findings there?

Ashley: Yes, they compared the performance of NeoHorse-1 models at 4 billion and 9 billion parameter scales.

The results showed consistent improvements across all benchmarks.

For the 4B model, the macro-average score improved from 58.94 to 64.87, and for the 9B model, it increased from 65.60 to 69.04.

Evan: Were there specific areas where these improvements were most notable?

Ashley: The largest gains were observed in harness-based and execution-intensive evaluations.

For instance, in the QwenClawBench, the post-trained 4B model significantly closed the performance gap to the 9B base model, showcasing improvements in end-to-end execution and multi-step reasoning.

Evan: That's impressive.

Did the paper mention any qualitative insights about how these improvements manifest in real tasks?

Ashley: Yes, they analyzed representative agent trajectories to illustrate these gains.

For example, a QwenClawBench project-scheduling task showed that the NeoHorse-1-4B model could retrieve additional evidence, recognize updated dependencies, recompute schedules, and write artifacts to the correct locations—all steps that the base model failed to execute coherently.

Evan: Any other notable examples highlighted?

Ashley: Another example came from the WorkBuddy code-repair task.

The NeoHorse-1-4B model initially stopped after a single implementation attempt, failing to establish an effective testing and repair loop.

In contrast, the NeoHorse-1-9B model followed a comprehensive edit–test–inspect–repair cycle, incorporating feedback until the verifier passed.

Evan: Were there any insights into the efficiency gained with these improvements?

Ashley: Yes, in tasks where NeoHorse-1-9B demonstrated significant improvements, like the PinchBench data-analysis task, the model not only completed the requested report but did so more efficiently.

The revised strategy reduced the number of model requests, execution time, and token usage by about 70.8%, 76.7%, and 83.6% respectively.

Evan: How did the different sources of supervision data compare, like routing-harness data versus public synthetic tool-agent data?

Ashley: The authors compared the NeoHorse-1 models trained on routing-harness data with those trained on Toucan, a public synthetic tool-agent dataset, using the same routing-guided training configuration.

The models trained on routing-harness data outperformed on all five comparable benchmarks, with an average improvement of 6.26 percentage points.

Evan: And how did performance scale with more supervision data?

Ashley: The results demonstrated steady improvement as the amount of unique supervised tokens increased.

For example, the average score across five benchmarks increased from 69.31 for the base model to 71.45 when the scale of supervised tokens was maximized.

Evan: So, more high-quality supervision data directly translates to better performance, which is promising for further system improvements.

Ashley: These experiments and results highlight the efficacy of NeoHorse-1's novel approach to self-improvement and agentic training.

Evan: That wraps up the Experiment section of the paper.

Evan: Alright Ashley, let's move on to the Related Work section.

How does NeoHorse-1 build on or differ from existing methods in the field?

Ashley: To understand the placement of NeoHorse-1 within the landscape of AI research, let's break down the related work into three main categories: Agentic Model Post-Training, LLM Routing and Curriculum Learning, and Recursive Self-Improvement or RSI.

Evan: Let's start with Agentic Model Post-Training.

How does NeoHorse-1 relate to this?

Ashley: Agentic Model Post-Training involves using interaction trajectories to fine-tune models.

Prior work like FireAct and AgentTuning have shown that models can learn from the sequences of decisions made during task execution.

NeoHorse-1 expands on this by recording a broader set of interactions, including various tool uses and task outcomes, to create a more enriched training set.

Evan: And there's also trajectory-based supervised fine-tuning, right?

Ashley: Exactly.

Trajectory-based supervised fine-tuning, or SFT, generally involves learning from a fixed set of teacher-guided interactions.

NeoHorse-1 enhances this by employing on-policy distillation, where a student model generates responses and a teacher model supervises these in real-time, thereby closing the feedback loop dynamically.

Evan: Do we have any examples of significant prior work in this area?

Ashley: Yes, notable examples include Agent-FLAN, which emphasizes the importance of data composition and scale for generalization, and Llama 3, which uses synthetic multi-step tool-use data to optimize model interactions iteratively.

NeoHorse-1 builds on these concepts by integrating routing-guided progression directly into its training curriculum.

Evan: How about LLM Routing and Curriculum Learning?

How is NeoHorse-1 advancing this field?

Ashley: LLM routing is about directing queries to the most suitable model, balancing quality and cost.

FrugalGPT, for example, studies cost-aware model cascades to optimize this.

NeoHorse-1 adapts this concept for agentic interaction by introducing multi-agent systems where a routing layer decides which sub-agent or model should handle each request.

Evan: And how does curriculum learning fit into this?

Ashley: Curriculum learning stages training examples based on difficulty.

Previous methods often rely on explicit labels or heuristics to determine difficulty levels.

In NeoHorse-1, the routing system provides these difficulty estimates, dynamically ordering training examples to create a more efficient learning path.

Evan: So it's like continuously adjusting the curriculum based on real engagement data?

Ashley: Precisely.

This method ensures the model encounters a balanced range of easy to difficult tasks, enabling more comprehensive learning.

Evan: And the final category, Recursive Self-Improvement, how does NeoHorse-1 contribute here?

Ashley: RSI is about models improving themselves iteratively.

For example, MetaSkill-Evolve evolves both task skills and meta-skills that govern improvement, while AREX alternates between evidence gathering and answer auditing.

NeoHorse-1 contributes by utilizing a harness to capture and evaluate model capabilities, feeding this data back into the training loop.

Evan: Any examples of how automation is being leveraged in RSI?

Ashley: Yes, works like AI4AI-Bench evaluate whether models can improve training algorithms autonomously.

NeoHorse-1 aligns with this by using routing signals to guide what the model should learn next, automating parts of the model improvement process itself.

Evan: It sounds like NeoHorse-1 brings together several advanced concepts from these areas.

Ashley: Indeed, by integrating agentic post-training, dynamic routing, curriculum learning, and recursive self-improvement, NeoHorse-1 presents a holistic approach to developing advanced, self-improving AI systems.

Evan: That wraps up the Related Work section of the paper.

Evan: Alright Ashley, let's summarize the key contributions and takeaways from the NeoHorse-1 paper.

Ashley: NeoHorse-1 introduces a framework for Recursive Self-Improvement using a routing harness.

This harness captures interaction trajectories, guiding both the training curricula and the on-policy distillation process to dynamically improve the model's capabilities.

Evan: Through agentic post-training, routing-guided curriculum learning, and on-policy distillation, NeoHorse-1 shows significant performance gains across various benchmarks, particularly in tasks requiring multi-step reasoning and execution-intensive scenarios.

Ashley: The system effectively leverages these detailed interaction records to close the evaluation–selection–update loop, creating a structure for continuous learning and self-improvement.

Evan: By narrowing the performance gap between different model scales, NeoHorse-1 demonstrates how smaller models can catch up through intelligent training methods and feedback loops, making advanced AI capabilities more accessible.

Ashley: Indeed.

The integration of routing signals and curriculum learning ensures a balanced and comprehensive approach to model training, leading to substantial improvements and efficient learning processes.

Evan: That wraps up our discussion on NeoHorse-1.

Thanks for tuning into Daily Paper Cast.

Ashley: We hope you found this episode insightful.

Be sure to join us again for more in-depth discussions on the latest in AI research.

Evan: If you enjoyed the episode, don't forget to subscribe and leave a review.

Until next time, stay curious and keep exploring!