Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills

🤗 Upvotes: 257 | cs.AI, cs.CL

Authors:
Jianlyu Chen, Yuyang Hu, Hongjin Qian, Jiawei Liu, Wenqing Wei, Xiaolong Chen, Defu Lian, Zhicheng Dou, Chaozhuo Li, Qiwei Ye, Zheng Liu

Title:
Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills

Arxiv:
http://arxiv.org/abs/2609.02749v1

Abstract:
Autonomous agents are beginning to carry out machine-learning (ML) research end to end. These agents combine a model backbone with a harness for planning, execution, memory, and verification, but this architecture still leaves domain-specific know-how outside the agent. We call this missing layer operational knowledge, the know-how that separates knowing a method from making it work. That knowledge is not absent from the field. It appears in repositories and papers, but in forms written for human readers and too large to load during a task. Once distilled into compact, verified skills, this knowledge can be reused across tasks rather than rediscovered during each run. We present DisCo, a skill-powered research agent that creates skills and uses them during research. Its distillation runs in two complementary forms: task-agnostic, condensing the field's widely used repositories into reusable skills, and task-oriented, producing the skills a concrete task calls for. The former, applied across the open ecosystem, yields the AREX-Skill Library, with 5,000+ verified skills distilled from 1,000 widely used ML repositories and organized into 20 areas and 178 capability families. With the GPT-5.5 backbone, research harness, and downstream execution budget held fixed, the skill-equipped research agent scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet than the same agent without skills. These gains come from adding distilled operating context under that fixed setup.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast, your source for the latest in AI research.

Ashley: Today’s paper is from the Hugging Face daily paper list of September 3, 2026, and it's garnered 257 upvotes.

Evan: The title of the paper is 'Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills'.

Ashley: This paper is authored by Jianlyu Chen and Yuyang Hu, among others.

The corresponding author is Zheng Liu from the Beijing Academy of Artificial Intelligence.

Evan: So, Ashley, what’s the main focus of this paper?

Ashley: The paper addresses a critical gap in the functionality of autonomous machine-learning agents.

It highlights that while these agents have advanced in executing tasks like model implementation and experiment management, they lack operational knowledge.

This knowledge is crucial because it bridges the gap between knowing a method theoretically and making it work practically.

Evan: That’s an interesting observation.

Why exactly is operational knowledge so important?

Ashley: Operational knowledge is what separates knowing an algorithm from knowing how to effectively implement and deploy it.

Without this, agents end up wasting computational resources on trial-and-error to figure out package behaviors and configurations, leading to inefficient task execution and loss of valuable insights that could be reused across different tasks.

Evan: So what solution do the authors propose?

Ashley: The authors introduce DisCo, a skill-powered research agent designed to distill operational knowledge from repositories and papers into compact, reusable skills.

These skills are then used by the agent to enhance its research capabilities.

Evan: How does DisCo work?

Ashley: DisCo operates in two modes: task-agnostic and task-oriented.

The task-agnostic mode distills broadly applicable skills across widely used repositories, forming a comprehensive skill library called AREX-Skill.

The task-oriented mode, on the other hand, generates specific skills needed for a concrete research task at hand.

Evan: So it’s like having a dynamic toolkit that updates and adapts based on the agent’s specific needs.

That's fascinating.

And what does the AREX-Skill Library contain?

Ashley: The AREX-Skill Library is quite extensive.

It includes over 5,000 verified skills distilled from 1,000 widely used machine-learning repositories.

These skills are organized into 20 areas and 178 capability families, making it a rich resource for any machine-learning task.

Evan: That sounds incredibly useful.

Have the authors provided any evaluation of DisCo’s performance?

Ashley: Yes, they have.

The evaluation shows that equipping a research agent with these distilled skills leads to significant performance improvements across several benchmarks.

Specifically, the skill-equipped agent scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet compared to an agent without these skills.

Evan: Which elements contribute to these improvements?

Ashley: The improvements are attributed to the addition of distilled operating context.

Essentially, the skills provide relevant, pre-verified procedures and checks that guide the agent, eliminating much of the trial-and-error that would otherwise occur.

This operational layer complements the model and harness, leading to better-planned experiments and quicker recoveries from failures.

Evan: That's definitely an upgrade.

Summarizing the contributions of the paper, what stands out?

Ashley: Four key contributions stand out.

First, they identify operational knowledge as the missing layer of autonomous research agents.

Second, they introduce DisCo, the skill-powered research agent.

Third, they build the extensive AREX-Skill Library.

Finally, they provide a thorough evaluation of the benefits of using distilled skills in various research scenarios.

Evan: And that brings us to the end of the Introduction section of the paper.

A promising approach to enhancing autonomous research agents with distilled skills from repositories and papers.

Evan: Ashley, let's dive a bit deeper into the methods proposed by DisCo.

How exactly does it distill operational knowledge into skills?

Ashley: Sure, Evan.

DisCo uses a four-stage process for skill distillation: scope capability, ground evidence, construct skill graph, and verify & refine.

Evan: Alright, first things first, can you explain what’s involved in scoping capability?

Ashley: In scoping capability, DisCo identifies which capabilities are relevant to the given research task.

This involves understanding the problem domain and selecting the admissible evidence from sources like repositories or papers.

The goal is to define the boundaries of what the skill should accomplish.

Evan: And what happens in the grounding evidence stage?

Ashley: In the grounding evidence stage, DisCo collects evidence to support these capabilities.

This evidence could come from code examples, documentation, and academic papers.

The collected evidence is then mapped to the defined skill boundaries.

Evan: And then we move on to constructing the skill graph.

What's that about?

Ashley: Construction of the skill graph involves writing agent-facing skills that encode the use conditions and procedures.

Each skill is packaged with scripts and references, and dependency or composition links are added to form a connected, navigable graph.

Evan: So, it's making an interconnected map of skills from the gathered evidence.

Ashley: Exactly.

Finally, in the verification and refinement stage, the candidate skill graph is tested and refined through checks or task trials.

This ensures that all skills are operationally valid and any gaps or issues are documented and addressed.

Evan: That sounds comprehensive.

How are these skills actually organized and stored?

Ashley: Skills in AREX-Skill are organized into three layers: SKILL.md, references, and scripts.

SKILL.md provides the knowledge interface, outlining what the agent needs to know upfront.

References include API documentation, algorithmic details, and configuration parameters that are accessed as they become relevant.

Scripts are executable wrappers for routine actions.

Evan: So, the agent starts with a summary and only delves deeper when needed, allowing it to hold many skills efficiently.

Ashley: Yes, and this is known as the principle of 'progressive disclosure.' It helps the agent manage its context budget effectively.

Evan: What about the task-agnostic and task-oriented distillation you mentioned earlier?

How do they differ?

Ashley: Task-agnostic distillation works across the open ecosystem, distilling skills that can be reused in any relevant task.

Task-oriented distillation, on the other hand, focuses on creating skills specific to a particular research task, derived from the exact requirements and nuances of that task.

Evan: Interesting.

How does DisCo handle tasks in practice?

Ashley: DisCo handles tasks by creating an initial skill graph, refining it based on execution feedback, and storing verified skills in the AREX-Skill Library.

For example, in MLE-bench evaluations, DisCo constructs separate operational-knowledge skill graphs for each competition task, first exploring modeling and execution decisions, and then optimizing the solution through skill-guided trials.

Evan: And what does the verification process involve specifically?

Ashley: Verification involves running assertion-backed usability cases, safe native checks, and static checks.

Any failures or gaps found during this process are addressed locally until the skill graph is verified as operationally valid.

Evan: This must be quite resource-intensive.

How is efficiency managed?

Ashley: Efficiency is managed through careful evidence scoping and iterative refinement.

For example, each repository skill undergoes pre-scoping to define an inclusion boundary, preventing unrelated or redundant data from entering the skill graph.

This keeps the construction phase lean while ensuring completeness.

Evan: How does this all tie together with the overall research agent workflow?

Ashley: In researcher mode, DisCo loads relevant skills from the AREX-Skill Library, based on the task at hand.

The agent then uses these skills to guide its execution steps efficiently.

During each step, it only loads the necessary parts of the skill graph, following the indexed paths.

This ensures the agent operates within its context and computation budget while leveraging extensive operational knowledge.

Evan: So, in essence, DisCo acts as both a creator of autonomous research skills and a user of these skills within a task-specific context.

That’s a well-rounded approach.

Ashley: Yes, and this dual mode of operation—creator and researcher—not only optimizes the task execution but also scales operational knowledge efficiently across different tasks and research settings.

Evan: It’s impressive how they’ve integrated this progressive, layered approach to knowledge handling.

Definitely a step forward in autonomous research.

Ashley: Indeed.

And that wraps up the Method section of the paper.

We’ve covered how DisCo creates and uses operational knowledge, the detailed four-stage distillation process, and its applications in real-world research tasks.

Evan: Now, let's dive into the Experiment and Results section.

How did the authors evaluate DisCo, and what did they find?

Ashley: The authors conducted evaluations across four different benchmarks: MLE-bench, PaperBench, FrontierCS, and PassNet.

These evaluations compared the performance of a research agent equipped with AREX-Skill against the same agent without these distilled skills.

Evan: Right.

Let's start with MLE-bench.

What were the findings there?

Ashley: For MLE-bench, the skill-equipped agent scored 72.89% Any-Medal across 75 tasks, compared to just 31.11% for the agent without skills.

This represents a significant 134.3% improvement.

The performance gains were consistent across all difficulty tiers, with the most substantial improvement seen in high-difficulty tasks, where the score increased from 13.33% to 62.22%.

Evan: That’s a huge improvement.

What about the PaperBench evaluations?

Ashley: On PaperBench, which evaluates the ability to replicate results from 20 papers, the skill-equipped agent achieved an average replication score of 39.59%, up from 29.45% without skills.

Noteworthy improvements were found in papers related to complex domains like deep reinforcement learning and domain adaptation.

Evan: And what did the results look like for FrontierCS?

Ashley: FrontierCS, which involves solving open-ended computer science problems, showed an average score increase from 70.63% to 77.14%, marking a 9.22% relative improvement.

The enhanced agent performed better particularly on tasks below a 50-point baseline, effectively recovering low-scoring problems.

Evan: That’s impressive.

And lastly, what were the findings for PassNet?

Ashley: In the PassNet benchmark, which targets graph-compiler pass generation, the agent with AREX-Skill scored an AS Score of 1.5313 compared to 1.343 for the agent without skills, representing a 14.0% improvement.

Skill-equipped runs also reduced failure rates significantly, with failed samples dropping from 14 to just 5.

Evan: Wow.

These numbers clearly indicate it’s beneficial to equip agents with operational skills.

Did the authors provide any qualitative insights or specific examples from these evaluations?

Ashley: Yes, they did.

For instance, in MLE-bench, skill-equipped agents were able to avoid common pitfalls by using pre-verified workflows, resulting in more efficient resource use.

In PassNet, the agent leveraged procedural guidance from skills to produce more reliable compiler passes, highlighting both increased correctness and performance optimization.

Evan: How does this translate to real-world improvements?

Ashley: In practical terms, using AREX-Skill means that research agents can achieve higher computational efficiency and effectiveness.

This reduces the overall time and resources needed for complex machine-learning tasks, accelerates innovation, and facilitates reproducibility in research.

Evan: Were there any specific benchmarks or tasks where the impact of skills was most noticeable?

Ashley: The impact was most noticeable on tasks characterized by complexity and a high potential for trial-and-error.

Large gains were observed in high-difficulty tasks within MLE-bench and complex problem-solving tasks in FrontierCS, demonstrating that the benefits of operational knowledge become more pronounced as task difficulty increases.

Evan: That's a critical insight.

Anything else the authors discussed in the Results section?

Ashley: They also performed ablation studies to show the individual contributions of different components in the skill distillation process.

It was clear that each component—scoping, grounding, constructing, and verifying—contributed to the overall improvements, emphasizing the importance of a comprehensive approach to operational knowledge.

Evan: That makes sense.

When you isolate the contributions, you can see how each part of the process adds to the overall gains.

Ashley: Indeed.

And that brings us to the end of the Experiment and Results section.

We've examined how the agent equipped with AREX-Skill outperformed its baseline across various challenging benchmarks, demonstrating the powerful impact of integrating distilled operational knowledge.

Evan: Now, let’s move on to the Related Work section.

How does this paper situate itself within the broader landscape of autonomous machine-learning research?

Ashley: The authors review various strands of related work to contextualize their contributions.

They discuss autonomous systems for machine-learning research, focusing on how current systems handle ideation, implementation, experimentation, and writing in a loop.

Evan: What examples did they provide in terms of autonomous research systems?

Ashley: They reference several notable systems.

For example, systems that generate and refine research ideas over the scientific literature, as studied by Baek et al.

in 2025, and large-scale human studies by Si et al.

in 2025 that compare LLM-generated and expert research ideas.

Evan: That’s interesting.

How does DisCo differentiate itself from these prior systems?

Ashley: DisCo differentiates itself by focusing on operational knowledge, which is the missing link in many current systems.

While other systems may focus on improving the model or the orchestration, DisCo emphasizes creating and using distilled, reusable skills that guide the agent through complex tasks with less trial-and-error.

Evan: Where does this fit within the field of ML engineering and data science?

Ashley: In ML engineering and data science, several approaches are noted for their propose–run–evaluate loops, reuse of prior solutions, or multi-agent competition pipelines.

DisCo complements these by offering a scalable operational-knowledge layer that can be consumed by compatible agent harnesses.

Evan: Do the authors discuss any performance benchmarks for these autonomous research agents?

Ashley: Yes, the authors discuss existing benchmarks that measure end-to-end ML experimentation, engineering, research, and paper reproduction.

These include MLE-bench, PaperBench, FrontierCS, and others.

They argue that DisCo’s skill-based operational layer can enhance performance on these benchmarks.

Evan: What about the concept of agent skills specifically?

How does this paper build on previous work?

Ashley: Agent skills, as defined by prior works like those from Anthropic in 2025, include SKILL.md files that describe activation conditions, procedures, and tool-use strategies.

However, there are inconsistencies in authoring these skills at scale.

DisCo addresses these challenges by automating the creation of verified, high-quality skills that are integration-ready.

Evan: Are there examples of systems that automatically acquire reusable knowledge?

Ashley: Yes, systems like Voyager, Agent Workflow Memory, and ExpeL have investigated automatically acquiring procedural knowledge from agent experience.

Voyager, for instance, learns and composes executable skills across environments.

However, these methods often produce free-form workflows that are hard to verify.

DisCo, contrastingly, ensures skill quality through its rigorous verification process.

Evan: How does the verification in DisCo compare to these methods?

Ashley: The verification process in DisCo is more structured and focused.

While Voyager and similar methods derive skills from interaction traces, DisCo distills version-specific operational knowledge from static ML artifacts like repositories and papers, ensuring that each skill is grounded in proven evidence and documented effectively.

Evan: That sounds like a significant improvement in terms of reliability.

Ashley: Yes, the explicit validation records and the focus on documentation make DisCo’s skills far more reliable and beneficial for complex ML tasks.

This approach also means DisCo’s skills can be readily integrated into various research harnesses without needing substantial modifications.

Evan: So, the paper essentially situates DisCo within the larger trajectory of autonomous ML research but brings a clear emphasis on the practical deployment of operational knowledge.

Ashley: Exactly.

By addressing the operational knowledge gap, DisCo enhances the practical applicability of autonomous research agents, making them more resource-efficient and effective.

Evan: That's really insightful.

Understanding the broader context definitely helps appreciate the novelty and the necessity of DisCo.

Ashley: Indeed.

And that brings us to the end of the Related Work section.

We've seen how DisCo fits into the larger landscape of autonomous ML research and how it addresses specific gaps by focusing on operational knowledge.

Evan: Ashley, we’ve discussed various aspects of the paper 'Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills'.

Let’s summarize the key contributions and takeaways.

Ashley: The paper makes four main contributions.

First, it identifies operational knowledge as the missing layer in autonomous research agents.

This knowledge bridges the gap between theoretical method understanding and practical implementation.

Evan: Right, and second, it introduces DisCo, a skill-powered research agent that creates and uses these operational skills.

DisCo operates in two modes, task-agnostic and task-oriented, ensuring adaptability and relevance for various research tasks.

Ashley: Third, the paper presents the AREX-Skill Library, which contains over 5,000 verified skills distilled from 1,000 widely used ML repositories.

This extensive library is organized into 20 areas and 178 capability families, making it a comprehensive resource for ML research.

Evan: Lastly, the evaluation results are compelling.

They show that the skill-equipped agent significantly outperforms the baseline across multiple benchmarks, including MLE-bench, PaperBench, FrontierCS, and PassNet.

Ashley: In summary, the integration of operational knowledge through DisCo not only enhances the performance of autonomous research agents but also improves the efficiency and reliability of complex ML tasks.

This is a significant step toward more practical and effective ML research automation.

Evan: And that wraps up today’s episode.

We hope you found this deep dive into 'Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills' insightful.

Ashley: Thank you for joining us.

Be sure to tune in tomorrow for another episode of Daily Paper Cast, where we bring you the latest and greatest in AI research.

Evan: Until next time, stay curious and keep exploring the frontiers of AI.

Ashley: Goodbye, everyone!

Evan: Goodbye!