Rethinking On-Policy Distillation of Large Language Models II: One Training Example

🤗 Upvotes: 43 | cs.AI, cs.CL

Authors:
Zixuan Fu, Bingxiang He, Yuxin Zuo, Haohuan Huang, Jinqian Zhang, Ruhang Xiao, Cheng Qian, Qinyu Luo, Huan-ang Gao, Yudong Wang, Zhiyuan Liu, Ning Ding, Chaojun Xiao

Title:
Rethinking On-Policy Distillation of Large Language Models II: One Training Example

Arxiv:
http://arxiv.org/abs/2609.04172v1

Abstract:
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure \emph{state coverage}, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches \(71.5\%\), most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach \(98.9\%\) and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision increasingly slowly. The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. Task content and induced state coverage can therefore come apart. We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast, your go-to podcast for the latest in AI research.

Ashley: Today, we're discussing a paper from Hugging Face's daily paper list for September 4, 2026, which has already received 43 upvotes.

Evan: The paper is titled 'Rethinking On-Policy Distillation of Large Language Models II: One Training Example.'

Ashley: It’s authored by Zixuan Fu, Bingxiang He, and several others.

The corresponding author is Bingxiang He from Tsinghua University.

Evan: Alright, let's dive into the introduction.

Ashley, can you start us off with some background on On-Policy Distillation or OPD?

Ashley: Sure, Evan.

On-Policy Distillation, or OPD, is increasingly being used in post-training for large language models, often in combination with supervised fine-tuning and reinforcement learning.

Well-known models like Qwen3, MiMo, and GLM-5 integrate OPD into their training pipelines.

Evan: Interesting.

What makes OPD stand out compared to other distillation methods?

Ashley: What sets OPD apart is its combination of on-policy state visitation with dense, token-level supervision from a teacher model.

Basically, the student model generates its own rollouts and receives comprehensive feedback from the teacher for each token it predicts, unlike traditional reinforcement learning, which usually relies on a single scalar reward per outcome.

Evan: That sounds pretty detailed.

So, what gap is this paper trying to address?

Ashley: Great question.

While much research has focused on understanding OPD from an algorithmic perspective—like how the method works—little has been done to explore the role of training data in this process.

This paper specifically aims to investigate how minimal data can still be effective in training, using what's called a one-shot OPD.

Evan: One-shot OPD?

That's training with just a single example, right?

Ashley: Exactly.

The authors conducted experiments where the model was trained using just one query instead of large datasets.

Surprisingly, they found that the model continues to improve for hundreds of training steps and recovers a significant portion of the performance gains you'd expect from a full dataset.

Evan: That really is surprising.

How do they explain such efficiency with so little data?

Ashley: They explain this efficiency in terms of state coverage, which is the fraction of states visited by full-data OPD that are also reached by the rollouts of their one-shot scenario.

They measured that a single query could cover 71.5% of the state space that full-data OPD would visit.

Evan: Wow, that's a large percentage.

And it only takes one query?

Ashley: Yes, and most of that coverage happens within the first 100 steps of training.

By adding more semantically distinct queries, they could increase the state coverage and validation accuracy until it nearly matched full-data training with just 16 queries.

Evan: That’s quite efficient.

Did they find any limitations or interesting dynamics in this process?

Ashley: Yes, the alignment between the student and the teacher models slowed down over time, regardless of the number of queries used.

Even with a fixed set of states, the model takes hundreds of steps to effectively absorb the supervision.

Evan: So, what’s the key takeaway from the introduction of this paper?

Ashley: The key takeaway is that OPD is described as 'data-overfed but algorithm-starved.' This means that while a single query can provide abundant supervision signals, the student's ability to absorb this information is the bottleneck.

Hence, improving the algorithm’s step efficiency could be crucial for future advancements.

Evan: That's fascinating and certainly prompts a re-thinking of how we approach training data and algorithm design for OPD.

Ashley: Indeed, it does.

And that wraps up the introduction section of 'Rethinking On-Policy Distillation of Large Language Models II: One Training Example.'

Evan: Alright, moving on to the Method section of the paper.

Ashley, can you take us through the experimental setup and key methods proposed in this study?

Ashley: The authors conducted a series of experiments to establish the effectiveness of one-shot OPD across different domains.

They focused on four main task domains—mathematical reasoning, code generation, instruction following, and agentic tool use.

Evan: Got it.

So, how did they go about selecting the models and datasets for each domain?

Ashley: For the models, they paired a student model with a corresponding post-trained teacher model from the same family.

For mathematical reasoning, they used DeepSeek-R1-Distill-Qwen-1.5B paired with JustRL-DeepSeek-1.5B as the teacher.

For code generation, it was DeepSeek-R1-Distill-Qwen-1.5B with Nemotron-Research-Reasoning-Qwen-1.5B-v2-RLVE.

Instructions following used the same student model paired with UltraData-IF-1.5B, and for agentic tool use, they used Qwen2.5-Coder-1.5B-Instruct paired with Hammer2.1-1.5b as the teacher.

Evan: That's a comprehensive setup.

What about the datasets?

Ashley: The datasets were similarly varied.

They used DAPO-Math-17K for mathematical reasoning, Open-R1 Codeforces for code generation, a sampled subset of UltraData-SFT-2605 for instruction following, and xLAM-function-calling-60K for agentic tool use.

Evan: Interesting.

So, what kind of queries did they select for the one-shot OPD experiments?

Ashley: For mathematical reasoning, they selected three queries of varying difficulty.

The easy query had a pass rate of 8 out of 8 rollouts, the medium one had 4 out of 8, and the hard one had 0 out of 8.

In other domains, the one-shot query was sampled randomly from the respective datasets.

Evan: Got it.

Now, how did they implement the on-policy distillation?

Ashley: They implemented OPD using a flexible reinforcement learning framework called veRL.

For mathematical reasoning, they optimized using a top-k advantage method with k set to 16.

For the other domains, they used a sampled-token advantage approach.

Evan: Can you break down what top-k advantage and sampled-token advantage mean in this context?

Ashley: Sure.

The top-k advantage method involves the teacher model supplying the top k likely next tokens at each position, and the student model updating based on these token distributions.

The sampled-token advantage, on the other hand, has the student align its prediction with the actual token sampled from the teacher's distribution, thereby learning from a single token at each position.

Evan: That makes sense.

What about other training details?

Any specific hyperparameters or evaluation methods?

Ashley: Yes, they used AdamW optimizer with a learning rate of 10^(-6), batch size of 64 rollouts, and a training temperature of 1.0.

For evaluation, they sampled a specific number of responses per problem—16 for math, 3 for code generation, and evaluated accordingly.

For instruction following, they used the Multi-IF benchmark, evaluating three-turn conversations across eight languages.

Evan: It seems like they've covered a lot of bases.

How did they measure the effectiveness of one-shot OPD compared to full-data OPD?

Ashley: They employed several metrics.

First, they measured validation accuracy to track performance.

They also used state coverage, which calculates the fraction of states that the student model visits compared to the full-data OPD.

They found that a single query can cover 71.5% of the states, while 16 semantically diverse queries cover 98.9%.

Evan: That's a high coverage.

Any other alignment metrics they used?

Ashley: Indeed.

They also used gap recovery, which measures the improvement in performance as a ratio to the initial performance gap between the teacher and student models.

Additionally, they looked at top-k token overlap ratio and overlap-token advantage to assess how closely the student's predictions align with the teacher's.

Evan: How well did one-shot OPD perform across these metrics?

Ashley: Remarkably well, actually.

For mathematical reasoning, one-shot OPD reached an accuracy of 68.5 compared to 69.8 for full-data OPD.

The gap recovery was about 69%, and it maintained consistency across different steps.

The overlap ratio and alignment metrics also indicated that even with a single query, the student's predictions aligned closely with the teacher's states.

Evan: Did the method also hold up across different domains and models?

Ashley: Yes, it did.

In code generation, instruction following, and agentic tool use, one-shot OPD recovered 73%, 66%, and 64% of the teacher-student gaps, respectively.

This consistency across domains and models demonstrates the robustness of one-shot OPD.

Evan: Let's talk about the dynamics they observed during training.

How did the alignment process evolve over time?

Ashley: The alignment process was quite insightful.

They observed that the distance between the teacher and student models fell gradually, and the absorption rate, which is how quickly the student model absorbs information from the teacher, decreased over time.

This indicated that while initial learning was fast, it slowed down as training progressed.

Evan: So even with a fixed set of input states, the model continued to learn for hundreds of steps?

Ashley: Exactly.

They confirmed this by holding the states fixed and observing similar gains in validation accuracy and alignment metrics.

This means that a steady supply of fresh states isn’t strictly necessary to keep the training effective over a long period.

Evan: That's quite an achievement.

Does this mean that the method is more efficient in terms of data usage?

Ashley: Yes.

By maximizing the utility of limited data, one-shot OPD proves to be highly data-efficient.

It underscores that the rate at which the model absorbs supervision is the limiting factor, not the quantity of data.

Evan: Alright, I think that covers the Method section quite comprehensively.

Ashley: Yes, that brings us to the end of the Method section.

Evan: Now, let's move on to the Experiments and Results section of the paper.

Ashley, can you start us off?

Ashley: Sure.

The authors conducted comprehensive experiments to evaluate the effectiveness of one-shot OPD across different tasks and domains.

They began by validating the method on mathematical reasoning tasks, such as MATH-500, AMC 2023, and AIME 2025.

Evan: How did one-shot OPD perform on these benchmarks?

Ashley: One-shot OPD showed impressive performance.

For instance, on MATH-500, one-shot OPD improved validation accuracy from an initial score of 59.1 to 66.9, while full-data OPD reached 70.8.

This means that one-shot OPD recovered about 94.4% of the gains achieved by full-data OPD by step 300.

Evan: That’s pretty close to full-data performance.

What about the other benchmarks?

Ashley: The results were consistent across AMC 2023 and AIME 2025 as well.

By step 300, one-shot OPD reached 68.5 compared to 69.8 for full-data OPD, recovering 87% of the performance gains on average.

Evan: Impressive results for mathematical reasoning.

How did one-shot OPD fare in other domains, like code generation and instruction following?

Ashley: In code generation, they evaluated using LiveCodeBench v6, which reports the average accuracy over sampled code solutions.

One-shot OPD achieved 25.5, while full-data OPD reached 27.3, recovering around 93.4% of the gains.

Evan: That is indeed consistent.

How did it perform in the instruction following and agentic tool use tasks?

Ashley: For instruction following, evaluated using Multi-IF, one-shot OPD improved accuracy from 24.2 to 27.5, while full-data OPD reached 28.3.

In agentic tool use, measured by BFCL v3, one-shot OPD achieved 72% of the full-data OPD’s gains, maintaining consistency across domains.

Evan: What do you think contributes to such robust results across different tasks and domains?

Ashley: It boils down to the efficient state coverage provided by even a single query, which we discussed earlier.

This allows the student model to align well with the teacher even when trained on minimal data, leveraging the dense token-level feedback to its maximum potential.

Evan: Did they explore the robustness of one-shot OPD against different query difficulties or other parameters?

Ashley: Yes, they did.

They tested one-shot OPD under a variety of conditions, including different query difficulties, varying response-length budgets, and different rollout sampling temperatures.

One-shot OPD consistently showed significant gains regardless of these variations.

Evan: That’s good to know.

How about the dynamic metrics during training, like overlap ratios and absorption rates?

Ashley: The top-16 overlap ratio and overlap-token advantage improved steadily during training, reaching levels comparable to full-data OPD.

Even the absolute entropy gap between the student and teacher models nearly closed, indicating effective alignment even with limited data.

Evan: So, it seems like one-shot OPD maintains strong alignment throughout training.

Were there any specific observations related to the training dynamics over time?

Ashley: Yes, they found that while the student model's alignment with the teacher model improved continuously, the absorption rate—how quickly the student absorbs the teacher’s supervision—declined over time.

This aligns with their earlier observation that OPD is 'algorithm-starved,' meaning that the learning process becomes slower as training progresses, regardless of the data amount.

Evan: So the model keeps learning but at a decreasing rate.

Were there any practical implications for OPD identified in their experiments?

Ashley: One key practical takeaway is the potential to significantly reduce training data without substantially compromising on performance.

This could lower computational costs and make the training process more efficient, especially for resource-intensive models.

Evan: That’s a valuable insight.

Anything else worth noting from their experimental results?

Ashley: They also tested the method in a multi-teacher OPD (MOPD) setup, training across multiple domains.

They found that using just 16 semantically diverse queries per domain could match the performance of full-data MOPD, further validating the method’s robustness and efficiency.

Evan: Fascinating.

It's impressive how well one-shot OPD generalizes across different tasks, models, and even multi-domain setups.

Ashley: This section highlights the effectiveness and robustness of one-shot OPD across a variety of settings, reinforcing its potential as a highly efficient training method.

Evan: That wraps up our coverage of the Experiment section.

Evan: Next, let's delve into the Related Work section.

Ashley, can you walk us through the work that has been done in this area before and how this paper fits into the broader landscape?

Ashley: Of course.

The paper sits at the intersection of several research domains, including on-policy distillation, data-efficient reasoning post-training, and synthetic and unsupervised post-training data.

Let's start with on-policy distillation.

Evan: Perfect.

What have been the key developments in on-policy distillation?

Ashley: On-policy distillation has gained traction as a method for training language models on their own rollouts with supervision from a teacher model.

Notable works include MiniLLM, which optimized distillation under reverse KL divergence by policy gradient, and GKD, which generalized the method across various divergences and on- and off-policy mixtures.

Evan: Sounds like substantial groundwork has been laid.

How does this paper differentiate itself from those efforts?

Ashley: Previous research primarily focused on the algorithmic aspects of OPD, such as its objective function and update geometry.

This paper, in contrast, zeroes in on how training data influences OPD, particularly through the lens of minimal data use.

It explores the one-shot setting to see how a single query can achieve significant performance gains.

Evan: Interesting.

What about data-efficient reasoning post-training?

How does that relate?

Ashley: Data-efficient reasoning post-training aims to minimize the amount of data required to elicit strong reasoning capabilities from models.

This includes works like LIMR and LIMO, which focus on pruning the training set to retain high-impact examples.

One-shot RLVR has also shown that a single example can keep a model improving over many steps.

This paper extends that concept to OPD, demonstrating that significant gains can be made with extremely limited data.

Evan: How does synthetic and unsupervised post-training data come into play?

Ashley: Synthetic and unsupervised post-training data models rely less on curated, human-written data.

For instance, Self-Instruct and Evol-Instruct use language models to generate training data.

Magpie goes even further, generating task data from simple chat templates.

This paper's use of content-light templates and off-domain WildChat queries shows that even without explicit task content, useful training signals can be generated, aiding OPD’s effectiveness.

Evan: So, there's a significant focus on generating useful training data efficiently.

Ashley: Exactly.

The efficiency of data utilization is a recurring theme.

By showing that one-shot OPD achieves a large fraction of the performance gains of full-data OPD, this work aligns with the broader trend of maximizing the value derived from minimal data.

Evan: Any other key works or concepts mentioned that are particularly influential?

Ashley: Yes, the paper references a growing body of work on reinforcement learning without ground-truth labels.

Techniques like relying on model-generated signals, entropy, and majority agreement across rollouts are gaining traction.

This paper's findings align well with these approaches, emphasizing that rich, actionable feedback can be derived even from non-traditional sources of data.

Evan: That’s quite comprehensive.

How well does this paper integrate with these existing approaches?

Ashley: It integrates seamlessly.

By addressing the data-minimal limit, this paper builds on prior works focusing on data efficiency and unsupervised learning.

It bridges gaps between purely algorithmic advancements and practical data limitations, providing a more holistic perspective on OPD.

Evan: So, would it be fair to say this paper pushes the envelope on our understanding of OPD?

Ashley: Definitely.

It not only builds on the existing algorithmic insights but also introduces a new dimension by rigorously quantifying the role of training data.

This shifts the focus from purely expanding data collections to optimizing how the available data is utilized.

Evan: That’s a significant contribution.

Any final thoughts on the related work before we move on?

Ashley: Just that this paper provides essential insights for future research directions, particularly in areas where data collection is expensive or time-consuming.

It underscores the importance of focusing not just on quantity but also on the quality and effective use of training data.

Evan: Great insights, Ashley.

That wraps up our discussion on the Related Work section.

Evan: We've covered a lot of ground today, so let's wrap up by summarizing the key contributions and takeaways from this paper.

Ashley: The paper 'Rethinking On-Policy Distillation of Large Language Models II: One Training Example' makes several significant contributions.

Firstly, it demonstrates that one-shot OPD recovers a substantial portion of the performance gains of full-data OPD across various domains and tasks.

Evan: That's right.

The authors showed that even with a single query, the student model could cover 71.5% of the states visited by full-data OPD, and using just 16 diverse queries, it could achieve 98.9% coverage.

Ashley: They also found that the alignment process of the student model with the teacher model slowed over time, indicating that while the model continues to learn, the rate of absorption declines.

This highlights the idea that OPD is 'data-overfed but algorithm-starved,' meaning the bottleneck lies in the model's ability to absorb supervision efficiently.

Evan: Another key takeaway is the method's robustness.

One-shot OPD showed consistent performance across different domains, including mathematical reasoning, code generation, instruction following, and agentic tool use.

Ashley: Lastly, the research emphasized the efficiency of data utilization, showing that significant performance improvements can be achieved with minimal data.

This has important implications for reducing training costs and optimizing resource use.

Evan: Thanks for those insights, Ashley.

That brings us to the end of today’s episode.

Ashley: We hope you found this discussion on one-shot OPD as fascinating as we did.

If you enjoyed the episode, don't forget to subscribe and leave us a review.

Evan: And make sure to tune in next time for more cutting-edge research insights.

Thank you for listening to Daily Paper Cast.

Ashley: Until next time, stay curious and keep learning!