Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation

🤗 Upvotes: 57 | cs.LG, cs.CL

Authors:
Youngrok Park, Sangmin Bae, Hojung Jung, Jongwoo Ko, Yunseon Choi, Young Jin Kim, Pashmina Cameron, Aaron Courville, Se-Young Yun

Title:
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation

Arxiv:
http://arxiv.org/abs/2609.08798v1

Abstract:
Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This question is particularly important for successive model generations and multi-domain consolidation, where repeating frontier-scale post-training from scratch can be prohibitively expensive. Yet conventional distillation treats the weak teacher as an optimization target, potentially imposing its capacity ceiling on the student. We introduce On-Policy Reverse Distillation (OPRD), which evaluates the teacher's policy shift relative to its reference policy on student rollouts and amplifies the component of the student's verifier-driven policy gradient along that direction. By rescaling only verifier-supported updates, OPRD preserves the stationary points of policy optimization while accelerating learning beyond the teacher. In both successive model transfer and multi-teacher distillation, OPRD achieves higher performance with fewer student updates than existing RL and distillation approaches. Response-style analysis shows that OPRD students remain closer to models trained with verifier-based RL alone than to their weak teachers, suggesting that teacher guidance accelerates rather than redirects the student's own optimization. Results in conventional strong-to-weak distillation further demonstrate that OPRD effectively combines verifier-driven policy optimization with teacher guidance regardless of capacity ordering.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Ashley: Today, we're diving into a fascinating paper from the Hugging Face daily paper list of September 9, 2026, which has garnered 57 upvotes.

Evan: The paper is titled 'Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation.'

Ashley: It's authored by Youngrok Park and Sangmin Bae from KAIST AI, with correspondence to Se-Young Yun, also from KAIST AI.

Evan: Alright, let's jump into the introduction.

Ashley, why is this research significant?

Ashley: Great question, Evan.

The paper tackles the intriguing concept of weak-to-strong generalization.

This concept explores whether stronger models can learn from weaker supervisors and then surpass them in performance.

Evan: Why is this important in the broader context of AI development?

Ashley: It's particularly relevant for the development of successive generations of models and consolidating multiple domains.

Starting from scratch with large models for each iteration can be prohibitively expensive.

If stronger models can learn efficiently from weaker ones, it would significantly reduce computational resources and time.

Evan: I see.

How does conventional knowledge distillation relate to this?

Ashley: Conventional distillation transfers knowledge from a 'teacher' model to a 'student' model.

However, in this process, the weak teacher's limitations often become the optimization target, potentially capping the student model's performance.

Evan: So, how do the authors propose to overcome this limitation?

Ashley: The authors introduce On-Policy Reverse Distillation, or OPRD.

This method evaluates the teacher's policy shift relative to its reference policy using the student model's rollouts.

It then amplifies the student's verifier-driven policy gradient along that direction.

Evan: Interesting.

And why is this beneficial compared to traditional methods?

Ashley: By focusing only on updates supported by the verifier, OPRD accelerates learning beyond the teacher's capabilities without imposing the teacher's limitations.

This ensures that the student can surpass the teacher model and achieve higher performance with fewer updates.

Evan: That sounds innovative.

Any key results from this approach?

Ashley: Yes, indeed.

The authors demonstrate superior performance with both scenarios: successive model transfer and multi-teacher distillation.

The student models trained with OPRD not only catch up to their weak teachers faster but also achieve higher final performance than existing reinforcement learning and distillation approaches.

Evan: And how do they measure this improvement?

Ashley: The evaluation focuses on mathematical and logical reasoning tasks.

For example, in successive model transfer, an 8-billion parameter student model trained with OPRD reaches weak-teacher performance with 33 to 67 percent fewer updates.

It also shows up to 22.7 percentage points higher performance at early checkpoints.

Ashley: In multi-teacher settings, OPRD consolidates four smaller specialized teacher models into a single stronger student model, achieving teacher-level performance with 55 percent fewer updates, and ultimately outperforming all four specialists.

Evan: Remarkable results.

Any insight into how OPRD compares to existing methods, aside from the results?

Ashley: The response-style analysis shows that OPRD-trained students remain closer to models trained with verifier-based reinforcement learning alone than to their weak teachers.

This suggests that teacher guidance accelerates the student's optimization rather than redirecting it.

Evan: What about scenarios outside weak-to-strong distillation?

Ashley: Interestingly, their results in conventional strong-to-weak distillation also demonstrate that OPRD effectively combines verifier-driven policy optimization with teacher guidance, regardless of the capacity ordering.

Evan: So, what are the key contributions outlined by the authors?

Ashley: The paper's key contributions include a comprehensive study of transferring post-training gains from weaker models to stronger students without making the weak policy or its shift an optimization target.

They introduce and validate OPRD, showing substantial improvements over existing reinforcement learning and distillation methods in various tasks.

Evan: Fascinating overview.

And that wraps up the introduction section of this paper.

Stay tuned as we dive into the methods next.

Evan: Alright, Ashley, let's dive into the methods behind On-Policy Reverse Distillation.

How does this mechanism fundamentally work?

Ashley: Sure, Evan.

The core idea of On-Policy Reverse Distillation, or OPRD, revolves around transferring the policy changes learned during a weak model's post-training to accelerate the optimization of a stronger student model.

Evan: How does OPRD achieve this transfer?

Ashley: OPRD works by evaluating the teacher model's policy relative to its reference policy at each state visited by the student.

This helps in identifying the local direction of the teacher's policy change.

OPRD then uses its alignment with the verifier-driven student gradient to rescale only the gradient component along that direction.

Evan: So, rather than matching the teacher policy directly, it modifies the student’s gradient.

Why is rescaling important here?

Ashley: Good question.

By rescaling the student's gradient, OPRD accelerates the optimization process without setting an independent optimization target based on the weak teacher.

This ensures that the student can progress beyond the teacher's limitations.

Evan: Can you break down the steps involved in this method?

Ashley: The steps are as follows: First, during the student’s on-policy rollouts, OPRD computes the verifier-driven policy gradient at each token.

It then extracts the policy shift of the weak teacher relative to its reference policy and projects the student gradient onto this local direction of change.

Evan: What happens after projecting the gradient?

Ashley: Once the student gradient is projected onto this direction, the method amplifies only this projected component while keeping the orthogonal component unchanged.

This way, it augments the verifier-supported updates without deviating from the student’s own optimization trajectory.

Evan: Got it.

How does OPRD manage the alignment between the student gradient and the teacher shift?

Ashley: OPRD uses positive-alignment scaling, which is active from the beginning, to amplify the updates that both the verifier and the teacher support.

Negative-alignment scaling is gradually introduced to reinforce updates that deviate from the teacher when supported by the verifier, thus enabling the student to extend beyond the teacher’s learned behavior.

Evan: That's pretty detailed.

Can you explain how the policy shift is captured and used?

Ashley: Sure.

The teacher’s policy shift is computed as the mean-centered difference between the logits of the teacher’s post-trained policy and its reference policy at the student-visited prefix.

This difference, denoted as Δ_t, captures the changes induced during the teacher’s post-training and is then normalized to a unit direction vector, d_t.

Evan: And how is the gradient scaled?

Ashley: At each token, the student’s gradient is decomposed relative to d_t into aligned and orthogonal components.

The aligned component is then adjusted by a scaling factor λ_t, which is non-negative.

Depending on the alignment coefficient—whether the gradient follows or opposes the teacher direction—this scaling either reinforces or redirects the update.

Essentially, the new gradient, g_t tilde, is the sum of the original gradient and the scaled projection onto d_t.

Evan: That sounds like a refined mechanism.

How does OPRD ensure the optimization remains stable?

Ashley: The method ensures stability by maintaining the stationary points of the policy objective in the logit space.

The amplification only adds a non-negative first-order alignment gain, allowing the gradient to retain its core optimization properties, ensuring that the optimization progresses meaningfully.

Evan: Let’s talk about experiments then, how did they evaluate OPRD?

Ashley: In their experiments, the authors evaluated OPRD across mathematical reasoning and logical reasoning tasks.

They employed weak-to-strong scenarios, such as successive model transfer and multi-teacher distillation.

Evan: What did the experiments reveal?

Ashley: For successive model transfer, OPRD allowed an 8-billion parameter student model to reach the performance of a 4-billion parameter weak teacher significantly faster than the baseline methods.

Specifically, it reached the teacher's performance with 33 to 67 percent fewer updates.

Evan: And they saw improvement with fewer updates, right?

Ashley: Correct.

It also demonstrated up to 22.7 percentage points improvement at early checkpoints compared to existing reinforcement learning and distillation techniques.

Evan: What about in a multi-teacher setting?

Ashley: In a multi-teacher distillation scenario, OPRD effectively distilled the capabilities of four specialized 4-billion parameter teachers into a single 8-billion parameter student.

Interestingly, it achieved teacher-level performance with 55% fewer updates than Mix-RL, another method evaluated.

Ultimately, the student outperformed all four specialized teachers.

Evan: Impressive.

Did they observe any changes in the student models’ behavior?

Ashley: Yes, the models trained with OPRD showed a closer resemblance to models optimized with verifier-based reinforcement learning alone rather than the weak teachers.

This suggests that teacher guidance under OPRD accelerates the student’s optimization rather than simply directing it.

Evan: So, essentially, OPRD helps the student not just learn faster but potentially achieve a higher end performance.

Ashley: Exactly, Evan.

And it's worth mentioning that the authors confirmed that these gains aren't just limited to weak-to-strong settings.

They also demonstrated effectiveness in traditional strong-to-weak distillation scenarios.

Evan: Comprehensive work indeed.

And that wraps up the Methods section of the paper.

Next, we'll dig into the experiments and results to see the practical outcomes.

Evan: Let's dive into the core of the research— the experiments and the results.

Ashley: The experiments conducted in this paper are divided into two main scenarios: successive model transfer and multi-teacher weak-to-strong distillation.

Evan: Alright, Ashley.

Let's start with the experimental setup.

What tasks and models did they use?

Ashley: For the mathematics tasks, they trained on the DAPO-Math-17K dataset and evaluated on various benchmarks like AIME'24, AIME'25, HMMT'25, and OlympiadBench.

For the reasoning tasks, they used four benchmarks from Reasoning Gym: Knights & Knaves, Quantum Lock, String Manipulation, and Countdown.

Evan: What about the models they tested?

Ashley: They used teacher and student models from the Qwen3 family.

Each teacher was a post-trained 4-billion parameter model, while the students were larger, 8-billion parameter models.

Evan: Got it.

And who or what did they compare OPRD against?

Ashley: For single-teacher experiments, they compared OPRD with GRPO, OPD, and KDRL baselines.

For multi-teacher distillation, they included Mix-RL, MOPD, and KDRL as baselines, representing a range of verifier-only, distillation-only, and hybrid approaches.

Evan: Alright, let's talk about the results in the successive model transfer scenario.

What did they find?

Ashley: In the successive model transfer setting, the student models trained with OPRD outperformed the other baselines by a significant margin.

For mathematics, it reached the weak teacher’s performance with 33 to 67 percent fewer updates than GRPO.

Additionally, the student achieved up to 22.7 percentage points higher performance at early checkpoints.

Evan: What was the trend in the Reasoning Gym tasks?

Ashley: On Reasoning Gym tasks, OPRD also surpassed the baselines.

For example, on Knights & Knaves, OPRD reached the weak-teacher performance faster and continued to improve beyond it, achieving an average improvement of approximately 10.80 percentage points over the strongest baseline by the end of the training.

Evan: That’s impressive.

What about in the multi-teacher distillation scenario?

Ashley: In multi-teacher settings, OPRD effectively distilled four specialized 4-billion parameter teachers into a single 8-billion parameter student.

It reached teacher-level performance with 55% fewer updates than Mix-RL, and ultimately outperformed all four specialists.

Evan: That’s quite a feat.

How did OPRD fare in terms of efficiency?

Ashley: By using the same number of rollouts per update, the improved sample efficiency was a clear outcome.

OPRD achieved higher performance with fewer updates, which implies a significant reduction in computational resources and training time.

Evan: What about its performance with more conventional strong-to-weak distillation setups?

Ashley: Good question.

Interestingly, OPRD also demonstrated superiority in conventional strong-to-weak distillation.

For example, when distilling from a stronger Qwen3-8B teacher to a smaller Qwen3-1.7B student on AIME'24, OPRD improved the Mean@16 score significantly over OPD by 3.79 points.

Similarly, on Knights & Knaves, it outperformed by 29.20 points.

Evan: That’s quite a scope of applicability.

Any specific analysis on model behaviors or response styles?

Ashley: Yes, indeed.

The response-style analysis showed that OPRD-trained students remained stylistically closer to models trained purely with verifier-driven reinforcement learning as opposed to their weak teachers.

This indicates that OPRD helps the student not just learn faster but also develop its own optimized reasoning path.

Evan: Very thorough.

So, in summary, OPRD not only accelerates learning but also enhances model performance even beyond the teacher's capabilities?

Ashley: OPRD facilitates faster and more effective training, leading to robust model performance and efficiency—crucial factors as AI models continue to scale and diversify.

Evan: And that wraps up the Experiment section of the paper.

Stay tuned, we’ll be breaking down the analysis and broader implications next.

Evan: We've covered the experiments and results, Ashley.

How does this work fit into the landscape of related research?

Ashley: To understand where On-Policy Reverse Distillation, or OPRD, stands, it's crucial to look at the related work in weak-to-strong generalization, on-policy distillation, and reinforcement learning with distillation.

Evan: Let's start with weak-to-strong generalization.

What has previous research shown?

Ashley: Weak-to-strong generalization has been observed across multiple domains, including language understanding, reward modeling, and reasoning tasks.

Research shows that despite the name, weak supervision can often partially close the gap to strong supervision.

Evan: What are the key mechanisms behind these observed gains?

Ashley: Several factors contribute.

One key mechanism is correcting weak pseudo-labels and extending coverage beyond the weak teacher.

Another factor is leveraging differences between teacher and student hypothesis classes or representations to map out additional optimization pathways.

Evan: I see.

But conventional knowledge distillation has its limits.

How does on-policy distillation address these issues?

Ashley: On-Policy Distillation, or OPD, was developed to tackle the mismatch between training and inference distributions.

It queries the teacher along the states the student actually visits, providing dense token-level supervision for the student’s inference-time state distribution.

Evan: How has this approach evolved over time?

Ashley: Recent advancements have integrated OPD into post-training for reasoning tasks and capability consolidation across domains.

OPD is now a common method for refining large language models after initial training.

Evan: Okay, but teacher matching in OPD can still become a ceiling.

How do OPRD and related methods differentiate themselves?

Ashley: That’s where methods combining distillation with reinforcement learning come in.

These approaches integrate teacher guidance with outcome-based optimization, enhancing both coverage and efficiency.

However, earlier techniques often treated teacher matching as a separate objective, which could conflict with reward maximization.

Evan: So, how does OPRD specifically address these conflicts?

Ashley: OPRD distinguishes itself by scaling the student’s policy gradient using the teacher’s policy shift without making it an independent target.

This avoids the pitfalls of matching the weak policy directly or incorporating it as a competing objective.

Evan: And how does this compare to other recent works on transferring policy shifts?

Ashley: Several methods have explored transferring the delta between a post-trained policy and its reference.

While these methods often use the shift as a reward or alignment target, which can limit gains to the teacher's improvements, OPRD leverages the shift solely to rescale the student gradient, retaining first-order reward optimization.

Evan: So, OPRD’s approach is more nuanced in using the policy shift as a scaling tool?

Ashley: Exactly.

By doing this, OPRD preserves the stationary points of the student’s reward optimization objective and adds a non-negative alignment gain, which is unique among current methods.

Evan: What about the influence of multi-task optimization paradigms on OPRD?

Ashley: Multi-task optimization techniques often manipulate gradients directly to balance conflicts between tasks.

Inspired by these methods, OPRD amplifies only the gradient aligned with the teacher direction while retaining the rest, which resembles but also extends multi-task optimization strategies.

Evan: Quite a thorough integration of concepts.

How does this pave the way for future research?

Ashley: OPRD opens avenues for reusing post-training improvements across models or scaling applications without the constraints of direct policy matching.

It points towards efficient recursive self-improvement, where each model generation can potentially refine its successor using these distillation techniques.

Evan: That's the nutshell of the related work section.

Ashley: Yes, understanding these contributions contextualizes the innovation that OPRD brings to the table.

And that wraps up the Related Work section.

Evan: Let's wrap up today's discussion by summarizing the key contributions and takeaways from this insightful paper on On-Policy Reverse Distillation.

Ashley: Evan.

First and foremost, the authors tackle the challenge of weak-to-strong generalization by proposing On-Policy Reverse Distillation, or OPRD.

This method enables stronger models to learn from weaker ones without being constrained by the limitations of the weak teachers.

Evan: OPRD works by using the policy shift learned during post-training of the weak model to accelerate the learning of a stronger student.

The key here is that it amplifies the student's verifier-driven policy gradient along the teacher shift direction without making it an optimization target.

Ashley: Exactly.

This ensures that the student can surpass the teacher model's performance while maintaining the integrity of its own optimization path.

OPRD has been shown to be effective in both successor model transfer and multi-teacher distillation, showcasing faster learning and higher final performance compared to existing methods.

Evan: And these benefits weren't limited to weak-to-strong distillation.

OPRD also demonstrated effectiveness in conventional strong-to-weak scenarios, broadening its potential applications.

Ashley: Moreover, response-style analysis revealed that models trained with OPRD closely align with those trained purely with verifier-driven reinforcement learning, suggesting that OPRD accelerates the student's optimization rather than just providing shortcuts.

Evan: In essence, OPRD presents a robust method for efficient and scalable model training, making it a valuable addition to the field of AI and machine learning.

Ashley: That's right.

And with that, we've reached the end of today's episode.

Evan: Thank you for tuning into Daily Paper Cast.

We hope you found this discussion insightful.

Ashley: Be sure to join us again for our next episode, where we'll continue to bring you the latest and most exciting research papers in AI and machine learning.

Evan: Until next time, have a great day, and keep exploring!