Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability

🤗 Upvotes: 77 | cs.CL, cs.AI

Authors:
Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang

Title:
Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability

Arxiv:
http://arxiv.org/abs/2610.08448v1

Abstract:
On-Policy Distillation (OPD) trains a student on its own generations using teacher feedback. With different tokenizers, comparing teacher and student predictions requires alignment at both sequence and vocabulary levels. In this paper, we examine whether expanding this alignment coverage improves learning. Across three heterogeneous teacher--student pairs on mathematical reasoning and code generation, strict 1:1 groups already cover most student-generated tokens despite substantial vocabulary mismatch. On responses sampled from the students before distillation, the shared vocabulary retains nearly all teacher and student probability mass at strictly aligned positions on average. Restricting reverse KL to a student-selected top-16 subset of the shared vocabulary at each strict position achieves accuracy comparable to full shared-vocabulary OPD, outperforming the evaluated cross-tokenizer baselines. Adding mean squared error supervision on span log-probabilities in mismatch groups gives complete supervision coverage, yet reduces accuracy. At checkpoints from training with only the strict loss, the span gradients show weak or negative directional agreement with the strict gradients and grow in magnitude relative to them. These diagnostics may help explain the accuracy drop from adding span supervision. Our findings motivate a shift from maximizing alignment coverage to prioritizing supervision reliability: compact supervision at strict positions can be more effective than broader coverage that introduces weakly aligned or conflicting training signals.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast!

Ashley: Today we're diving into a paper from the Hugging Face daily paper list dated October 7, 2026.

This paper has garnered 77 upvotes.

Evan: The title is 'Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability.'

Ashley: The first two authors are Bingxi Hou and Guochao Jiang, with Yuewei Zhang as the corresponding author.

They're affiliated with Alibaba Cloud Computing.

Evan: Alright, Ashley, let's dive into the Introduction section.

Can you give us a bit of background on On-Policy Distillation?

Ashley: Sure thing, Evan.

On-Policy Distillation, or OPD, involves training a language model—that's our student—on its own generated responses using feedback from a more advanced model, known as the teacher.

The key here is that the student is learning from its own outputs rather than from a fixed dataset, which allows it to continuously adapt based on the teacher's guidance.

Evan: So it's quite dynamic then, as opposed to traditional distillation methods we're used to.

And why do we need to rethink the way tokenizers are handled in this process?

Ashley: Exactly, the dynamic nature introduces complexities when you have models with different tokenizers.

Tokenizers are the systems that break down text into tokens.

Different tokenizers might split a sentence into tokens differently, resulting in mismatches that have to be harmonized carefully during distillation.

This paper aims to address these discrepancies.

Evan: Interesting.

What's the main question the authors are trying to answer here?

What objectives have they set?

Ashley: The authors are investigating whether expanding alignment coverage improves learning in the student model.

Their focus is on comparing teacher and student predictions across different tokenizers both at sequence and individual token levels.

More specifically, they want to see if increasing the amount of aligned tokens and spans—that's broader alignment coverage—leads to better learning outcomes or not.

Evan: And how did they set out to investigate this?

What was their approach?

Ashley: They looked at several pairs of teacher and student models on tasks like mathematical reasoning and code generation.

The interesting part is that despite substantial vocabulary mismatches between these models, they found that a strict 1:1 grouping already covers most student-generated tokens.

They measured the alignment and probability distribution at positions strictly aligned before any distillation took place.

Evan: That sounds quite promising, but what were their main discoveries?

Ashley: One of their key findings was that the shared vocabulary between teacher and student models retains nearly all of the predictive probability mass at strictly aligned positions.

In simple terms, even with minimal broader alignment, the majority of the useful information is already captured by these strictly aligned tokens.

Evan: It seems counterintuitive that expanding alignment would have minimal impact.

Did they propose any improvements or suggestions based on this discovery?

Ashley: Definitely.

They found that restricting the reverse Kullback-Leibler to a student-selected subset of the shared vocabulary at each strict position—focusing on the top 16 entries—achieves accuracy comparable to that of full shared-vocabulary OPD.

This approach not only streamlines the process but also outperforms evaluated cross-tokenizer baselines.

Evan: That's a significant insight.

Did they experience any setbacks or challenges during their research?

Ashley: Yes, when they tried to add mean squared error supervision on span log-probabilities within mismatch groups, it led to complete supervision coverage but actually reduced the accuracy.

Diagnostics showed weak or negative directional agreement with the strict gradients, which grew in magnitude relative to them and could explain the drop in accuracy.

Evan: It looks like they've made a strong case for prioritizing supervision reliability over broader alignment.

Can you summarize their main contributions to this field?

Ashley: Of course.

They've shifted the emphasis from maximizing alignment coverage to enhancing supervision reliability.

Their results suggest that compact and carefully selected supervision can be more effective than widespread coverage that introduces weakly aligned or conflicting signals.

This is a valuable shift in how we approach model distillation across different tokenizers.

Evan: That's the end of the Introduction section of the paper.

Evan: Alright, Ashley, let's dive into the methodology that the authors used in this study.

What approach did they take to compare and align different tokenizers?

Ashley: Great question, Evan.

To start, let’s talk about the preliminaries.

The authors define all their notations and key concepts that are necessary to grasp the methodology.

They introduce two main components: a student model, denoted as πθ, and a frozen teacher model, denoted as πT.

Each of these models has its own set of vocabulary and ways to tokenize input sequences.

Evan: So, essentially they are dealing with models that have different ways to break down and understand text, right?

How do they ensure these differences are accounted for?

Ashley: Exactly.

One of the crucial steps in their methodology is something called token-group alignment.

They tokenize the same student-generated response using both the student and the teacher's tokenizers.

By doing this, they can identify shared boundaries within the text—these are positions where the token boundaries align across both models.

Evan: Got it.

So, they align both models' tokenizations based on those points.

What happens with sequences that don't perfectly match?

Ashley: For those sequences, they create what they call 'mismatch groups.' In simple terms, a mismatch group involves multiple tokens from one model aligning with multiple tokens from the other model to cover the same span of text.

This helps them handle token sequences that don't align neatly.

Evan: That seems like a solid approach.

But how do they measure and utilize these alignments during training?

Ashley: They focus on something known as 'strict 1:1 groups' for their primary objective.

Strict 1:1 groups involve a single token from the student aligning perfectly with a single token from the teacher.

They then use a reverse Kullback-Leibler divergence—commonly known as reverse KL—to compare the predictive distributions of these tokens.

The reverse KL divergence between these aligned tokens serves as their strict Cross-Tokenizer objective.

Evan: Interesting.

So, their main focus is on these strict alignments.

Did they incorporate any other objectives?

Ashley: Yes, they did introduce an additional objective to cover those parts of the sequence that don’t fall into strict 1:1 groups.

They applied a mean squared error (MSE) loss on span log-probabilities within these mismatch groups.

This was aimed at increasing the supervision coverage.

Evan: But they mentioned earlier that this actually reduced accuracy?

How exactly does adding MSE supervision impact their results?

Ashley: You're right.

While this additional supervision provided complete coverage, it introduced weakly aligned or conflicting training signals.

Their diagnostics showed that the gradients from this span supervision often had weak or negative directional agreement with the gradients from the strict supervision.

This misalignment in gradients might explain why adding span supervision reduced the overall accuracy.

Evan: That makes sense.

So, how effective were these strictly aligned supervision methods compared to broader coverage in practice?

Ashley: In practice, they found that a significant portion of the useful predictive information was already retained by focusing on a strict subset of align positions.

They conducted experiments across various teacher-student pairs in tasks like mathematical reasoning and code generation to verify this.

Evan: Did they test any specific configurations to see how well strict supervision holds up?

Ashley: They did.

They specifically tested configurations where they restricted the reverse KL objective to a student-selected top-k subset of the shared vocabulary at each strict position.

Interestingly, they found that a top-16 subset preserved almost all the learning benefits obtained from full shared-vocabulary distillation.

This means that a compact supervision approach could retain most of the performance gains without needing to cover the entire vocabulary.

Evan: Wow, so a smaller, more focused set of alignments can be just as effective!

Did they achieve any notable improvements over existing baselines?

Ashley: Indeed.

The strict full and top-16 variants surpassed all evaluated cross-tokenizer baselines in both mathematical reasoning and code generation tasks.

This not only demonstrates higher effectiveness but also greater efficiency, as the smaller subsets mean less computational overhead.

Evan: That’s impressive.

But I’m curious, how did they measure the alignment and the predictive mass in the shared vocabulary?

I believe they mentioned something about probability-mass measurements?

Ashley: Good memory, Evan.

They measured the probability mass by evaluating the student and teacher predictions specifically on the shared vocabulary at strictly aligned positions.

They confirmed that nearly all of the predictive mass—over 99.5%—was captured within the shared vocabulary.

This was an important diagnostic to confirm that the alignment approach was effective.

Evan: And what about the importance of the shared vocabulary in training?

How crucial was it?

Ashley: The shared vocabulary was extremely crucial.

By focusing on entries with high predictive probability, they found that restricting the training to these focused subsets of the shared vocabulary could effectively streamline the distillation process without sacrificing performance.

This compact focus provided a competitive edge in ensuring a reliable supervision signal.

Evan: Thus, balancing alignment coverage with supervision reliability appears to be the crux of their proposed method.

They also discussed a bit about gradient diagnostics, right?

Ashley: Yes, they analyzed gradient diagnostics to assess the quality of the supervision signals.

They found that span-loss gradients grew in magnitude but were weakly or negatively aligned with strict supervision gradients.

This misalignment suggested that the added span supervision might not be improving the overall training regime but rather introducing noisier signals.

Evan: It’s clear now how the authors navigated the complexities of cross-tokenizer distillation, focusing more on supervision reliability than just broadening alignment.

Is there anything else in their methodology that stands out?

Ashley: Their approach stands out in how it makes a compelling case for streamlined, effective supervision.

By rigorously analyzing different supervision strategies and their impacts, they provide valuable insights on refining cross-tokenizer distillation.

Evan: That's comprehensive, Ashley.

We've covered quite a bit here.

And that’s the end of the Method section for the paper.

Evan: Alright Ashley, now let's move on to the Experiments and Results section.

How did the authors set up their experiments to verify their hypotheses?

Ashley: Sure, Evan.

The authors conducted a comprehensive set of experiments across three different model pairs on tasks such as mathematical reasoning and code generation.

The teacher-student pairs they used were Qwen2.5-7B-Instruct to Llama-3.2-3B-Instruct, Granite-4.1-8B to Phi-4-mini-instruct, and Granite-4.1-8B to Qwen2.5-7B-Base.

These model pairs represent a wide range of complexity and tokenization differences.

Evan: Interesting mix.

So, did they use any particular datasets for these tasks?

Ashley: For mathematical reasoning, they used datasets like MATH500, GSM8K, AIME-2024, AIME-2025, AIME-2026, AMC23, and Minerva-Math.

For code generation, they evaluated on HumanEval, MBPP, and LiveCodeBench.

They covered a lot of ground to ensure that their findings were robust across different domains.

Evan: That seems pretty thorough.

How did they measure the effectiveness of their approach?

Ashley: They primarily looked at the accuracy of the student models on these benchmarks.

Specifically, they recorded mean accuracies across multiple samples for each problem—32 samples for math problems and 8 samples for code problems.

Evan: And what were their main findings?

Ashley: Interestingly, they found that strict supervision—using the reverse KL divergence on strictly aligned tokens—was highly effective.

To give you a sense, they found that the strict 1:1 groups, which involve exact student-to-teacher token alignments, covered between 85.57% to 96.98% of the student tokens across different pairs, despite substantial vocabulary mismatches.

Evan: Wow, so even with substantial vocabulary mismatches, the strict alignment was quite comprehensive.

Ashley: Yes, exactly.

They also measured the shared vocabulary's probability mass retention.

The key takeaway here was that the shared vocabulary retained nearly all the predictive mass—over 99.5% for both the student and teacher models.

This suggests that focusing training on a shared subset of vocabulary can be highly effective.

Evan: And how did the different supervision settings affect the results?

Ashley: They found that strict supervision alone achieved the best accuracy scores compared to settings with additional span MSE loss.

For example, the strict full objective consistently outperformed the span-MSE settings across all teacher-student pairs.

Adding span supervision did not improve performance; in fact, it lowered it slightly across all evaluated configurations.

Evan: So, sticking with the strict supervision really proved to be the best approach.

What about the compact supervision subsets?

Did they hold up well against the full shared-vocabulary setup?

Ashley: Yes, they did.

They tested configurations that limited the reverse KL divergence to the student-selected top-16 subset of the shared vocabulary at each strict position.

These configurations retained at least 96% of the improvement achieved by full shared-vocabulary OPD.

For instance, in the Qwen2.5-7B-Instruct to Llama-3.2-3B-Instruct pair, the strict top-16 configuration scored 32.64% in the full-average accuracy, compared to 32.86% with the full shared vocabulary.

Evan: That's remarkable.

So, a focused set of high-probability tokens can almost match the comprehensive approach.

Did they benchmark against other methods in the field?

Ashley: They did.

They compared their strict supervision method with several baselines like ULD, Extended ULD, GOLD, and SimCT.

The strict full and the top-16 subset configurations consistently outperformed all these baselines in both math and code benchmarks.

This shows that their method isn't just efficient but also more effective.

Evan: And how did they evaluate the diagnostic metrics?

Ashley: They went a step further by evaluating gradient diagnostics.

They found that while span-loss gradients grew in magnitude, they were weakly or negatively aligned with strict gradients.

This misalignment in supervision signals confirmed why additional span supervision reduced accuracy.

Evan: So, the gradient diagnostics really helped them understand the impact of their supervision methods better.

Ashley: Exactly.

It revealed how span supervision introduced noisier signals into the training, which wasn't beneficial.

Their results strongly support the idea that better results come from reliable, compact supervision rather than broader but less reliable alignment.

Evan: Got it.

This section really highlights the practical gains they've achieved through their approach.

It's compelling how they balanced alignment coverage with supervision reliability.

That wraps up the Experiments and Results section.

Evan: Alright, Ashley.

We've talked about the experiments and results.

Now let's move on to related work.

How does this paper fit into the broader landscape of research in On-Policy Distillation and Cross-Tokenizer Distillation?

Ashley: Great question, Evan.

The authors situate their work primarily within two domains: On-Policy Distillation and Cross-Tokenizer Distillation.

Let's start with On-Policy Distillation.

This technique involves training the student model on its own rollouts, with the teacher providing supervision.

Evan: And how has On-Policy Distillation been explored previously?

Ashley: Previous research has leveraged various methods to make On-Policy Distillation more effective.

For instance, Song and Zheng in 2026 and MiniLLM by Gu and colleagues in 2024 use reverse KL divergence to ensure that the student doesn't assign high probabilities to regions the teacher finds unlikely.

Another method, GKD by Agarwal et al.

in 2024, supports different divergences and a mix of student-generated sequences with fixed data.

Yang et al.

in 2026 view OPD as dense KL-constrained reinforcement learning with token-level rewards.

Evan: So, it seems like there's a variety of approaches being taken.

What about the specific guidance signals in these methods?

Ashley: Exactly.

Recent work has delved into which teacher signals are most effective at the prefixes visited by the student.

For example, Fu et al.

in 2026 looked at unreliable guidance and proposed distribution matching on teacher-selected local top-k supports.

Jin et al.

in 2026 augmented reverse KL with forward KL in high-entropy regions, while Xu et al.

introduced something called TIP, or Token Importance in On-Policy Distillation, focusing on student entropy and teacher-student disagreement.

Evan: That's a diverse set of strategies.

How does this paper's approach compare to these existing methods?

Ashley: This paper builds on the idea that concentrating on high-probability predictions can be highly effective.

For example, Li et al.

in 2026 found that the intersection of teacher and student top-k predictions retains most probability mass and benefits from compact, focused supervision.

Our paper expands on this by addressing how to handle heterogeneity in tokenizers while still benefiting from high-probability supports.

Evan: Interesting.

Now, what about Cross-Tokenizer Distillation?

That seems like a more recent area of focus.

Ashley: Yes, Cross-Tokenizer Distillation has emerged to address the discrepancies in token boundaries and vocabularies between different models.

A variety of methods have been proposed.

For instance, ULD by Boizard et al.

in 2025 compares sorted output probabilities without needing matched token identities.

Other methods, like DSKD by Zhang et al.

in 2024 and CDM by Chen et al.

in 2025, create mappings between model representations or vocabularies.

Evan: So, models like DSKD and CDM are looking to bridge the gap between different token vocabularies?

Ashley: Exactly.

Then you also have approaches that focus on aggregating tokens into span representations before distillation, like SRA by Dao et al.

in 2026, which uses span representation alignment.

Singh and colleagues in 2026, on the other hand, utilize Byte-Level Distillation, converting teacher predictions to byte probabilities.

Evan: A lot of innovative techniques there.

How does this paper's proposal of strict supervision fit within these existing methods?

Ashley: This paper’s contribution is significant in that it refines the goal of aligning coverage to prioritize supervision reliability.

Unlike previous work that aims to broaden alignment, this paper shows that compact, high-probability supervision can be just as effective.

They even propose an effective technique for managing byte-prefix relations to map teacher probability mass onto student tokens, as explored by Wang et al.

in 2026.

Evan: Balancing between alignment coverage and supervision reliability certainly adds value.

What about techniques like SimCT and Byte-Prefix Marginalization?

Ashley: SimCT, proposed by Sun et al.

in 2026, and Byte-Prefix Marginalization by Wang et al.

also aim to recover lost supervision and minimize harmful supervision signals.

The paper we’re discussing contributes to this line of work by focusing on ensuring that strict alignment positions retain a high proportion of learning benefits, while minimizing the potential downsides of broad but less focused coverage.

Evan: So, they've effectively built upon and refined these recent methods.

Given the vast landscape of related work, what stands out in their approach for you?

Ashley: For me, it's their targeted approach to balancing supervision efficiency and effectiveness.

By conducting comprehensive diagnostic checks, they've been able to substantiate their claims with solid experimental evidence.

Their focus on maintaining high supervision reliability without expansive coverage is a pivot that could influence future work in this area.

Evan: It definitely sounds like they've made a tangible impact on Cross-Tokenizer Distillation techniques.

That wraps up the related work section.

Evan: Alright, Ashley.

We've discussed the introduction, methods, experiments, and related work.

Let's wrap up with a summary of the key contributions and takeaways from this paper.

What are the main points our listeners should remember?

Ashley: Sure thing, Evan.

To sum it up, this paper makes several important contributions to the field of Cross-Tokenizer On-Policy Distillation.

First, the authors shift the focus from maximizing alignment coverage to prioritizing supervision reliability.

They demonstrated that compact supervision, particularly at strict 1:1 token alignments, can be more effective than broader, less reliable coverage.

Evan: Right.

And they also discovered that the shared vocabulary between teacher and student retained nearly all of the predictive probability mass.

How did they use this insight?

Ashley: They used this insight to design a compact supervision approach.

Specifically, they showed that restricting the reverse Kullback-Leibler divergence to the student-selected top-16 subset of the shared vocabulary at each strict position achieved accuracy comparable to full shared-vocabulary OPD.

This technique not only streamlined the training process but also outperformed several cross-tokenizer baselines.

Evan: That's a key takeaway.

They also highlighted the challenges with adding span log-probability supervision, right?

Ashley: Exactly.

They found that while adding mean squared error supervision on spans provided full coverage, it actually reduced accuracy.

Diagnostics revealed that this additional supervision led to weak or negative gradient alignment with the strict supervision, explaining the dip in performance.

Evan: In summary, the paper advocates for focusing on the quality of supervision signals rather than the quantity of aligned tokens.

This pivot could guide future research in the area of model distillation.

Anything else to add, Ashley?

Ashley: That's about it, Evan.

Their rigorous methodology and comprehensive diagnostics offer valuable insights that could help improve the efficiency and effectiveness of cross-tokenizer model distillation in practice.

Evan: Well, that brings us to the end of today's episode.

Thank you for breaking down this paper, Ashley.

Ashley: My pleasure, Evan.

As always, it was great to dive into such fascinating research.

Evan: And thank you to our listeners for tuning in.

We hope you found today's discussion insightful.

Ashley: Remember to check back tomorrow for another episode of Daily Paper Cast, where we'll bring you more cutting-edge research from the world of AI and machine learning.

Evan: Until next time, keep exploring and stay curious!