Self-Supervised Visual On-Policy Distillation

🤗 Upvotes: 149 | cs.CV, cs.AI

Authors:
Yijiang Li, Yijun Liang, Yunjie Tian, Bingyang Wang, Ke Zhang, Zhenfei Yin, Di Fu, Philip Torr, Nuno Vasconcelos

Title:
Self-Supervised Visual On-Policy Distillation

Arxiv:
http://arxiv.org/abs/2608.14144v1

Abstract:
Visual on-policy distillation relies heavily on an informative teacher-student asymmetry, through either a larger, stronger teacher or privileged supervision, such as reference answers or ground-truth regions of interest. This raises a fundamental question: where can informative asymmetry come from when nothing privileged is available? We answer this by inverting where the asymmetry comes from. Rather than adding privileged information to the teacher, we subtract information from the student. This asymmetry creates the same effective learning signal for free as a teacher with access to information unavailable to the student, without ground-truth annotations, rewards, or a separate stronger teacher model. Building on this principle, we introduce Self-Supervised Visual On-Policy Distillation (S$^2$VOPD), a simple yet effective method that constructs on-policy learning signals from asymmetric augmented views. S$^2$VOPD distills the teacher's distribution conditioned on the original image on-policy into the student distribution conditioned on a strongly augmented view of the same image. We systematically explore a broad design space of visual augmentations and uncover that (1) asymmetry matters: all four augmentation families improve performance, while symmetric self-distillation degrades it; (2) strength matters: performance peaks at a moderate strength; and (3) the gap must remain task-consistent: augmentations that completely remove the question-relevant evidence can induce large but uninformative discrepancies. Across six fine-grained perception benchmarks, S$^2$VOPD improves Qwen3.5-4B from 70.7% to 77.4%, above all open-source models compared, up to Qwen3-VL at 235B, and surpasses GPT-5.4. While holding training data the same, it recovers 96% of the improvement achieved by methods with privileged information. Website is at https://williamium3000.github.io/s2vopd

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast!

Evan: Today's paper is from the Hugging Face daily paper list of August eighteenth, 2026, and it has one hundred and forty-nine upvotes.

Ashley: The title of the paper is 'Self-Supervised Visual On-Policy Distillation.'

Evan: It's authored by Yijiang Li and Yijun Liang, with Yijiang Li listed as the corresponding author from UC San Diego.

Ashley: Let's dive into the introduction of this paper.

Evan: Visual on-policy distillation traditionally relies on a strong teacher-student asymmetry.

This is usually achieved by using either a larger and stronger teacher model or by providing privileged supervision such as reference answers or ground-truth regions of interest.

Ashley: That's right.

This kind of privileged information often acts as a significant advantage for the teacher.

But the fundamental question this paper raises is: where can we derive this informative asymmetry when such privileged information isn't available?

Evan: So, what solution do they propose for this problem?

Ashley: The authors invert the direction of the asymmetry.

Instead of adding privileged information to the teacher, they propose subtracting information from the student.

This means the teacher works with the original inputs, while the student works with a degraded or augmented version of the input.

Evan: Interesting.

This setup creates a natural predictive discrepancy between the teacher and student that can serve as the learning signal, right?

Ashley: Exactly.

This method doesn't use ground-truth annotations, rewards, or a stronger separate teacher model.

It's akin to self-supervised learning, where supervision is generated from multiple views of the same image.

Evan: How does this new method, which they're calling Self-Supervised Visual On-Policy Distillation, or S2VOPD, specifically work?

Ashley: The method constructs on-policy learning signals from asymmetric augmented views.

Essentially, the teacher observes the original image, while the student observes a strongly augmented or degraded view of the same image.

The discrepancy between the teacher and student perspectives provides the learning signal needed to improve the student's performance.

Evan: They explore various types of visual augmentations to see what works best.

What were their key findings about these augmentations?

Ashley: They identified three main points.

First, asymmetry itself matters.

All four augmentation families they tested—geometric, photometric, information reduction, and occlusion—improved performance.

But symmetric self-distillation, where both teacher and student see the same input, actually degraded performance.

Evan: So, the idea is that the strength of the augmentation should be moderate rather than extreme?

Ashley: Yes, exactly.

They found that performance peaks when the augmentation strength is moderate.

If the augmentation is too strong, it removes too much relevant information, which makes the learning signal less useful.

Evan: And the augmentations need to remain task-consistent, right?

What does that mean in this context?

Ashley: That means the augmentation shouldn't completely remove evidence that's relevant for the task.

For instance, heavy cropping can sometimes strip away key parts of the image needed to answer the question, thus making the discrepancies large but uninformative.

Evan: Got it.

What kind of improvements did they observe with their approach?

Ashley: Across six fine-grained perception benchmarks, S2VOPD significantly improved the performance of the model they worked with, Qwen3.5-4B, boosting average accuracy from 70.7% to 77.4%.

Evan: That’s a substantial improvement.

And how does this compare to models using privileged information?

Ashley: Interestingly, the improvements achieved by S2VOPD recovered 96% of the gains obtained by methods using privileged information, but without requiring any such privileged data.

Evan: Very efficient.

So, they get nearly all the benefit without the associated costs.

Ashley: Exactly.

It's a compelling approach to enhance model performance in scenarios where privileged data is scarce or unavailable.

Evan: And that wraps up our overview of the Introduction section of this paper.

Evan: Alright, let's move on to discuss the method proposed in the paper.

They call this method Self-Supervised Visual On-Policy Distillation, or S2VOPD.

How exactly does S2VOPD work?

Ashley: S2VOPD constructs the teacher-student asymmetry directly from their visual inputs.

The teacher observes the clean, original input, while the student works with a transformed or degraded version of the same input, which we refer to as an augmented view.

Evan: And what kind of transformations or augmentations are applied to the student's view?

Ashley: The paper outlines a broad range of visual augmentations they explored.

These are organized into four main families: information reduction, geometric transformations, photometric transformations, and occlusions.

Evan: Could you give us a brief overview of what each family includes?

Ashley: Sure.

Information reduction includes techniques like downscaling, Gaussian blur, pixelation, and adding Gaussian noise.

These methods reduce the overall information content while preserving the spatial layout of the image.

Evan: So the student might see a lower-resolution or noisier version of the image?

Ashley: Exactly.

Then we have geometric transformations, which include rotations, translations, cropping, and zooming out with padding.

These transformations can modify the spatial organization and object locations within the image frame.

Evan: Interesting.

And what about photometric transformations?

Ashley: Photometric transformations alter the appearance while keeping the geometry intact.

Examples include adjusting brightness, contrast, saturation, hue, and adding effects like gamma correction and histogram equalization.

Evan: And finally, occlusions.

What do those involve?

Ashley: Occlusions involve removing parts of the image.

Methods here include random erasing, grid masking, and filled crops.

These are designed to create scenarios where parts of the image are obscured, making it challenging for the student.

Evan: How do they ensure that this augmentation creates an effective learning signal for the student?

Ashley: They found that the augmentation must balance three key properties.

First, the asymmetry itself matters—having a noticeable difference between what the teacher and student see is crucial.

Second, the strength of the augmentation—too weak or too strong included structures—affects performance.

Third, the augmentation must remain task-consistent, meaning it shouldn't overly disrupt the evidence needed to solve the task.

Evan: What design choices did they make in terms of how to apply these augmentations?

Ashley: They describe a design space where each augmentation is defined by three components: the type of operation, the probability of applying it, and the strength of the operation.

They used a stochastic process to apply these augmentations during training, ensuring diverse and effective samples for learning.

Evan: So how exactly does the distillation process work in this framework?

Ashley: Here's where it gets interesting.

The teacher model uses an exponential moving average, or EMA, of the student model's parameters.

This EMA teacher observes the original, unaltered image and generates predictions based on that.

Meanwhile, the student generates on-policy trajectories—basically predictions—based on the augmented view.

Evan: And the learning objective here is based on the difference between the teacher's and student's predictions?

Ashley: Yes, the goal is to minimize the divergence between the teacher’s and student’s distributions over the next predicted token in the sequence.

They used a specific divergence measure called the Jensen-Shannon divergence, which balances between forward and reverse Kullback-Leibler divergences.

Evan: Why did they choose the Jensen-Shannon divergence?

Ashley: Jensen-Shannon divergence helps balance the trade-off between covering all possibilities and focusing on the most probable segments.

It remains bounded when the two distributions have limited overlap, making it robust for this scenario.

Evan: Interesting.

And how do they control the application of these augmentations during training?

Ashley: They use a structured approach.

For every image-question pair sampled from their dataset, they apply a global probability to determine if an augmentation will be applied.

If it will, they then sample specific operators and their strengths from predefined distributions and apply them in sequence to the student’s view.

Evan: And what was the most effective augmentation strategy they found?

Ashley: After extensive experimentation, they found that downscaling the image followed by adding Gaussian noise provided the best results.

Every student view was augmented in this specific manner, ensuring consistent and strong asymmetry between the teacher and student views.

Evan: So even simple augmentations can be very effective given the right context.

Ashley: The right combination of augmentations creates the necessary information gap for effective learning.

Evan: And how did they evaluate the effectiveness of S2VOPD?

Ashley: They conducted evaluations across six fine-grained perception benchmarks.

These include V*Bench, ZoomBench, HR-Bench 4K and 8K, MME-RealWorld, and its Chinese subset MME-RealWorld-CN.

They also tested on three math reasoning benchmarks: MathVista, MathVerse, and MathVision.

Evan: What were the primary metrics they used for evaluation?

Ashley: They mainly focused on accuracy for the perception benchmarks and near-exact matching for the math reasoning tasks.

In some cases, particularly for unresolved answers, they even had an LLM judge to adjudicate.

Evan: And in terms of baseline comparisons, who did they compare against?

Ashley: They compared S2VOPD with several baseline models, including their base model, symmetric self-distillation without augmentation, methods using privileged information, and self-rewarding reinforcement learning methods like TTRL, Intuitor, and RENT.

Evan: And how did S2VOPD stack up against these methods?

Ashley: In their results, when trained on a 12K subset of the FineVision dataset, S2VOPD improved the accuracy of their 4 billion parameter Qwen model from 70.68% to 77.44% on average across six perception benchmarks.

They also observed substantial improvements in math reasoning tasks.

Evan: That’s impressive.

Did their method also outperform models that had access to privileged information?

Ashley: Yes, it did.

S2VOPD performed better than several models that used privileged supervision, matching or even surpassing their performance without using such data.

Evan: It seems like a very robust method with broad applicability.

Ashley: Indeed.

This concludes our discussion of the Method section.

Evan: Now let's dive into the experiments and results section of the paper.

How did they set up their experiments?

Ashley: They utilized the Qwen3.5-4B and Qwen3.5-9B models as their base models for all experiments.

For inference, they used the vLLM inference engine to handle the rollouts.

Evan: And what were the training configurations?

Ashley: They trained with a batch size of ninety-six prompts, generating eight rollouts per prompt.

They set the learning rate at two times ten to the minus six, with ten warmup steps and a total of sixty-five or one hundred thirty optimizer steps, equivalent to one epoch over their six thousand or twelve thousand training samples.

Evan: That's a very specific setup.

How about the data they used for training?

Ashley: For training, they sampled twelve thousand questions from the natural image domain of FineVision.

They trained their models for one hundred thirty optimizer steps.

Evan: And how were the models evaluated?

Ashley: For evaluations, they assessed their models across six perception benchmarks and three mathematical reasoning benchmarks.

For the perception benchmarks, they used greedy decoding with a maximum of four thousand ninety-six tokens.

For the math benchmarks, they used a twenty-four thousand five hundred seventy-six token budget, with a temperature of 0.3, top-p of 0.95, top-k of twenty, and a presence penalty of 1.5.

Evan: And what exactly were these benchmarks?

Ashley: The perception benchmarks were V*Bench, ZoomBench, HR-Bench 4K and 8K, MME-RealWorld, and its Chinese subset MME-RealWorld-CN.

The mathematical reasoning benchmarks were MathVista, MathVerse, and MathVision.

Evan: So how did S2VOPD perform on these benchmarks?

Ashley: S2VOPD significantly improved their Qwen3.5-4B model's performance.

The average accuracy across the six perception benchmarks increased from 70.68% to 77.44%, a gain of 6.76%.

Evan: That's a quite substantial improvement.

And how does this result compare with other models and methods?

Ashley: Interestingly, this improvement positioned their 4 billion parameter model ahead of many larger models, including Qwen3-VL-Instruct-235B, which had an average accuracy of 75.75%.

S2VOPD also matched the performance of Qwen3.5-397B, a much larger model.

Evan: That's impressive.

Were there any other noteworthy comparisons?

Ashley: Yes.

When comparing with proprietary models, S2VOPD outperformed the GPT-5 series and was on par with Gemini-3-Flash.

It trailed behind only Gemini-3.5-Flash and Gemini-3.1-Pro by less than 2%.

Evan: And how about in comparison to methods using privileged information?

Ashley: S2VOPD also outperformed methods utilizing privileged information, such as Vision-OPD, ZwZ, and OPSD, despite not relying on ground-truth regions or answers during training.

Evan: What about the consistency and stability of these results?

Ashley: The results were consistent across different model scales.

For example, when comparing performance under the same training conditions using the Vision-OPD-6K dataset, S2VOPD achieved better overall averages than other methods without privileged information, improving over the strongest self-rewarding baseline by 2.0% at 4B and 0.7% at 9B.

Evan: So, the method not only works but also scales well.

Ashley: Exactly.

They also noted that methods with privileged information show improved performance on perception tasks but don't fare as well on math reasoning.

On the other hand, self-rewarding RL methods generally improved math reasoning but showed limited gains on perception tasks.

Evan: So S2VOPD strikes a good balance between both perception and reasoning tasks.

Ashley: Exactly.

S2VOPD derived a perception-aligned learning signal without relying on privileged annotations, illustrating its robustness and versatility.

Evan: This ends our discussion of the Experiment and Results section.

Evan: Next, let's move on to the Related Work section of the paper.

Ashley, what does this paper compare itself to?

Ashley: The paper situates itself within several domains of existing research: self-supervised learning, semi-supervised learning, vision-language models' augmentation, and knowledge distillation, especially on-policy distillation.

Let's go through each of these areas.

Evan: Alright, starting with self-supervised and semi-supervised learning.

Ashley: The core idea here revolves around deriving supervision from augmented views of unlabeled images.

In self-supervised learning, methods like SimCLR and MoCo align different augmented views of the same image.

Similarly, methods like BYOL and SimSiam match a student model to a 'momentum' or 'stop-gradient' teacher under different augmentations.

Evan: How does S2VOPD's approach differ from or relate to these methods?

Ashley: While these methods use augmentation to define invariances, allowing models to learn without external labels, S2VOPD uses augmentation to invert where the asymmetry comes from, creating an informative asymmetry without privileged information.

Essentially, this shifts the application of these principles from representation-level learning to token-level generative distillation.

Evan: And what about semi-supervised consistency learning?

How is it relevant?

Ashley: In semi-supervised consistency learning, weakly augmented inputs supervise strongly augmented ones, as seen in methods like FixMatch and Noisy Student.

S2VOPD adopts a similar principle but applies it in token-level generative distillation by having the teacher observe the clean image and the student an augmented one.

Evan: Moving on to visual augmentation in vision-language models.

How is augmentation typically used in this subfield?

Ashley: Augmentation isn't as commonly used in vision-language models compared to representation learning.

This is because perturbations can alter instruction-relevant content.

However, some reinforcement learning approaches for visual reasoning have used perturbations.

NoisyRollout, for example, uses perturbed images to diversify exploration, while methods like VPPO and PRPO compare policy distributions across clean and perturbed views to identify critical perception tokens.

Evan: So, how does S2VOPD utilize augmentation differently?

Ashley: In contrast, S2VOPD applies augmentation only to the student view, creating a discrepancy from a teacher observing a clean image.

This discrepancy itself becomes the training signal, eliminating the need for rewards or annotations.

Evan: Let's touch upon knowledge distillation and on-policy distillation next.

Ashley: Knowledge distillation traditionally involves training a student on the knowledge transferred from a larger teacher model.

On-policy distillation, on the other hand, trains the student on its own trajectories with dense token-level supervision from a teacher.

Previous work has explored various optimization strategies, improved divergence objectives, and selective supervision to better preserve visual grounding.

Evan: And how does on-policy self-distillation fit into this framework?

Ashley: On-policy self-distillation removes the need for a larger teacher by sharing a single model between the student and teacher.

Existing methods, however, require the teacher to access privileged information, such as verified reasoning traces or additional context.

S2VOPD, by contrast, inverts this requirement by subtracting information from the student rather than adding it to the teacher.

Evan: So, the crux of S2VOPD's novelty lies in how it redefines the informational advantage required for distillation.

Ashley: Exactly.

This approach yields a similar effective learning signal to that obtained with privileged information, but without relying on external labels or annotations.

The paper demonstrates that an effective student-teacher discrepancy can be built purely through augmentation.

Evan: That’s quite an interesting approach, merging principles from different domains effectively.

Ashley: Indeed, it’s a smart consolidation of effective practices from various fields, applied in a novel way to on-policy distillation.

Evan: This brings us to the end of the Related Work section.

Evan: Let's wrap up today's discussion by summarizing the key contributions and takeaways of the paper titled 'Self-Supervised Visual On-Policy Distillation.'

Ashley: This paper introduces S2VOPD, a method that inverts the traditional teacher-student asymmetry by subtracting information from the student rather than adding it to the teacher.

This creates an effective learning signal without relying on privileged information or external annotations.

Evan: They explored a wide design space of visual augmentations, finding that the augmentation's asymmetry, strength, and task consistency are crucial for effective learning.

Downscaling with Gaussian noise was the most effective augmentation.

Ashley: S2VOPD was tested across six fine-grained perception benchmarks and three math reasoning benchmarks, showing substantial improvements.

It enhanced the Qwen3.5-4B model's performance from 70.68% to 77.44% in average accuracy, outperforming many larger models and even those using privileged supervision.

Evan: The method was shown to be robust, scalable, and versatile, striking a balance between improving both perception and reasoning tasks without the need for privileged data.

Ashley: Indeed.

S2VOPD demonstrates that valuable learning signals can be created through careful augmentation, providing a cost-effective alternative to methods requiring extensive annotations or privileged data.

Evan: That sums up our deep dive into 'Self-Supervised Visual On-Policy Distillation.' We hope you found this discussion as insightful as we did.

Ashley: Remember to tune in for our next episode, where we break down more cutting-edge research from the world of AI and machine learning.

Evan: Thanks for joining us today on Daily Paper Cast.

See you next time!

Ashley: Goodbye, everyone!