Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout

🤗 Upvotes: 34 | cs.CV

Authors:
Zhuoran Zhao, Shengju Qian, Tongtong Liang, Xianghao Kong, Songchun Zhang, Junchao Huang, Guian Fang, Xin Wang, Pan Hui, Anyi Rao

Title:
Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout

Arxiv:
http://arxiv.org/abs/2609.09123v1

Abstract:
Autoregressive (AR) video diffusion models have shown great potential in real-time video generation. Recent methods distill pretrained bidirectional video diffusion models into causal AR students through Distribution Matching Distillation (DMD), but the generated videos often suffer from over-saturation and over-smoothing issues, resulting in limited visual quality and realism. The key contributing factor is the mode-seeking behavior of the reverse KL objective in DMD, which can cause the student distribution to collapse onto only a few modes of the teacher distribution. To address this, we propose Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the AR student self-rollout to mitigate mode collapse induced by reverse-KL mode seeking. The core idea is to inject cleaner signals into noisy rollout inputs via random masks along spatial and temporal axes during the self-rollout process of AR diffusion distillation. Such perturbations encourage the student rollouts to explore more regions of the teacher distribution, allowing DMD to provide learning signals beyond the modes already covered by the student. Moreover, the cleaner tokens act as denoising guidance for other noisier tokens, improving the intermediate rollout predictions and reducing error accumulation. Extensive experiments demonstrate that our method improves multiple AR video diffusion distillation methods with higher visual quality efficiently, without incorporating real video data or additional post-training stages.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast, where we discuss the latest in AI and machine learning research.

Ashley: Today's paper is from the Hugging Face daily paper list of September 9, 2026, and it has received 34 upvotes.

Evan: The title of the paper is 'Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout.'

Ashley: The first two authors are Zhuoran Zhao and Shengju Qian, with corresponding author Anyi Rao from the Hong Kong University of Science and Technology.

Evan: Let's dive into the Introduction section of the paper.

Ashley: Video diffusion models have advanced rapidly over recent years, driving significant improvements in long-duration, high-fidelity video generation.

However, these models often rely on bidirectional attention and multiple denoising timesteps, making them less suitable for real-time streaming generation scenarios.

Evan: So, are autoregressive diffusion models a promising solution for real-time video generation?

Ashley: Exactly.

Autoregressive, or AR, diffusion models have emerged as a promising paradigm for real-time video generation.

These models leverage causal attention mechanisms, enabling the generation of future video chunks sequentially.

This means that each new chunk is conditioned on previously generated chunks, supporting real-time inference and unbounded video generation without recomputing previously generated frames.

Evan: Can you elaborate on the training paradigms used in these models?

Ashley: Sure.

Some representative paradigms include Teacher Forcing, Diffusion Forcing, and Self Forcing.

Teacher Forcing trains models to predict the next frames using ground-truth frames for context, but this can lead to exposure bias.

Diffusion Forcing assigns each frame independently sampled noise, which helps but still fails to fully align training with inference.

Self Forcing addresses this by training the model with self-rollout, where the model generates the next frame based on previously generated frames rather than ground-truth context.

Evan: It sounds like Self Forcing helps bridge the train-test gap.

But what are the current limitations?

Ashley: You're right.

While Self Forcing bridges the train-test gap and includes a knowledge distillation process via Distribution Matching Distillation, or DMD, there are critical limitations.

Specifically, videos produced can often be over-saturated and over-smoothed, resulting in low visual quality and realism.

Evan: Why does this happen?

Ashley: This phenomenon is primarily due to the mode-seeking behavior of the reverse KL objective in DMD.

This objective tends to cover only the high-probability regions of the teacher’s distribution, leading to mode collapse.

This reduces diversity in the generated videos.

Additionally, intermediate rollout predictions lack explicit training signals at each step, causing error accumulation.

Evan: So, how does Mask Forcing aim to address these issues?

Ashley: To tackle these challenges, the authors propose Mask Forcing, a Dual-Noise Masking Rollout strategy.

This method introduces cleaner signals into noisy rollout inputs during the self-rollout process, using random masks applied along spatial and temporal axes.

These perturbations encourage the AR student rollouts to explore a broader range of the teacher distribution, providing learning signals beyond already covered modes.

Evan: Interesting.

How do these cleaner tokens help in the training process?

Ashley: Cleaner tokens act as denoising guidance for noisier tokens, improving intermediate rollout predictions and reducing error accumulation.

Essentially, they provide a more stable context for the model to learn from, generating higher visual quality and more realistic videos.

The method improves multiple AR video diffusion distillation approaches without incorporating real video data or requiring additional post-training stages.

Evan: It sounds like a promising approach to improving visual quality in real-time video generation.

That's the end of the Introduction section.

Let's move on to the specifics of their methodology.

Evan: Let's dive deeper into the methods proposed in this paper.

Ashley: Sure, the authors propose a novel strategy called Mask Forcing, specifically designed to address the issues seen in autoregressive video diffusion distillation.

Evan: What exactly does Mask Forcing entail?

Ashley: Mask Forcing involves a dual-noise masking rollout strategy within the self-rollout DMD training process.

It introduces cleaner signals into the noisy rollout inputs via randomly applied masks along both spatial and temporal dimensions.

Let's break down how this works.

Evan: Sounds good.

Start with the basics.

Ashley: An autoregressive video diffusion model represents a video as a sequence of chunks, with each chunk containing one or more latent frames.

The model generates these chunks sequentially, conditioned on previously generated chunks and a text prompt.

This AR model factorizes the text-conditioned joint distribution by using a causal attention mechanism.

Evan: And how does the noise come into play?

Ashley: During the generation process, each video chunk is generated via a denoising diffusion process.

A noisy sample can be represented as a mixture of the clean chunk and Gaussian noise, with the level of noise controlled by a timestep variable.

In essence, the model learns to predict the clean chunk from this noisy input.

Evan: So where does Mask Forcing fit into this process?

Ashley: During self-rollout, the student model generates a noisy video chunk and then iteratively denoises it over a fixed number of timesteps.

Mask Forcing introduces cleaner signals during this rollout by applying random masks.

These masks select portions of the noisy sample to be replaced with cleaner versions at a lower noise level.

Evan: How are these masks applied?

Ashley: The masks are applied both within individual chunks and across consecutive chunks in the sequence.

For each denoising step, a mask determines which parts of the chunk will have lower noise.

The positions selected by the mask are re-noised to a lower noise level, while the rest retain the original noise level.

This creates a dual-noise input for the model to process.

Evan: And the model then denoises this dual-noise input?

Ashley: Yes, exactly.

The model is conditioned on the original timestep while processing this mixed-noise input.

By doing so, the model learns to use the cleaner parts as context to denoise the noisier parts.

This process diversifies the student rollout trajectories and mitigates the mode-seeking behavior of the reverse KL objective, thereby reducing visual artifacts and improving overall video quality.

Evan: Interesting.

How do the authors evaluate the effectiveness of Mask Forcing?

Ashley: The authors conduct extensive experiments using multiple AR video distillation methods as baselines, including Self Forcing, LongLive, and Causal Forcing.

They use the Wan2.1-T2V-1.3B model as the base and Wan2.1-T2V-14B as the teacher.

The generated videos consist of 81 frames at a resolution of 832 by 480 pixels, trained on prompts from the VidProM dataset.

Evan: What metrics do they use to evaluate the results?

Ashley: Several metrics are employed.

Overall visual quality is assessed using HPSv3, motion quality using Dynamic Degree, and instruction-following capabilities using VisionReward.

They also evaluate the computational efficiency using training convergence speed measured by the Maximum Mean Discrepancy (MMD) between student and teacher distributions.

Evan: Do they present both qualitative and quantitative results?

Ashley: Yes, they do.

Qualitatively, the results show that Mask Forcing leads to videos with higher visual quality, richer details, and fewer artifacts compared to baselines.

Quantitatively, Mask Forcing consistently improves metrics across different settings.

For example, it enhances visual quality scores significantly and reduces errors in motion dynamics.

Evan: And what about the ablation studies?

Do they provide insights into the impact of different components of Mask Forcing?

Ashley: They conduct comprehensive ablation studies to investigate the effects of varying the mask ratio, timestep window size, and mask application schemes.

These experiments reveal that a moderate mask ratio and appropriately sized timestep window offer the best balance between visual quality and motion dynamics.

Evan: Did they test Mask Forcing in different video generation scenarios?

Ashley: Yes, they did.

Besides short-video generation, they evaluated Mask Forcing on long-video settings and interactive video world models with camera control.

Results show that Mask Forcing not only improves visual quality and realism but also maintains consistency over longer sequences.

Evan: It seems like Mask Forcing could be a valuable addition to AR video diffusion models.

That's all for the Method section.

Now, let's move on to the experiments and results.

Evan: Now, let's delve into the experiments and results section.

Ashley: The authors conduct extensive experiments to validate the effectiveness of Mask Forcing across multiple dimensions.

They utilize several AR video distillation methods as baselines, specifically Self Forcing, LongLive, and Causal Forcing, leveraging the Wan2.1-T2V-1.3B model as the base and Wan2.1-T2V-14B as the teacher.

Evan: How did they set up these experiments?

Ashley: The experiments generate videos with 81 frames each, at a resolution of 832 by 480 pixels, using prompts from the VidProM dataset.

They test the models' performance in both chunk-wise and frame-wise generation settings.

Evan: What metrics do they use to evaluate the performance?

Ashley: The primary metrics include HPSv3 for overall visual quality, VisionReward for vision reward scores, motion quality (MQ), instruction following (Instruct.), and Dynamic Degree (Dynamic.) for motion evaluation.

They also utilize Maximum Mean Discrepancy (MMD) to assess training convergence speed by measuring the alignment between student and teacher video distributions.

Evan: Interesting.

What were their major findings quantitatively?

Ashley: Quantitatively, Mask Forcing improves all baseline methods.

For example, in the chunk-wise setting, Self Forcing with Mask Forcing achieves higher HPSv3 scores, better VisionReward metrics, and improved motion quality.

Specifically, the visual quality score HPSv3 increased from 9.55 to 9.84, while the Dynamic Degree, indicative of better motion representation, soared from 70 to 82.

Evan: And how does it perform in the frame-wise setting?

Ashley: In the frame-wise setting, similar improvements are observed.

For instance, Self Forcing with Mask Forcing increased its HPSv3 score from 9.34 to 9.79, and the motion quality metrics improved across the board.

This demonstrates that Mask Forcing consistently enhances the visual quality and realism of generated videos.

Evan: It's great to hear that the improvements are consistent.

Did the paper include qualitative comparisons as well?

Ashley: Yes, the paper presents qualitative comparisons showing that Mask Forcing generates videos with richer high-frequency details and fewer visual artifacts compared to baselines.

For example, scenes generated with Mask Forcing are less prone to over-saturation and appear more realistic.

Evan: That sounds promising.

Did they explore long-video generation or any other specific settings?

Ashley: Indeed.

They evaluated Mask Forcing on single-prompt long-video generation settings as well.

Incorporating Mask Forcing into LongLive, for example, led to improved visual quality and more detailed textures, maintaining high fidelity even in longer videos.

Evan: How about the ablation studies?

What insights did they provide?

Ashley: The ablation studies explored the impact of the mask ratio, timestep window size, and mask application schemes.

These studies demonstrate that a moderate mask ratio of around 0.2 and a timestep window size of 250 balance visual quality and motion dynamics effectively.

They also found that varying the mask application in both spatial and temporal axes influences the outcomes, with certain schemes providing better results.

Evan: Any other interesting aspects covered?

Ashley: Yes, they also compared Mask Forcing to other state-of-the-art methods like Causal-rCM and DistillAlign.

Mask Forcing consistently outperformed these methods in terms of visual quality, motion dynamics, and semantic alignment.

Furthermore, human preference studies showed that videos generated with Mask Forcing were favored significantly over the baselines.

Evan: That's a robust evaluation.

Anything about convergence speed?

Ashley: Incorporating Mask Forcing accelerates convergence, as indicated by faster reductions in Maximum Mean Discrepancy (MMD) between the student and teacher models.

This means Mask Forcing not only improves the visual quality but also makes the training process more efficient.

Evan: It sounds like Mask Forcing could be a significant step forward for AR video diffusion models.

That concludes the Experiment section.

Next, let's take a look at the concluding remarks and future work.

Evan: Let's move on to the Related Work section.

Ashley: In this section, the authors discuss various approaches and prior work related to autoregressive video generation, video diffusion models, and masked modeling techniques.

They start with autoregressive video generation, which sequentially generates frames conditioned on past frames.

Evan: What are some of the main approaches in autoregressive video generation?

Ashley: One significant approach is Teacher Forcing, which denoises the current chunk conditioned on clean ground-truth context.

This method, however, suffers from a train-test gap and exposure bias since the model can only condition on its own predictions during inference.

Diffusion Forcing, on the other hand, assigns each frame independently sampled noise to approximate rollout distributions, but it still fails to align training with inference adequately.

Evan: And how does Self Forcing address these issues?

Ashley: Self Forcing bridges the train-test gap by employing autoregressive self-rollout.

It generates the next frame based on previously self-generated frames rather than using ground-truth context.

This process distills a bidirectional teacher into the causal student model through a DMD loss.

Additionally, Causal Forcing uses an autoregressive teacher for ODE initialization to reduce the architecture gap.

Evan: What about long video generation models?

Any notable approaches?

Ashley: Yes, LongLive extends causal autoregressive generation to long video sequences using short window attention, frame sinks, and streaming long-tuning.

Despite these advances, many distilled AR models still exhibit limitations in visual quality and realism.

To address these, some methods introduce additional training signals, real data, or post-training stages.

Evan: Can you mention some specific advancements in addressing visual quality and realism?

Ashley: For instance, DMD2 introduces a GAN loss and real training data to improve sample fidelity.

Although effective, this method suffers from training instability and requires additional real data curation.

Similarly, Astrolabe leverages reinforcement learning post-training on distilled AR models but is limited by the dependency on reward models and the sensitivity to hyperparameters.

Evan: It seems like balancing mode-seeking and mode-covering behaviors is a significant challenge.

How have researchers approached this?

Ashley: That's correct.

Methods like Mode Seeking meets Mean Seeking combine complementary objectives for long video generation.

They use separate heads for supervised flow matching and reverse-KL distribution matching.

Another example is DistillAlign, which optimizes a balance between consistency and distribution matching objectives.

Evan: Let's talk about video diffusion distillation and loss functions.

What are the key points here?

Ashley: Distribution Matching Distillation, or DMD, uses score-based reverse KL distribution matching to align the outputs of student and teacher models.

However, this objective can result in mode-seeking behavior, concentrating the student model distribution on a limited set of teacher modes.

To address this, research has introduced consistency trajectory objectives, which promote mode-covering behaviors but may result in lower sample quality.

Evan: And how does Mask Forcing fit into this landscape?

Ashley: Mask Forcing addresses the mode-seeking issues of reverse-KL objectives by introducing dual-noise masking rollout in self-rollout DMD training.

This strategy broadens the student rollout distribution and provides cleaner contexts for denoising noisy tokens, leading to improved intermediate rollout predictions and reduced error accumulation.

Evan: Got it.

Now, how about related work on masked modeling?

Ashley: Masked modeling has been influential in representation learning and generative models.

For instance, Masked Autoencoder, or MAE, masks a significant portion of an image's patches and trains the model to reconstruct them, leveraging visible context.

In generative modeling, MaskGIT uses a mask-then-predict objective with parallel iterative decoding, enhancing image synthesis.

Evan: Any other notable works mentioned?

Ashley: Yes, Self-Flow combines mix-timestep scheduling with masking for multi-modal representation alignment.

Inspired by these works, Mask Forcing integrates the concept of dual-noise masking into AR video distillation, diversifying self-rollout trajectories and improving training effectiveness.

Evan: Now I see how Mask Forcing builds on the foundations laid by prior work while offering novel contributions.

That’s the end of the Related Work section.

Let's move on to the conclusion and future directions next.

Evan: Let's wrap up by summarizing the key contributions and takeaways from this paper.

Ashley: This paper, titled 'Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout,' makes several significant contributions to the field of autoregressive video generation.

Evan: To start with, the authors propose Mask Forcing, a novel approach that introduces a dual-noise masking rollout strategy during the self-rollout DMD training process.

This method injects cleaner signals into noisy rollout inputs through randomly applied masks, effectively mitigating mode collapse and reducing visual artifacts like over-saturation and over-smoothing.

Ashley: Indeed.

The perturbations created by the cleaner signals encourage the student model to explore beyond the high-probability modes and refine its intermediate rollout predictions.

This helps in achieving higher visual quality and realism, as validated by extensive experiments on multiple AR video distillation methods.

Evan: In addition to qualitative improvements, Mask Forcing also delivers quantitative benefits.

It consistently enhances metrics such as HPSv3 for visual quality, VisionReward for motion quality, and reduces Maximum Mean Discrepancy, indicating faster convergence during training.

Ashley: The robustness of Mask Forcing is further validated through ablation studies, which demonstrate that a moderate mask ratio and well-sized timestep window provide the best balance between visual quality and motion dynamics.

Moreover, human preference studies show a significant favorability for videos generated with Mask Forcing over baseline methods.

Evan: Overall, Mask Forcing offers a simple yet effective strategy to improve AR video diffusion models without needing real video data or additional post-training stages.

Ashley: That sums up today’s discussion on this innovative approach to video generation.

We hope you found this episode insightful.

Evan: Thank you for tuning in to Daily Paper Cast.

Join us next time for more discussions on cutting-edge research in AI and machine learning.

Ashley: Don’t forget to subscribe and stay updated with the latest in AI research.

Until then, take care and see you in the next episode.