Subscribe
Share
Share
Embed
🤗 Upvotes: 55 | cs.CV
Authors:
Jiyoung Kim, Paul Hyunbin Cho, Jisu Nam, Donghoon Lee, Hyunsung Go, Yeonkyeong Lee, Hansaem Kim, Seungryong Kim
Title:
GRACE: Generation-aware latent compression for efficient video generation
Arxiv:
http://arxiv.org/abs/2610.10524v1
Abstract:
Highly compressed video autoencoders offer an effective way to accelerate video diffusion models, as the Diffusion Transformer (DiT) operates on far fewer tokens. However, such autoencoders are challenging to train, since a higher compression ratio degrades reconstruction quality and recovering it requires more channels, which is known to slow the convergence of the DiT. The compressed latent also differs from the one the DiT was trained on, so the pretrained DiT must be either retrained from scratch or adapted at considerable cost. Compressing the autoencoder the DiT was trained with appears to preserve compatibility, yet optimizing it for reconstruction alone still shifts the latent away from the distribution the DiT has learned. To address this, we propose Generation-Aware Latent Compression for Efficient Video Generation (GRACE), a two-stage framework that compresses a pretrained video autoencoder while keeping it compatible with the pretrained DiT. Specifically, we keep a frozen base latent from the pretrained encoder and learn a residual latent for the information lost under stronger compression, while aligning the compressed latent with the pretrained latent in the feature space of the frozen DiT so that the autoencoder is optimized for generation. We then adapt the DiT with lightweight fine-tuning and asymmetric denoising, where the base is denoised ahead of the residual. GRACE reduces the token count of Wan2.1-I2V-14B by 8x and its latency by 11.1x at 480x832x81, while matching the generation quality of the pretrained pipeline before compression on VBench.
We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: dailypapercast.ai@gmail.com
Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com
Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236
Cover Image by Kawen Kuang https://kawen.art
Evan: Welcome to Daily Paper Cast.
Ashley: Today’s paper is from Hugging Face’s daily paper list of October 8, 2026, and it has received 55 upvotes.
Evan: The title of the paper is 'GRACE: Generation-Aware Latent Compression for Efficient Video Generation.'
Ashley: The first two authors are Jiyoung Kim and Paul Hyunbin Cho, both from KAIST AI, and the corresponding author is Seungryong Kim from the same organization.
Evan: Let's dive right into the introduction section of this intriguing paper.
Evan: Recent advances in video diffusion models have certainly caught our attention, right, Ashley?
The computational cost remains a prominent obstacle.
Ashley: Evan.
These advancements, highlighted by works from Wan et al., 2025, and Ma et al., 2025, show remarkable generation quality.
However, the issue of computational cost persists, especially during both training and inference.
Evan: Why is this computational cost such a bottleneck?
Ashley: The cost is mainly due to the number of tokens processed by the Diffusion Transformer, or DiT.
The attention mechanism within the DiT scales quadratically with the sequence length, which means a high number of tokens can significantly increase the computational load.
Evan: Ah, so the key here is the number of input tokens, which scales with both the spatial and temporal resolution.
What does the paper propose to address this?
Ashley: Exactly.
The paper proposes a two-stage framework called GRACE, which stands for Generation-Aware Latent Compression for Efficient Video Generation.
The idea is to compress the latent space while preserving generation quality by fine-tuning a pretrained video autoencoder.
Evan: Using fewer tokens while keeping the quality intact sounds challenging.
How does GRACE approach this?
Ashley: GRACE addresses two key challenges: first, efficiently adapting pretrained autoencoder and DiT models while fully leveraging their pretrained knowledge, and second, closing the gap between reconstruction and generation quality introduced by compression.
Evan: How do they ensure a high compression ratio without degrading quality?
Ashley: They take a unique approach by introducing a dual-latent representation.
In the first stage, they retain the base latent produced by the frozen pretrained encoder, capturing the core information, and learn a residual latent that captures the details lost under stronger compression.
Evan: So the base latent holds the main structure, while the residual latent recovers finer details.
What happens in the second stage?
Ashley: In the second stage, they adapt the Diffusion Transformer, or DiT, through lightweight fine-tuning.
They use something called 'asymmetric denoising,' where the base latent is denoised ahead of the residual latent.
This essentially helps to stabilize the generation process.
Evan: Does this mean the DiT has to retrain from scratch?
Ashley: Not entirely.
One of GRACE’s strengths is that it avoids the need for extensive retraining by keeping the pretrained DiT frozen during the initial phase.
Only specific components are fine-tuned, which significantly reduces the computational cost.
Evan: Alright, so to summarize, GRACE compresses the latent space with minimal token usage, retains compatibility with pretrained models, and achieves faster generation while maintaining quality.
What specific contributions have they outlined in the paper?
Ashley: They highlight four main contributions: introducing GRACE as a framework, applying dual-latent representation with asymmetric denoising to preserve pretrained knowledge, a generation-aware alignment objective to keep compatibility with the pretrained DiT, and demonstrating significant reductions in token count and latency while maintaining high generation quality.
Evan: That sounds comprehensive and meticulously crafted.
And that wraps up the introduction section.
Stay with us as we dive deeper into the methods and experiments in the following segments.
Evan: Welcome back to Daily Paper Cast.
Let's continue by delving into the Method section of the paper, 'GRACE: Generation-Aware Latent Compression for Efficient Video Generation.'
Ashley: Indeed, Evan.
GRACE is structured into two key stages: the compression of the video autoencoder and the adaptation of the Diffusion Transformer, or DiT.
Evan: Alright, let's start with Stage 1.
How does the autoencoder compression work?
Ashley: Stage 1 focuses on leveraging the pretrained video autoencoder by introducing a dual-latent framework.
Specifically, they retain the base latent from the frozen pretrained encoder and add a residual encoder.
Evan: Interesting.
How does this dual-latent framework operate?
Ashley: The base latent, which is encoded from a spatially and temporally downsampled input, preserves the core representation that the pretrained autoencoder has learned.
For example, it takes the reduced resolution video input and processes it with the pretrained weights.
Evan: And the residual encoder?
What exactly does it do?
Ashley: The residual encoder captures the lost information by encoding the full-resolution video, generating additional channels that complement the base latent.
This means it carries the fine details and intricate motion information, which the downsampled input might miss.
Evan: Ah, I see.
By combining these two, they achieve an efficient compression without losing important video details.
Ashley: Precisely.
And then for decoding, the autoencoder concatenates the base and residual latents along the channel axis and decodes them together.
The training of the residual channels starts from zero, ensuring it does not interfere with the pretrained reconstruction quality.
Evan: How is the compressed latent aligned with the pretrained Diffusion Transformer?
Ashley: Great question, Evan.
GRACE uses a generation-aware alignment.
The compressed latent is matched with the pretrained latent in the feature space of the frozen DiT.
This alignment loss measures the distance between the noised latents using cosine similarity within the intermediate feature spaces of the transformer blocks.
Evan: That sounds like a crucial aspect for maintaining quality.
Is this alignment applied throughout the autoencoder's training?
Ashley: This ensures that the compressed latent remains compatible with the pretrained DiT’s learned distribution, which is vital for high-quality generation.
Evan: Alright, let's move on to Stage 2.
How do they adapt the DiT to the compressed latent?
Ashley: Stage 2 involves fine-tuning the DiT to handle the compressed latent.
The DiT's input and output projections are extended to match the expanded channels.
Additionally, LoRA—Low-Rank Adaptation—is applied to the transformer blocks, which keeps most of the pretrained architecture while adapting the model to the new latent space.
Evan: I remember you mentioned 'asymmetric denoising.' How does that fit in here?
Ashley: Asymmetric denoising is a key innovation.
It denoises the base latent ahead of the residual latent during the training process.
This sequential approach allows the residual latent to refine information based on a stable and cleaned base latent.
Evan: That's pretty clever.
It avoids needing to start from scratch while still adapting efficiently.
Ashley: Exactly.
This method supports leveraging the pretrained model, thereby reducing the training costs significantly.
They sample a range of offsets and schedule times during training to cover various noise levels and ensure robust performance.
Evan: What datasets and metrics do they use for evaluation?
Ashley: GRACE is trained and evaluated using the Panda-70M dataset, which contains diverse video captions.
They evaluate the reconstruction quality using metrics like PSNR and LPIPS, and generation quality through VBench, measuring attributes such as subject consistency, motion smoothness, and aesthetic quality.
Evan: I see they use both objective metrics and human evaluations to ensure comprehensive assessment.
How do their results compare with other benchmarks?
Ashley: Their results show significant improvements.
GRACE reduces the token count of Wan2.1-14B by nearly eight times and its latency by 11.1 times at 480x832 resolution, while maintaining the generation quality comparable to the original model.
Evan: That's highly impressive.
Reducing tokens and latency without compromising on quality is a major achievement.
Ashley: Indeed.
This validates their approach to leveraging pretrained models and introducing efficient compression with minimal retraining costs.
Evan: And that concludes the Method section.
Next up, we’ll discuss the experiments and results reported in this paper.
Stay tuned!
Evan: Welcome back.
Now, let's dive into the experiments and results presented in the paper 'GRACE: Generation-Aware Latent Compression for Efficient Video Generation.'
Ashley: The authors conducted extensive experiments to evaluate the performance of GRACE.
They compared it against several state-of-the-art video autoencoders like Step-Video-VAE, Video DC-AE, and LTX-Video.
Evan: Let's start with the setup.
What datasets and benchmarks did they use for these experiments?
Ashley: The primary dataset used for training and evaluation is Panda-70M, which contains 70 million videos with captions.
They used VBench, a comprehensive benchmark suite, to evaluate both text-to-video and image-to-video generation performance.
Evan: What specific metrics were evaluated with VBench?
Ashley: VBench measures multiple dimensions of video quality, including subject consistency, motion smoothness, aesthetic quality, background consistency, temporal flickering, and dynamic degree.
Additionally, it assesses semantic aspects like object class, multiple objects, human action, color, spatial relationship, scene, and appearance style.
Evan: Sounds comprehensive.
How does GRACE compare with the benchmarks in terms of these metrics?
Ashley: GRACE achieves significant improvements in various aspects.
For instance, it scores highly in subject consistency, background consistency, and motion smoothness.
Overall, GRACE maintains the generation quality of the pretrained Wan2.1-14B pipeline while using nearly eight times fewer tokens.
Evan: Can you share some specific numbers to highlight these improvements?
Ashley: Evan.
At a resolution of 480 by 832, GRACE reduces the token count by nearly eight times and achieves an 11.1 times reduction in latency compared to Wan2.1-14B.
At a higher resolution of 736 by 1280, this reduction is even more pronounced, with a 15.5 times reduction in latency.
Evan: That's remarkable.
Reducing the computational load while retaining quality is no small feat.
How does GRACE handle the trade-off between reconstruction fidelity and generation quality?
Ashley: Interestingly, while GRACE does sacrifice some reconstruction fidelity for higher compression ratios, it more than compensates in generation quality.
For instance, even though GRACE's PSNR is 1.13 dB lower than their single-latent baseline, it surpasses other benchmarks in VBench scores for generation tasks.
Evan: Is there a qualitative aspect to these evaluations?
Like human judgment or visual comparison?
Ashley: Yes, the authors included human evaluations along with quantitative metrics.
They conducted a blind user study comparing GRACE with both Wan2.1-14B and DC-Gen.
Participants judged visual quality, temporal consistency, and text alignment of generated videos.
Evan: And what did the human evaluators think?
Ashley: Human evaluators preferred GRACE over DC-Gen by a wide margin in both text-to-video and image-to-video settings, rating GRACE better in terms of visual quality, temporal consistency, and alignment with the provided text descriptions.
Evan: It sounds like GRACE not only excels in technical benchmarks but also in practical evaluations.
Were there any notable qualitative results in the paper?
Ashley: Indeed, the paper includes visual samples comparing GRACE’s output with that of the pretrained Wan2.1-14B and DC-Gen.
GRACE’s videos preserve scene layout, subject integrity, and motion while using significantly fewer tokens.
They also achieved diverse visual styles like watercolor painting and cyberpunk in text-to-video samples.
Evan: Alright, to summarize: GRACE demonstrates substantial gains in efficiency and quality.
It reduces token count and latency significantly while maintaining high generation quality, confirmed by both objective benchmarks and human evaluations.
Ashley: Precisely.
This confirms the potential of GRACE in leveraging pretrained models with efficient compression strategies to advance video generation technologies.
Evan: And that wraps up the Experiments and Results section.
Next, we’ll explore the related work and conclusions drawn in this paper.
Stay tuned!
Evan: Welcome back to Daily Paper Cast.
Moving forward, let’s delve into the Related Work section of the paper 'GRACE: Generation-Aware Latent Compression for Efficient Video Generation.'
Ashley: This section is critical as it places GRACE within the broader context of existing research and highlights how it builds upon and diverges from prior work.
Evan: So, what key areas of related work do the authors touch on?
Ashley: They discuss two main areas: latent compression for efficient video generation and methods for adapting autoencoders to pretrained generators.
Evan: Let’s start with the latent compression for efficient video generation.
What’s the main challenge here?
Ashley: The primary challenge is balancing the trade-off between high compression ratios and maintaining reconstruction quality.
Video diffusion models need to process large numbers of latent tokens, making computation expensive during both training and inference.
Evan: Which previous works in this area do the authors reference?
Ashley: They reference several key works, such as HaCohen et al., 2024, Zheng et al., 2026, and Chen et al., 2025.
These prior works explore different strategies to enhance spatial compression or introduce new latent tokens that better capture video characteristics.
Evan: Are there any specific methods in these references that particularly stand out?
Ashley: Yes, for example, Tian et al., 2025, and Mahapatra et al., 2025, propose strategies to regularly increase temporal compression.
Another approach seen in Chen et al., 2025c, uses wider latents, which maintain reconstruction quality at the cost of slowing convergence.
Evan: And how does GRACE differentiate itself from these methods?
Ashley: GRACE distinguishes itself by introducing a dual-latent representation that leverages a pretrained autoencoder to compress the latent space effectively.
This bypasses the need to train from scratch and focuses on aligning the compressed latent with the pretrained latent in the feature space of a frozen Diffusion Transformer, or DiT.
Evan: Got it.
Now, what about adapting autoencoders to pretrained generators?
What do the authors say on that front?
Ashley: Adapting autoencoders to pretrained generators is another challenge, given the shift in latent space distribution when the autoencoder is optimized for reconstruction alone.
Previous works like those by Zhao et al., 2024, and He et al., 2026, attempt to align patch embeddings or constrain the latent to remain reconstructable by the pretrained decoder.
Evan: How does GRACE advance the solutions proposed in these prior works?
Ashley: GRACE advances these solutions by optimizing the autoencoder’s latent directly in the pretrained DiT’s feature space.
Rather than aligning patch embeddings post divergence, it maintains alignment within the DiT during the autoencoder’s training, ensuring compatibility and stability in generation quality.
Evan: Can you elaborate on how GRACE maintains this alignment during training?
Ashley: Certainly.
GRACE uses generation-aware alignment, which measures the distance between the compressed and pretrained latents in the feature space of the frozen DiT.
This is done using cosine similarity, which is averaged over the tokens from intermediate features in the transformer blocks, ensuring that the compressed latent stays compatible with the pretrained model's expectations.
Evan: That makes sense.
So, overall, how does GRACE leverage existing research while providing novel contributions?
Ashley: GRACE builds on existing research by incorporating the strengths of previous latent compression techniques and autoencoder adaptation strategies.
Its dual-latent approach, along with generation-aware alignment, presents a significant advancement over traditional methods that require extensive retraining or compromise on quality.
Evan: It's interesting to see how GRACE not only addresses existing challenges but also sets a new benchmark for efficiency and quality.
And that’s the end of the Related Work section.
Next, we’ll cover the conclusion and the future directions proposed by the authors.
Stay with us!
Evan: Welcome back to Daily Paper Cast.
Let’s dive into the final part of today's episode and discuss the key contributions and takeaways from the paper 'GRACE: Generation-Aware Latent Compression for Efficient Video Generation.'
Ashley: Certainly, Evan.
This paper introduced GRACE, a novel framework that significantly enhances the efficiency of video generation while maintaining high quality.
At its core, GRACE leverages a dual-latent representation to compress video autoencoders and adapt Diffusion Transformers.
Evan: One of its standout features is how it preserves compatibility with pretrained models, avoiding extensive retraining costs.
This is achieved through generation-aware alignment and asymmetric denoising.
Both innovations ensure the compressed latent retains compatibility with the pretrained Diffusion Transformer, or DiT.
Ashley: Moreover, GRACE stands out by its remarkable efficiency.
It reduces token count by nearly eight times and latency by over tenfold at standard resolution, showcasing the effectiveness of its approach.
At higher resolutions, the reductions are even more pronounced.
Evan: Indeed, that's impressive.
GRACE also showed significant improvements in generation quality across various metrics and tasks, validated through extensive benchmarks and human evaluations.
This underscores its practicality in real-world applications.
Ashley: The paper also highlighted the versatile nature of GRACE, capable of handling diverse video styles and maintaining subject and background consistency across different settings.
Evan: That brings us to the end of today's episode.
We hope you found the discussion as enlightening as we did.
GRACE’s contributions truly indicate a significant advancement in efficient video generation.
Ashley: Thanks for tuning in!
Don’t forget to stay updated with our daily episodes where we unpack the latest research papers in AI, NLP, CV, and more from Hugging Face’s daily paper list.
Evan: We look forward to having you with us next time.
Keep exploring the fascinating world of AI research with us!
Ashley: Until then, take care and keep learning.