LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes

🤗 Upvotes: 75 | cs.CV, cs.AI

Authors:
Chuyan Chen, Haoxing Chen, Kun Chen, Zhenglin Cheng, Long Cui, Ruishan Fang, Zhangxuan Gu, Zhicheng Huang, Zhenzhong Lan, Yuanting Lei, Haoquan Li, Jianguo Li, Rongchuan Li, Sidu Li, Tao Lin, Deyuan Liu, Jiacheng Liu, Lin Liu, Yuxuan Lou, Zhisheng Lu, Yuxin Ma, Shuheng Shen, Peng Sun, Chaoyang Wang, Hongjun Wang, Xiaomei Wang, Yongxin Wang, Chengzhang Wu, Hongru Wu, Jun Xie

Title:
LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes

Arxiv:
http://arxiv.org/abs/2609.03796v1

Abstract:
We introduce LLaDA-Image, a unified framework that pairs a 6B Diffusion Transformer (DiT) trained from scratch with a frozen vision-language understanding module built on the LLaDA2.0-Mini diffusion language model backbone. Instead of relying heavily on paired image-text data from the beginning, we first build a strong visual generative prior through image-only pre-training and mid-training. The generation pipeline comprises 220M samples, 98 of which are real images. For efficient and scalable optimization, we use parameter-free RMSNorm throughout the DiT together with the Muon optimizer. The resulting unified model produces highly photorealistic images while accurately following fine-grained editing instructions. We further distill LLaDA-Image into LLaDA-Image-Turbo, enabling fast inference in 2-4 sampling steps. On Qwen-Image-Bench, LLaDA-Image achieves overall scores of 53.53 and 53.38 on the English and Chinese tracks, respectively, setting a new state-of-the-art among open-source models on both tracks. To support further research on capable and efficient generative models, we release our model weights, training code, and detailed recipes.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to the Daily Paper Cast.

Ashley: Today, we're bringing you a fascinating paper from the Hugging Face daily paper list of September 4, 2026, which has received 75 upvotes.

Evan: The paper we're discussing is titled 'LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes.'

Ashley: The first two authors are Chuyan Chen and Haoxing Chen, with Jianguo Li as the corresponding author.

They are affiliated with the AGI Research Center, Inclusion AI.

Evan: Alright, Ashley, let's dive into the Introduction.

What's the background and objective of this study?

Ashley: LLaDA-Image addresses a crucial question in the world of image generation: it's all about what it takes to train a strong image generator from scratch using a fully open recipe.

The paper acknowledges the evolution of image generation systems, which are transforming from specialized text-to-image models to more general-purpose visual creation systems.

Evan: Interesting.

So, how do these evolving capabilities impact the system requirements, especially in terms of data and model design?

Ashley: Exactly, Evan.

The study notes that modern systems need to understand complex and multilingual instructions, synthesize photorealistic content, preserve reference information during editing, and render text accurately.

They also have to be practical in terms of inference budgets.

Evan: Got it.

So, these systems not only need high-quality generation but also efficient and scalable deployment.

Ashley: Precisely.

The authors highlight that proprietary systems have progressed impressively in these areas.

However, their data, models, and training recipes are often inaccessible.

For open models, the challenge is to combine broad capabilities with an attainable data budget, stable large-scale training, and efficient deployment.

Evan: That sounds like a daunting task.

What specific challenges and bottlenecks do they identify?

Ashley: The linked bottlenecks emerge across data collection, model design, and deployment.

Existing training paradigms often couple visual-prior learning with language alignment from the beginning, relying heavily on paired image-text data.

However, captions can be expensive and lossy.

Evan: Lossy in what sense?

Ashley: When training at low resolutions, the details mentioned in a caption may disappear after downsampling the image.

Synthetic image-text pairs can help early convergence but might introduce artifacts that limit long-term realism.

Evan: So, the quality of the paired data is a key limiting factor.

How does LLaDA-Image propose to overcome this?

Ashley: LLaDA-Image separates visual prior learning from language alignment by starting with image-only pre-training and mid-training.

They use image-only data in more than 90 percent of their 220 million training samples, with 98 percent being real images.

Evan: That's quite a significant deviation from relying on paired caption data.

What are the other main contributions?

Ashley: The study introduces a unified framework with a 6 billion parameter Diffusion Transformer trained from scratch.

The framework includes a frozen vision-language understanding module combined with the LLaDA2.0-Mini diffusion language model backbone.

For optimization, they employ parameter-free RMSNorm and the Muon optimizer.

Evan: That sounds promising.

What about the actual generation capabilities?

Ashley: LLaDA-Image achieves highly photorealistic image generation while following fine-grained editing instructions.

They also introduce LLaDA-Image Turbo, which enables fast inference in just 2 to 4 sampling steps.

Evan: Fast inference in 2 to 4 steps?

That's impressive.

How did it perform on benchmarks?

Ashley: On the Qwen-Image-Bench, LLaDA-Image achieved overall scores of 53.53 on the English track and 53.38 on the Chinese track, setting a new state-of-the-art among open-source models on both tracks.

Evan: Those are substantial results.

So, the overall goal was to develop a capable visual creation system with an efficiently reusable open recipe for further research.

Ashley: Exactly.

They've released their model weights, training code, and detailed recipes to support future advancements in generative models.

Evan: Great.

That's the end of the Introduction section for LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes.

Evan: Let's dive into the core methods behind LLaDA-Image.

Ashley, can you break down the approach they took with this model?

Ashley: Evan.

To build LLaDA-Image, the authors designed a comprehensive training pipeline that integrates several key stages, carefully sequenced to maximize the model's performance and flexibility.

Evan: Sounds intricate.

What's the foundational architecture of LLaDA-Image?

Ashley: The architecture consists of three principal components: a vision-language model, an understanding-to-generation connector, and a single-stream Diffusion Transformer, or DiT, for image synthesis.

The vision-language model is based on the LLaDA 2.0 Mini backbone with the SigLIP-VQ vision encoder.

Evan: And how do these components interact with each other?

Ashley: Great question.

The vision-language model processes text prompts or editing instructions to guide the generation.

Its outputs are then projected into the generator’s conditioning space by the understanding-to-generation connector.

This connector comprises a Residual Query Adapter and a Transformer Connector that translate these multimodal representations into a format suitable for the Diffusion Transformer.

Evan: So, am I right in understanding that this connector is like a bridge between comprehension and generation?

Ashley: Exactly.

The connector aligns the high-level semantic understanding from the vision-language model with the specific conditioning signals needed for visual generation, effectively acting as a bridge.

Evan: Alright, let’s go step-by-step through the training pipeline.

What does it start with?

Ashley: The training begins with image-only pre-training and mid-training stages.

In these stages, the visual generator learns a robust generative prior without any paired image-text data.

This helps decouple the visual learning process from language alignment initially.

Evan: Image-only pre-training and mid-training?

Can you explain these stages?

Ashley: Sure.

The pre-training stage operates at a resolution of 256 by 256 pixels and uses image-only data where the generator learns to predict masked parts of an image based on visible patches.

This teaches the model contextual and compositional understanding purely from visual input.

They use visual self-conditioning where the condition for generation is derived from the image itself.

Evan: So it’s learning to fill in the gaps based on what it sees in the image.

That makes sense.

What comes next?

Ashley: Following pre-training is mid-training, which increases the image resolution to 512 by 512 pixels.

This stage continues to use image-only data but introduces aspect-ratio-bucketed training, where images are sampled and resized while preserving their aspect ratios.

This step ensures the model gets exposed to more spatial structure without introducing text alignment yet.

Evan: Adapting to different resolutions incrementally seems smart.

How do they handle the actual text-to-image alignment?

Ashley: That happens in the supervised fine-tuning stage, which starts with images at 512 by 512 pixels to establish text-to-image alignment.

By this point, the model already has a strong visual generative prior, so the focus here is on associating text descriptions with corresponding visual content while preserving the visual prior.

Evan: So, this aligns textual concepts with visual elements the model has already learned.

What about higher resolutions?

Ashley: After that alignment, the training progresses to 1024 by 1024 pixels.

This enables the model to generate finer textures and sharper details, while maintaining semantic alignment with higher-resolution images.

Evan: What’s the role of the real images versus synthetic ones in their dataset?

Ashley: Real images dominate their dataset, making up 98% of the 220 million training samples.

Their methodology prioritizes the realism and diversity found in real-world images, improving the model’s ability to generate realistic and high-quality images.

Evan: Fascinating.

I see they also mention parameters like RMSNorm and the Muon optimizer.

How do these come into play?

Ashley: RMSNorm is used throughout the Diffusion Transformer to stabilize optimization.

It’s a parameter-free normalization technique that improves the efficiency and scalability of the training.

The Muon optimizer further enhances this stability, especially in large-scale training scenarios.

Evan: So, the combination of this normalization technique and an advanced optimizer supports more stable and effective training.

What’s their approach to efficient inference?

Ashley: They distill the full model into a few-step variant called LLaDA-Image Turbo, using a method called TwinFlow.

This process reduces the number of sampling steps required for inference to just 2 to 4 steps, significantly cutting down the computational cost and time for inference without sacrificing much on the quality front.

Evan: That’s impressive.

And how do they evaluate the model’s performance?

Ashley: For evaluation, they use several benchmarks.

As I mentioned earlier, LLaDA-Image set new state-of-the-art scores on the Qwen-Image-Bench for both English and Chinese tracks.

They also evaluate on LongText-Bench for text rendering and CVTG-2K for complex visual text generation, both of which display the model’s robust text rendering and generation capabilities.

Evan: In summary, their method combines robust visual prior training with efficient text-to-image alignment, backed by substantial real-data usage and advanced optimization techniques to deliver a highly efficient and versatile image generator.

Ashley: Indeed.

That wraps up the Methods section for LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes.

Evan: Now that we've covered the methods, let's move on to the experiments and results.

Ashley, how did they evaluate LLaDA-Image?

Ashley: The authors conducted several rigorous experiments to evaluate the performance of LLaDA-Image across various tasks, focusing on text-to-image generation and image editing.

Evan: Let's start with text-to-image generation.

Which benchmarks did they use and what were the findings?

Ashley: They used the Qwen-Image-Bench, which assesses visual quality, aesthetics, prompt alignment, real-world fidelity, and creative generation across both English and Chinese tracks.

LLaDA-Image achieved overall scores of 53.53 on the English track and 53.38 on the Chinese track, establishing a new state-of-the-art among open-source models.

Evan: Those scores are impressive.

How does it perform in terms of specific aspects like visual quality and alignment?

Ashley: LLaDA-Image ranked first among open-source models in quality, aesthetics, and alignment on both the English and Chinese tracks.

It also excelled in creative generation on the English track.

Evan: What about the LongText-Bench and other benchmarks?

Ashley: On LongText-Bench, which tests the ability to render long text strings accurately, LLaDA-Image scored 0.923 on the English subset and 0.913 on the Chinese subset, demonstrating balanced bilingual behavior.

Evan: Balanced bilingual behavior is crucial for many applications.

How did it perform on the CVTG-2K benchmark?

Ashley: CVTG-2K focuses on complex visual text generation like advertisements and street views.

LLaDA-Image achieved an average word accuracy of 0.875, with stable performance across different numbers of text regions, and a normalized edit distance of 0.945, along with a CLIPScore of 0.818.

This indicates strong text rendering fidelity and alignment.

Evan: What other benchmarks did they consider, and how consistent were the results?

Ashley: While benchmarks like GenEval and DPG-Bench provided fine-grained evaluations, the paper notes that these may not consistently indicate practical capabilities.

However, LLaDA-Image still performed strongly, scoring 0.85 overall on GenEval, particularly excelling in single-object generation and two-object composition.

Evan: So, across various standardized benchmarks, LLaDA-Image demonstrated strong performance in both general image generation and text rendering.

Ashley: Exactly.

Now, let's shift focus to image editing.

They used GEdit-Bench to evaluate this capability, which involves diverse real-world editing tasks and measures both semantic consistency and perceptual quality.

Evan: How did LLaDA-Image fare in image editing?

Ashley: On GEdit-Bench, LLaDA-Image achieved overall scores of 7.336 on the English track and 7.294 on the Chinese track.

It showed strong semantic consistency but had a slightly lower perceptual quality score, especially on the English track.

Evan: What specific tasks were included in these editing evaluations?

Ashley: The tasks included style changes, text modifications, object removals, colorization, gesture changes, background replacements, and object additions.

LLaDA-Image performed well across these diverse tasks, demonstrating its versatility in handling complex editing instructions while preserving unedited content.

Evan: That's quite a range of editing capabilities.

Any mention of potential areas for improvement?

Ashley: Yes, while the model showed strong semantic consistency, the authors noted that further improvements in perceptual quality, particularly for high-fidelity editing, would be beneficial.

Evan: Improving perceptual quality could definitely enhance realism.

So, Ashley, what do these results suggest about the efficacy of the LLaDA-Image framework?

Ashley: The results clearly indicate that LLaDA-Image is highly capable of generating and editing photorealistic images with fine-grained control.

The use of a predominantly real-image dataset, combined with innovative training strategies, has enabled the model to achieve state-of-the-art performance on multiple benchmarks.

Evan: Moreover, the introduction of LLaDA-Image Turbo shows a significant improvement in inference efficiency with minimal quality loss, making it highly practical for scalable deployment.

Ashley: Indeed, and that wraps up the Experiment and Results section for LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes.

Evan: Let's now delve into the Related Work segment to understand the context within which LLaDA-Image fits.

Ashley, can you walk us through how this paper positions itself in relation to prior research?

Ashley: Of course, Evan.

The Related Work section in the paper situates LLaDA-Image among existing models and methods, highlighting both the foundations it builds upon and the advancements it introduces.

Evan: So, what are the key areas of previous work they discuss?

Ashley: The authors touch on several critical areas: text-to-image generation models, visual language models, and recent advances in optimization and training techniques.

Evan: Let's start with text-to-image generation.

How does LLaDA-Image relate to prior models in this domain?

Ashley: In terms of text-to-image generation, the paper references several influential models, like DALL-E, Imagen, and GLIDE.

These models have set high standards for generating realistic and coherent images from textual descriptions.

However, many of these models rely heavily on large-scale supervised training using paired image-text data, which can be expensive and challenging to obtain.

Evan: So, LLaDA-Image aims to overcome this by separating visual learning from language alignment early on.

Is that the key distinction?

Ashley: Exactly.

LLaDA-Image introduces a significant shift by relying predominantly on image-only data in the initial training stages.

This allows the model to develop a strong visual generative prior without the immediate need for paired image-text data, thus reducing dependence on costly datasets.

Evan: Interesting.

How about the visual language models?

What does the paper cite in this area?

Ashley: For visual language models, the paper discusses the evolution from models like CLIP, which effectively learn joint visual and linguistic embeddings, to more advanced structures like Flamingo, which improves zero-shot learning capabilities.

The authors emphasize that while these models excel in cross-modal understanding, LLaDA-Image advances this further by integrating a dedicated understanding-to-generation connector.

Evan: So, they are leveraging prior advancements in cross-modal embeddings and extending their application to generation tasks.

Let's move to the optimization and training techniques.

What are the notable mentions here?

Ashley: This section highlights significant contributions like RMSProp and Adam optimizers, which have been foundational in training deep learning models efficiently.

The paper builds on this foundation by using RMSNorm and the Muon optimizer to enhance the stability and scalability of training LLaDA-Image, especially given its large parameter count.

Evan: How do these specific choices in optimization contribute to LLaDA-Image's performance?

Ashley: RMSNorm is particularly effective for stabilizing training in models with large parameter spaces by normalizing without introducing additional parameters.

The Muon optimizer further refines this process, ensuring that the training remains efficient and the model converges successfully even with extensive datasets and high-resolution images.

Evan: Do they also discuss contributions from other open-source models or initiatives?

Ashley: Yes, the paper acknowledges the critical role of open-source initiatives in driving forward research in this field.

They reference platforms like Hugging Face, which host a wealth of models and datasets freely available to the research community, fostering collaborative development and benchmarking.

LLaDA-Image itself contributes back to this ecosystem by releasing its weights, code, and training recipes.

Evan: It’s great to see such collaborative ecosystem engagement.

Any final remarks on how LLaDA-Image stands out?

Ashley: The key takeaway is that LLaDA-Image brings together several strands of prior research into a cohesive, high-performing model.

By leveraging extensive real-image data and innovative training schedules, it pushes the boundaries of what open-source image generation systems can achieve, setting new benchmarks in both efficiency and effectiveness.

Evan: That makes it clear.

And with that, we conclude the Related Work section of LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes.

Evan: We've covered the methods, experiments, and related work for LLaDA-Image.

Ashley, let's summarize the key contributions and takeaways from this paper.

Ashley: LLaDA-Image makes several notable contributions.

First, it introduces a training pipeline that decouples visual prior learning from language alignment, which significantly reduces the need for paired image-text data early in the training stages.

This method predominantly uses real-image data, which enhances the model's ability to generate highly realistic images.

Evan: Right, and they achieved a cost-efficient and scalable system by focusing on image-only pre-training and mid-training before fine-tuning for text-to-image generation.

Ashley: Exactly.

Another key contribution is the introduction of the LLaDA-Image Turbo variant, which allows for efficient inference with only 2 to 4 sampling steps, making it practical for deployment while maintaining high generation quality.

Evan: The utilization of RMSNorm and the Muon optimizer further stabilizes training, proving effective even with large datasets and extensive model parameters.

Ashley: Moreover, the model sets new benchmarks on multiple evaluation datasets, demonstrating robust performance in both image generation and editing, across multiple languages.

Evan: And let's not forget their commitment to the open-source community.

By releasing their model weights, training code, and detailed recipes, they support further advancements and collaboration in image generative models.

Ashley: Indeed.

LLaDA-Image exemplifies how integrating comprehensive real-image data and innovative training techniques can push the boundaries of open-source image generation systems.

Evan: That wraps up our discussion on LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes.

Thanks for walking us through, Ashley.

Ashley: My pleasure, Evan.

I hope our listeners found it as insightful as we did.

Evan: Thank you to our audience for tuning in today.

Make sure to subscribe to the Daily Paper Cast for more in-depth discussions on cutting-edge research.

See you next time!

Ashley: Looking forward to it!

Goodbye, everyone.