SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models

🤗 Upvotes: 103 | cs.CV

Authors:
Junchao Huang, Guian Fang, Shengju Qian, Xianghao Kong, Zhuoran Zhao, Wei Huang, Yihua Du, Zixin Zhang, Justin Cui, Yuchao Gu, Yukang Chen, Xinting Hu, Tianyu He, Shaoshuai Shi, Zhuotao Tian, Xin Wang, Mike Zheng Shou, Li Jiang

Title:
SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models

Arxiv:
http://arxiv.org/abs/2609.02886v1

Abstract:
We introduce SolarWM, a fully open foundation for building interactive video world models from data preparation through long-horizon inference. Training across heterogeneous data sources and video backbones is challenging: datasets differ in temporal scale, camera geometry, visual quality, motion, and captioning styles, while video generators use distinct representations and architectures. Naive data mixing and model-specific implementations therefore produce inconsistent supervision and make results difficult to reproduce and compare. SolarWM addresses this coupling with a reconfigurable multi-source data engine and a backbone-native adaptation framework. The engine converts 1.43 million canonical clips from 10 datasets into a unified, frame-aligned contract covering visual observations, metric camera geometry, captions, quality metadata, selection decisions, and provenance, while decoupling source processing from mixture construction. Under shared camera-conditioning, training, and inference interfaces, we instantiate four 5B--33B models based on Wan2.2, LTX-2.5, and MiniMax-H3 while preserving their native representations and objectives. A unified three-stage recipe combines bidirectional adaptation, teacher-forced autoregressive initialization, and distribution matching distillation. The resulting causal models enable real-time interaction over rollouts ranging from minutes to hours after being trained on only 5s sequences. By releasing the resulting data, pipeline, recipes, weights, and framework, SolarWM provides a reproducible and extensible foundation for interactive world-model research.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast, where we break down the latest in AI research.

Ashley: Today's paper comes from the Hugging Face daily paper list of September 3, 2026, with 103 upvotes.

It’s titled 'SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models.'

Evan: This paper is authored by Junchao Huang and Guian Fang among others, with Li Jiang from CUHK-SZ as the corresponding author.

Ashley: Let's dive into the Introduction section to understand the context and significance of this work.

The authors quote Carl Sagan, 'Imagination will often carry us to worlds that never were.' This sets the stage for their exploration into interactive video world models.

Evan: Ashley, can you explain what interactive video world models are and why they are important?

Ashley: Sure, Evan.

Interactive video world models are systems that generate visual observations of the future, based on inputs like camera motion, player actions, or semantic instructions.

Instead of just passively generating video, these models create environments that can be interactively explored and controlled.

The potential applications are vast, from simulation and embodied learning to interactive content creation.

Evan: That sounds transformative.

What are the main challenges in transitioning from short-clip generation to these long-horizon, interactive models?

Ashley: The key challenges lie in maintaining visual quality and temporal coherence over longer periods while ensuring the models remain responsive to control inputs.

Existing datasets differ significantly in temporal scale, visual quality, motion distribution, captioning styles, and camera conventions.

This heterogeneity can lead to inconsistent supervision, making it difficult for models to generalize well across different sources.

Evan: So, how does SolarWM address these challenges?

Ashley: SolarWM introduces a fully open and unified foundation for building interactive video world models.

It has a reconfigurable multi-source data infrastructure and a scalable backbone-native adaptation framework.

The data engine converts diverse sources into a common training contract with consistent temporal, geometric, semantic, and quality supervision.

On the model side, SolarWM provides shared interfaces for camera conditioning, optimization, and rollout while preserving the native representations of individual video generators.

Evan: That sounds comprehensive.

What specific contributions do the authors claim in this paper?

Ashley: The authors summarize their contributions in three main points: First, SolarWM offers a fully open and unified foundation, integrating a reconfigurable data infrastructure with scalable backbone-native adaptation.

Second, it processes approximately 1.43 million clips from 10 datasets, providing a reproducible and flexible training-mixture construction.

Finally, they train four models ranging from 5 billion to 33 billion parameters using a simple three-stage training procedure.

These models achieve state-of-the-art performance and support long-horizon rollouts after training on short 5-second sequences.

Evan: It’s impressive how they managed to streamline the training process while supporting such long rollouts.

Anything else in the introduction worth noting?

Ashley: Yes, they emphasize that their infrastructure can be reproducibly reconstructed, extended, and reused by the research community.

By decoupling computationally intensive preprocessing from mixture construction, researchers can easily define alternative training recipes without rerunning the entire preprocessing pipeline.

This makes SolarWM not just a tool, but a foundation for future research in building interactive world models.

Evan: And that wraps up the Introduction section of the paper.

We’ll now move on to the details of their methodology.

Evan: Let’s delve into the method section of the paper to see how the authors approached the development and training of SolarWM.

Ashley: Great.

The core methodology of SolarWM revolves around three main stages: bidirectional adaptation, teacher-forced autoregressive initialization, and distribution matching distillation, or DMD-based causal training.

These stages are designed to build strong, scalable, and robust interactive video world models from diverse data sources.

Evan: Can you break down what each of these stages entails?

Ashley: Of course.

The first stage, bidirectional adaptation, involves adapting pretrained video generators to camera-conditioned world data using bidirectional temporal attention.

Here, the model predicts the native flow or velocity target of its backbone from noisy latent states, minimizing the distance between the flow prediction and ground-truth target.

Camera control is integrated directly into the existing attention path using fused projective rotary positional embeddings, or PRoPE.

Evan: So the idea is to leverage the model's understanding of how to handle sequences both forwards and backwards in time, correct?

Ashley: Exactly.

This ensures the backbone model maintains its pretrained capabilities while being adapted to the new data.

It sets the stage for the next phase, which is the teacher-forced autoregressive initialization.

This stage switches the model to causal attention where the sequence is divided into ordered blocks.

The model attends to the current noisy state and clean ground-truth history while predicting each block, training it to generate future content autoregressively.

Evan: Interesting.

How does the model handle the transition between bidirectional and causal generations seamlessly?

Ashley: This transformation is critical.

The second stage is made efficient by training the model with an AnyFlow loss, which directly supervises the flow map between arbitrary noise levels.

This strategy effectively teaches the model to perform few-step autoregressive generations without needing additional complex initialization stages.

The model uses short, teacher-forced causal adaptation to fine-tune its autoregressive generation capabilities.

Evan: That sounds quite optimized.

What about the final stage, DMD-based causal training?

Ashley: In the Distribution Matching Distillation stage, the goal is to align the model’s generation with the target distribution it will use during inference.

The model generates trajectories using the same rules that will apply during actual use, conditioning on its own predictions rather than ground truth.

A frozen, bidirectional teacher model estimates the distribution of the target, while a trainable fake-distribution model follows the student's evolving rollout distribution.

The objective then is to minimize the difference between these distributions, which is used to update the student model.

Evan: So it's all about bringing the model's training distribution as close as possible to its operational distribution, right?

Ashley: That's right.

This approach ensures the model’s generation quality remains high even when it generates long sequences based solely on its own outputs.

It effectively addresses the exposure bias common in autoregressive models, where inconsistencies can snowball during extended generation tasks.

Evan: Now, what about the datasets?

You mentioned 1.43 million clips from 10 different datasets.

How are these datasets integrated into the training pipeline?

Ashley: The data processing pipeline is quite robust.

SolarWM’s data engine converts these diverse sources into a common, consistent format that aligns visual observations, camera geometry, captions, and quality metadata.

Each dataset is preprocessed separately to extract key features before being unified.

The engine applies various annotations and filters to ensure quality and consistency across the different datasets.

This results in a canonical corpus of approximately 25 terabytes covering real-world, synthetic, and game environments.

Evan: That’s impressive.

How do they ensure these diverse datasets work well together?

Ashley: They use a reconfigurable and comprehensive training resource, decoupling the computationally intensive source preprocessing from mixture construction.

This makes it easy for researchers to modify filtering criteria, sampling ratios, and source weights without needing to redo the entire preprocessing from scratch.

Evan: It’s an elegant way to handle dataset heterogeneity.

What models did they instantiate using this framework?

Ashley: They instantiated four models based on different backbones: Wan2.2, LTX-2.5, and MiniMax-H3.

The model sizes range from 5 billion to 33 billion parameters.

These models maintain their native representations and objectives but are adapted to a unified three-stage training recipe.

Evan: And how do these models interact with camera controls during training and inference?

Ashley: Camera control is a shared requirement for all models.

Each camera trajectory is converted into a geometric condition aligned with video frames, injected directly into the video attention computation.

This ensures that camera motion is accounted for in the scene generation seamlessly, integrating into the model’s flow without a separate control token or branch.

Evan: It’s quite remarkable how they manage to integrate these disparate components into a cohesive framework.

Ashley: Indeed, it shows the depth and detail put into developing SolarWM.

And that’s the end of the Method section, showcasing a comprehensive and scalable approach to training interactive video world models.

Evan: Let’s move on to the Experiment and Results section to see how SolarWM performed and what the key findings were.

Ashley: The experiments are designed to evaluate the performance of the SolarWM models through two main stages: bidirectional pretrained models and distilled causal generation.

These stages are tested across various horizons and input domains to assess model robustness and capability.

Evan: Alright, so let’s start with the bidirectional pretrained models.

What did they do here?

Ashley: In this stage, they evaluated each model route after bidirectional pretraining but before any causal adaptation or distillation.

The models generated 10-second videos from an initial out-of-domain, or OOD, first frame created using tools like GPT Image 2 and Krea.

This allowed the researchers to test the consistency and quality of the video generations across unseen visual domains and varying camera movements.

Evan: So they were essentially checking if the models could generalize well beyond the training data?

Ashley: Exactly.

They found that despite the variations in their input and camera paths, each of the four backbone routes maintained impressive visual consistency and quality, indicating strong generalization abilities.

Evan: Got it.

Now, how did the models perform in the distilled causal generation stage?

Ashley: This stage involved testing the models under both in-domain and out-of-domain settings, spanning 10-second, minute-scale, and hour-scale rollouts.

The objective was to observe how well the models could generate extended sequences based on their own previous outputs.

Evan: What were the specific findings for the in-domain validation?

Ashley: For in-domain validation, the evaluation examples were drawn from held-out samples of the same data sources used for training but not seen by the models during training.

They specifically looked at third-person and minute-scale rollouts.

Evan: And how did the models fare?

Ashley: The models generated coherent sequences, maintaining subject consistency and properly responding to prescribed camera trajectories.

This was true across real-world, synthetic, and game-domain scenes, indicating that the trained models could effectively handle a variety of environments.

Evan: What about the out-of-domain tests?

How did they perform there?

Ashley: For out-of-domain tests, the initial frames were synthesized using GPT Image 2 or Krea with deliberately novel and challenging prompts.

This setting stressed the models' ability to generalize and maintain coherence despite the unusual starting conditions.

Evan: And did they manage to hold up under such stress conditions?

Ashley: Impressively, yes.

The models successfully generated 10-second OOD rollouts that retained the initial style, palette, and main scene elements.

Even with significant visual deviations from the training corpus, they managed to create coherent and visually appealing sequences as camera movement revealed new regions.

Evan: What were the results for long sequence generation, say minute-scale and hour-scale rollouts?

Ashley: Minute-scale and hour-scale rollouts were particularly revealing.

During minute-scale tests, the models maintained scene coherence and adapted well to various camera trajectories, extending beyond the 5-second training sequences.

They generated realistic details and consistent visual quality throughout the longer sequences.

Evan: What about the hour-scale rollouts?

That’s quite a stretch!

Ashley: Indeed, it is.

The hour-scale evaluations were conducted using real, held-out first frames from the validation pool, following specific camera paths.

The models rolled out coherent video sequences over the full hour without additional reference frames or injections, demonstrating strong consistency in maintaining the scene layout and overall quality.

Evan: That’s impressive!

It shows a significant leap in generative model capabilities.

Ashley: Definitely.

The experimental results reinforce the effectiveness of the SolarWM training framework and its robust data processing pipeline.

The models not only performed well in controlled environments but also exhibited strong generalization across novel visual domains and extended time horizons.

Evan: And that wraps up the Experiment section, illustrating the robust performance and wide applicability of SolarWM models.

Evan: Now, we should consider the broader landscape of research that SolarWM fits into by looking at the Related Work section.

Ashley: Sure, Evan.

The authors position SolarWM within the context of various advancements in interactive video world models and training frameworks.

Starting with interactive video world models, these extend passive video generation into visual simulation where future observations are rolled out in response to actions, camera motion, or instructions.

Evan: How did early systems in this field approach the problem?

Ashley: Early systems focused on compact action spaces.

For instance, Genie learned latent actions from Internet videos, DIAMOND modeled Atari environments, and GameNGen simulated DOOM from recorded state-action trajectories.

However, these were often tied to specific environments and control vocabularies.

Evan: So we’ve come a long way with more recent advancements then.

Ashley: Exactly.

Recent work has pushed the envelope further by moving towards open-domain visual exploration.

Models like Yume, WorldPlay, and AlayaWorld support world extensions controlled by cameras or keyboards, ensuring long-range consistency.

Others like DreamX-World and SANA-WM have improved visual quality, rollout length, interaction, or deployment efficiency.

Evan: What makes SolarWM stand out among these advancements?

Ashley: SolarWM sets itself apart by providing a fully open and unified training framework that supports multiple video backbones.

This addresses the limitations of existing systems which often rely on restricted data sources or model-specific implementations, making systematic comparison and extensibility difficult.

Evan: What about the training frameworks and open research stacks?

How does SolarWM fit into this?

Ashley: Most high-quality video generators are pretrained using bidirectional temporal attention, but interactive rollout requires causal prediction from generated history.

Techniques like Diffusion Forcing have been developed to convert bidirectional video models through staged distillation, enabling few-step sampling.

However, these methods don’t provide the unified training stack that SolarWM introduces.

Evan: Interesting.

What similarity or differences are there in data frameworks?

Ashley: Data frameworks for interactive world modeling draw on diverse traditions.

Large video-text corpora like WebVid-10M and InternVid provide visual and semantic diversity but lack frame-aligned geometric control.

Conversely, datasets like DL3DV-10K and RealCam-Vid offer great pose, depth, or trajectory supervision.

Multi-domain collections broaden the range of scenes and interaction settings, but these sources aren’t directly interchangeable because of differences in scale, resolution, and visual quality.

Evan: So, aligning these diverse data sources into a consistent training contract is a challenge.

Ashley: Exactly.

SolarWM tackles this challenge by normalizing heterogeneous sources into a unified, frame-aligned representation.

This decouples source processing from mixture construction, allowing for flexible data mixture design without the need for expensive reprocessing.

Evan: Are there other systems that have released their training implementations?

Ashley: A few systems do indeed release training codes, like DIAMOND, Sana-WM, and WorldPlay.

However, even when code is available, it isn't always directly reproducible or configurable.

That’s another strong point for SolarWM, which makes the entire stack—from data processing to training recipes—fully open source and easily adaptable.

Evan: Seems like SolarWM provides a robust set of tools for researchers to build on.

Ashley: Definitely.

By combining bidirectional camera-conditioned adaptation, teacher-forced autoregressive initialization, and distribution matching distillation, SolarWM creates a robust and efficient training pipeline that is unprecedented in its openness and scalability.

This makes it a valuable resource in the field of interactive video world models.

Evan: That’s an excellent overview of the related work and how SolarWM fits into the broader landscape.

And that wraps up the Related Work section, giving us a clear picture of its significance.

Evan: Let's wrap up today's discussion by summarizing the key contributions and takeaways from the SolarWM paper.

Ashley: SolarWM offers a fully open and unified framework for building interactive video world models.

Its main contributions include a reconfigurable multi-source data infrastructure that processes around 1.43 million clips from 10 diverse datasets, a scalable backbone-native adaptation framework, and an efficient three-stage training methodology combining bidirectional adaptation, teacher-forced autoregressive initialization, and distribution matching distillation.

Evan: Through their experiments, the authors demonstrated that SolarWM models can generate high-quality, consistent videos even under long-horizon rollouts ranging from minutes to hours.

The models performed robustly in both in-domain and out-of-domain scenarios, maintaining visual coherence and responding accurately to varying camera trajectories.

Ashley: Furthermore, by decoupling computationally intensive source preprocessing from mixture construction, SolarWM allows researchers to easily redefine training configurations without reprocessing data.

This not only enhances the reproducibility of their results but also enables flexible experimentation across different datasets and conditions.

Evan: In summary, SolarWM sets a high standard for open, scalable, and reproducible interactive video world model research.

It's a valuable resource that can significantly accelerate advances in this field.

Ashley: And that brings us to the end of today's episode.

We hope you found this deep dive into SolarWM as fascinating as we did.

Evan: Thank you for tuning in to Daily Paper Cast.

Be sure to join us again for more insightful discussions on cutting-edge research papers from the world of AI and beyond.

Ashley: Feel free to subscribe so you never miss an episode.

Until next time, keep exploring and stay curious!