UniWAM: Unified World-Action Model

🤗 Upvotes: 43 | cs.RO

Authors:
Wenxuan Song, Jiayi Chen, Jingbo Wang, Shuai Zhou, Xicheng Gong, Zehua Fan, Ziyang Zhou, Junwu E, Haodong Yan, Fuhao Li, Qize Yu, Xu Huang, Pengwei Wang, Wen Chen, Shunbo Zhou, Haoang Li

Title:
UniWAM: Unified World-Action Model

Arxiv:
http://arxiv.org/abs/2610.02054v2

Abstract:
Vision-language-action models benefit from the understanding and reasoning capabilities of pretrained vision-language models, but action-only supervision provides limited grounding in world dynamics. Conversely, world-action models inherit spatiotemporal priors from video generation models, yet remain limited in semantic understanding and reasoning under distribution shifts. We introduce UniWAM, a unified architecture that integrates a physical reasoner, a world generator, and an action predictor to jointly learn semantic understanding of the physical world, visual generation, and action prediction. To ensure the quality of the training data, we developed a rigorous data cleaning and annotation pipeline for both human egocentric data and robot data. To adapt the vision-language component to embodied tasks while preserving its inherited language capabilities, we represent low-level actions in natural language and introduce a pre-training recipe that assigns complementary supervision from visual question answering (VQA) data, human egocentric data, and robot demonstrations to the appropriate model components. During post-training, future visual noise augmentation reduces reliance on precise future predictions, while history-conditioned flow matching uses encoded action history to initialize action generation. Together, these designs significantly reduce denoising steps while maintaining performance. UniWAM achieves state-of-the-art (SOTA) performance across multiple evaluations, including in-distribution performance, robustness, generalization, instruction following, and long-horizon task execution. Furthermore, we uncover a log-linear scaling law of unified human-robot co-training, demonstrating the effectiveness of large-scale pre-training on a mixture of human and robot data.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Ashley: Today's spotlight is on a paper from the Hugging Face daily paper list of October 8, 2026, which has garnered 43 upvotes.

Evan: The title of the paper is 'UniWAM: Unified World-Action Model'.

Ashley: The first two authors are Jiayi Chen and Jingbo Wang, with corresponding author Shuai Zhou from the Hong Kong University of Science and Technology in Guangzhou.

Evan: Ashley, could you give us some background here?

What issues is this paper addressing?

Ashley: Evan.

In recent years, robot foundation models that leverage vision-language models, or VLMs, have shown significant effectiveness.

These models can capitalize on the semantic understanding capabilities gained through large-scale pretraining.

Evan: So these are the models that combine visual and linguistic information for better understanding of tasks?

Ashley: Exactly.

However, there's a challenge when it comes to understanding world dynamics using actions alone as the supervision signal.

They just don't provide enough grounding.

Evan: Interesting.

Are there models designed specifically for understanding world dynamics?

Ashley: Yes, indeed.

World-Action Models, or WAMs, utilize spatiotemporal priors from video generation models.

By predicting the next frames, they enhance their grasp of dynamic changes.

But these models also have limitations, particularly in reasoning and handling out-of-distribution scenarios.

Evan: Got it.

It sounds like both approaches have their strengths and weaknesses.

Ashley: Exactly, Evan.

The key question posed by this paper is: Can we merge the reasoning and generalization strengths of vision-language models with the dynamic understanding of world-action models to build a truly generalist robot?

Evan: Ah, combining the best of both worlds!

That's an ambitious goal.

How do they propose to achieve this?

Ashley: They introduce UniWAM, a unified architecture that integrates a VLM-based physical reasoner, a VGM-based world generator, and an action predictor.

This system is inspired by the Mixture-of-Transformer, or MoT, architecture.

Evan: Let’s break that down a bit.

What exactly are the components of this architecture?

Ashley: Sure.

The architecture has three key components: a physical reasoner for semantic understanding, a world generator for visual prediction, and an action predictor for generating actions.

Evan: What are the main challenges in pre-training this kind of hybrid model?

Ashley: There are three main challenges.

First, VLMs need to be adapted to embodied tasks while retaining their language capabilities.

This is achieved by representing low-level robot actions in natural language, aligning with the VLM’s pretraining distribution.

Evan: And the second challenge?

Ashley: The second challenge involves designing suitable supervision signals for each expert using heterogeneous data sources to avoid conflicts in the optimization processes.

They crafted a pretraining mix of visual question answering, or VQA data, robot data, and human data.

Each type of data supervises specific components of the model.

Evan: And the third challenge?

Ashley: The third challenge is ensuring data quality in large-scale training.

They developed a robust data cleaning pipeline and an automated system for segmenting and annotating human data.

Evan: They’ve thought of everything.

How does the post-training process build upon this?

Ashley: During post-training, they continue using unified supervision for the three experts.

They introduce future-visual noise augmentation to handle unreliable future predictions and history-conditioned flow matching to improve action prediction efficiency.

Evan: Can you explain the augmentation and flow matching in more detail?

Ashley: Sure.

Future-visual noise augmentation perturbs future visual inputs to train the action expert to work with less precise future predictions.

History-conditioned flow matching initializes action generation from past action histories rather than starting from scratch, making the process more efficient.

Evan: That's smart.

And how did UniWAM fare in evaluations?

Ashley: UniWAM achieved state-of-the-art performance across various benchmarks, including in-distribution performance, robustness, generalization, instruction following, and long-horizon tasks.

They also discovered a log-linear scaling law of unified human-robot co-training, which emphasizes the effectiveness of large-scale pre-training on mixed human-robot data.

Evan: It sounds like a significant advancement.

Well, that wraps up the Introduction section of the paper.

Evan: Alright, Ashley, let's dive into the Method section of the UniWAM paper.

How did the authors construct their model architecture?

Ashley: Sure, Evan.

UniWAM’s architecture is built around a concept called Mixture-of-Transformers, or MoT.

This unified architecture integrates several key components: a physical reasoner, a world generator, and an action predictor.

Evan: Got it.

Let's start with the physical reasoner.

What is its role?

Ashley: The physical reasoner is responsible for semantic understanding and language modeling.

It projects features from a vision-language model called Qwen3-VL-2B-Instruct into semantic tokens, retaining a language head for autoregressive language generation.

Evan: And how does the world generator fit into this architecture?

Ashley: The world generator predicts future visual flow using a video generation model called Wan2.2-TI2V-5B.

It processes visual information in the latent space of a frozen autoencoder, keeping the current observation latent clean while predicting future visual latents.

Evan: That makes sense.

And the action predictor?

Ashley: The action predictor embeds the action chunks with positional information and a token encoding the current state.

It predicts continuous flow fields by mapping the action history into a high-dimensional latent space.

Evan: How do these components interact with each other?

Ashley: They exchange information through joint attention.

The tokens from all three modalities—understanding, video, and action—are normalized, modulated by timestep, and projected into a shared attention space.

This cross-modal interaction allows each component to incorporate semantic context and evolving visual predictions without losing their specialized processing.

Evan: That sounds quite integrated.

How did they approach pre-training the model?

Ashley: Pre-training involves constraining how the vision-language model, or VLM, adapts to embodied tasks while preserving its pretrained knowledge.

They use a fixed coordinate convention and structured templates to describe continuous actions in natural language, aligning the low-level robot actions with the VLM's pretraining input-output distribution.

Evan: Interesting.

What kind of data sources do they use for pre-training?

Ashley: They utilize a mixture of Visual Question Answering, or VQA data, human egocentric data, and robot teleoperation data.

Specifically, VQA data supervises the physical reasoner, human data supervises both the physical reasoner and the world generator, and robot data supervises all three components—reasoner, generator, and predictor.

Evan: How are the training objectives defined across these diverse data sources?

Ashley: They use flow matching losses to supervise visual and action velocity fields.

These losses are computed as the difference between noisy source samples and clean target samples for both visual and action velocities.

Evan: So, what does the total objective look like during pre-training?

Ashley: The total objective is a combination of action, visual, and language losses, balanced by specific weights to optimize the prediction performance across modalities.

Evan: And how does the post-training process build on this base?

Ashley: In post-training, they continue unified supervision over the components, adding future-frame noise augmentation and history-conditioned flow matching.

Future-frame noise augmentation perturbs the interpolated future visual inputs to train the model to handle imperfect predictions.

Evan: Can you elaborate on history-conditioned flow matching?

Ashley: Of course.

History-conditioned flow matching starts action generation from perturbed chunks of previously executed actions instead of Gaussian noise.

This leverages recent action history to achieve temporal consistency and allows refinement with fewer inference steps.

Evan: That's intriguing.

How does this method improve the efficiency of action prediction?

Ashley: The structured prior from recent actions reduces the number of denoising steps required to generate subsequent actions, improving the inference efficiency without compromising prediction quality.

Evan: Can you summarize the key contributions of the method section?

Ashley: Sure.

The key contributions include the construction of a three-source pre-training scheme, the formulation of history-conditioned flow matching with visual noise augmentation, and extensive evaluations demonstrating UniWAM’s SOTA performance across multiple benchmarks.

Evan: Well, that's a thorough breakdown of the method section.

Now, let's mention briefly that this was the end of the Method section.

Ashley: Next up, we’ll dive into the experiments section to see how these methods perform in practice.

Evan: Let's dive into the Experiments section and see how UniWAM performs in practice.

Ashley: Evan, the authors designed their experiments to answer several key questions about UniWAM's performance.

The first question they tackled is whether UniWAM can perform manipulation tasks under in-domain conditions, complex perturbations, and out-of-distribution, or OOD, conditions.

Evan: How did they set up these experiments?

Ashley: The authors used two main benchmarks for the simulated experiments: LIBERO and RoboTwin 2.0.

LIBERO focuses on in-distribution evaluation while RoboTwin 2.0 is tested under both clean and randomized conditions to assess generalization and robustness.

Evan: That's comprehensive.

What kind of metrics did they use?

Ashley: For LIBERO, they measured the success rate across different task dimensions like spatial, object, goal, and long-horizon tasks, as presented in Table 4 of the paper.

For RoboTwin 2.0, success rates were evaluated for both clean (C2C) and randomized (C2R) settings.

Evan: What were the results on these benchmarks?

Ashley: UniWAM achieved an average success rate of 99.2% on the LIBERO benchmark, outperforming the previous best model by 0.5%.

On RoboTwin 2.0’s C2C setting, it reached a success rate of 75.14%.

In the more challenging C2R setting, UniWAM achieved a success rate of 68.32%, which was significantly higher compared to other methods.

Evan: So, it performs well across both standard and challenging conditions.

What about the real-world experiments?

Ashley: In real-world settings, they focused on two types of tasks: instruction-following and long-horizon tasks.

The instruction-following tasks included Pick-Anything, Diverse-Interaction, Place-Relative, and Drawer-Storage, each designed to test different aspects of language grounding and manipulation.

Evan: Can you share some details on how these tasks are structured?

Ashley: Sure.

For example, in the Pick-Anything task, the robot had to pick a specific object from a set of eight on the table based on an instruction, like 'pick up the orange cup and place it on the plate.' Diverse-Interaction required the robot to perform varied interactions with objects, such as placing a lid on a cup or activating a calculator.

Evan: And how did UniWAM perform in these tasks?

Ashley: UniWAM showed strong performance across these tasks.

It achieved an average success rate of 67.5% and an instruction-following rate of 82.5%.

Notably, on the Diverse-Interaction task, the success rate was 72.5%, and the instruction-following rate was 80.0%, outperforming other models by a significant margin.

Evan: Impressive.

How did it fare in the long-horizon tasks?

Ashley: For long-horizon tasks, the setup required the robot to complete multiple stages, like placing several objects and then storing them in a drawer.

UniWAM achieved an average task progress of 5.0 out of 6 stages, compared to 4.8 for another model and 3.2 for a different baseline, indicating robust performance over extended manipulation sequences.

Evan: Did they analyze the impact of different training data on performance?

Ashley: Yes, they conducted an in-depth analysis of human-robot co-training by varying the scale of robot and human data used during pre-training.

They found that increasing the amount of robot data led to consistent improvement in OOD performance, and incorporating human data became beneficial as the dataset size increased.

Evan: Did they identify any notable scaling laws in these experiments?

Ashley: Interestingly, they discovered a log-linear scaling law.

The validation loss achieved at convergence showed a nearly perfect linear relationship with the logarithm of the dataset size, indicating that both robot and human data scale well to improve UniWAM's capabilities.

Evan: That’s quite insightful.

Did they conduct any ablation studies?

Ashley: They did.

The ablation studies demonstrated the importance of including human egocentric data and VQA supervision.

Training with both robot and human data, along with VQA, produced the highest success rates, showcasing the benefits of diverse data sources.

Evan: How about the action generation strategies?

Were there specific findings?

Ashley: They compared noise-to-action (N2A) and action-to-action (A2A) generation.

The results showed that A2A, which initializes action generation from recent action history, achieves better performance with fewer inference steps, thereby improving efficiency without sacrificing prediction quality.

Evan: That’s thorough, Ashley.

It seems UniWAM stands out in various aspects from robustness to instruction-following to long-horizon tasks.

Ashley: Exactly, Evan.

These experiments highlight UniWAM’s capability to integrate semantic understanding, visual prediction, and action generation effectively, making it a strong candidate for generalist robotic models.

Evan: Well, that wraps up our coverage of the Experiment section.

Next, we'll discuss related work to see how UniWAM fits within the existing landscape of research.

Ashley: Sounds good.

Stay tuned as we delve into the related work and benchmarks that frame UniWAM’s contributions.

Evan: Let's delve into the Related Work section and see how UniWAM fits into the broader context of robotics research.

Ashley: Robotic pretraining has advanced significantly, bringing us closer to creating generalist robots capable of performing a wide range of tasks in diverse environments.

One notable project is RT-2, which jointly trains on web and robot data with tokenized actions, as reported by Zitkovich et al.

in 2023.

Evan: That's fascinating.

How do other models compare with UniWAM?

Ashley: OpenVLA, introduced by Kim et al.

in 2025, adapts pretrained vision-language representations to robot demonstrations.

Another significant work is π0, developed by Black et al.

in 2025, which combines a vision-language backbone with flow matching.

Each of these models makes important strides toward integrating language understanding with robotic actions.

Evan: It sounds like there's a strong foundation being built in this field.

Any other models worth mentioning?

Ashley: Indeed, Evan.

π0.5, also by Black et al., expands on π0 by adding semantic subtask supervision and web data.

GEN-0 from the Generalist Team in 2025 uses Unified Manipulation Interface (UMI) data to acquire physical sense and scale laws.

More recently, in 2026, Xiaomi-Robotics-1 constructed a pretraining system that incorporates both UMI data for physical understanding and teleoperation data for embodiment-specific adaptation.

Evan: How do these models integrate reasoning and action?

Ashley: Galaxea G0.5, reported by Liu et al.

in 2026, unifies reasoning and action tokens through autoregressive training on robot and visual question answering data.

Another model, Motus, designed by Bi et al.

in 2026, incorporates motion information from heterogeneous data using optical-flow-derived latent actions.

Evan: So, models concentrating on language and action are showing promise.

What about those focused on video generation and action modeling?

Ashley: Various World-Action Models focus on video generation backbones, leveraging physical dynamics priors from large-scale video pretraining to predict the future visually.

Fast-WAM, GigaWorld-Policy, ImageWAM, and S-VAM emphasize the representation encoder role in video-generation backbones rather than synthesizing complete future frames.

These models facilitate learning stronger action-generation policies through diverse representation spaces.

Evan: How does UniWAM distinguish itself in this crowded field?

Ashley: UniWAM unites a vision-language model (VLM), a video generation model (VGM), and an action predictor within a single framework, optimizing it with complementary data and supervision.

This allows UniWAM to support comprehensive multimodal understanding, visual generation, and action prediction all within one architecture, making it versatile across various requirements.

Evan: That's a unique approach.

Are there more recent trends in leveraging egocentric data for robotic models?

Ashley: Egocentric data represents a scalable source of embodied experience, capturing diverse human-object interactions from the actor's viewpoint.

Datasets like EgoDex, EgoVerse, and VITRA provide enriched video data with language, human motion, geometry, and action-related signals, making them highly suitable for large-scale embodied pretraining.

Evan: How has this data been integrated into modeling efforts?

Ashley: Early efforts focused on representation learning, and recent models like those developed by Ye et al., and Kareer et al., introduced more structured supervision through latent actions, future prediction, and geometric motion cues.

Moving closer to policy learning, egocentric data has been reconstructed to align with robot action spaces, supporting vision-language-action, or VLA, pretraining.

Evan: So it sounds like these egocentric datasets are crucial for scalable action learning.

Any standout approaches in combining human and robot data?

Ashley: Yes, Luo et al.’s recent work and others unify human and robot experiences through shared action representations and cross-embodiment pretraining, allowing egocentric data to support both scalable action learning and world modeling.

UniWAM’s integration of both human and robot data falls squarely within these advancements, enhancing its robustness and generalization.

Evan: This synthesis of diverse data sources seems groundbreaking.

What benchmarks were used to validate these models?

Ashley: Benchmarks like LIBERO, RoboTwin 2.0, and others provide comprehensive evaluations across different dimensions, including spatial understanding, object manipulation, temporal predictions, and long-horizon tasks.

These evaluations help establish the effectiveness of models like UniWAM in real-world robotic applications.

Evan: It's clear that UniWAM is built on a solid foundation of previous research while pushing the boundaries further.

That wraps up the Related Work section.

Evan: Now that we've covered the main sections of the paper, let's summarize the key contributions and takeaways.

Ashley: Sure, Evan.

At its core, the paper presents UniWAM, a unified world-action model that integrates vision-language and video generation models with an action predictor to create a generalist robotic framework.

Evan: Key to this integration are the physical reasoner, world generator, and action predictor working in harmony through joint multimodal attention.

Ashley: Right.

They also introduced novel data cleaning and annotation pipelines to ensure high-quality data, automating many of the tedious processes typically involved in handling large-scale human egocentric and robot data.

Evan: The pre-training and post-training approaches are particularly impressive, leveraging future-visual noise augmentation and history-conditioned flow matching to enhance prediction accuracy and efficiency.

Ashley: Their experiments highlighted UniWAM’s strong performance across various benchmarks, achieving state-of-the-art results in both simulated and real-world tasks.

They proved its robustness, generalization capabilities, and efficiency in instruction follow-through and long-horizon task execution.

Evan: And let’s not forget the uncovering of a log-linear scaling law, which showcases the scalability of unified human-robot co-training, making a solid case for the benefits of large-scale, diverse pre-training data.

Ashley: Indeed, Evan.

Overall, UniWAM presents a significant advancement in the field of generalist robot models, paving the way for future research and practical applications in robotics.

Evan: That’s a wrap on today's episode.

We hope you found our discussion insightful.

Ashley: Thank you for joining us on this deep dive into UniWAM.

Be sure to check out the paper for more detailed information and findings.

Evan: And don’t forget to tune in to our next episode where we’ll continue exploring cutting-edge research from the Hugging Face daily paper list.

Ashley: Thanks for listening to Daily Paper Cast.

See you next time!

Evan: See you then!