Long-WAM: Scaling the Context of World-Action Models

🤗 Upvotes: 78 | cs.RO, cs.AI, cs.CV

Authors:
Wei Huang, Bohan Zhang, Chenzhi Liu, Isabella Liu, Shuai Yang, Weian Mao, Luozhou Wang, Yicheng Xiao, Weifeng Lin, Qixin Hu, Bryan Chu, Sifei Liu, Linxi Fan, Xiaojuan Qi, Song Han, Yukang Chen

Title:
Long-WAM: Scaling the Context of World-Action Models

Arxiv:
http://arxiv.org/abs/2610.10528v1

Abstract:
Real-time robot control demands enough visual history to infer motion and task progress, but processing that history can delay action. We present Long-WAM, a model-system framework for scaling the context of causal world-action models under real-time control constraints. Our central finding is that access to history is not the same as using it: longer histories pay off far more when the video foundation is pretrained autoregressively (AR). We first learn causal prediction from robot and egocentric videos without action labels, then preserve this history-to-future structure during world-action adaptation. On RoboCasa GR-1, increasing context from 0.0 to 19.2 seconds raises success from 63.3% to 78.7%, whereas a bidirectionally pretrained initialization shows no net gain; robot-domain AR pretraining further raises peak success on GR-1 and LIBERO-Long. Long-WAM also achieves the best results among compared methods on LIBERO-Long, RoboTwin 2.0, and DOMINO. Streaming observation encoding, asynchronous execution, and hardware-specific acceleration enable deployment on RTX 5090, DGX Spark, and Jetson AGX Thor without dropping future prediction; on RTX 5090, each action chunk, including future-video latent prediction, takes 107.4 ms. Real-time deployment on Unitree G1 and YAM supports dynamic and long-horizon manipulation, including 95% success on dynamic cup stacking, where Pi0.5 and Fast-WAM succeed in none of 20 trials. As a memory-informed executor, Long-WAM also complements higher-level planning in composite tasks.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast!

Evan: Today’s paper is from the Hugging Face daily paper list of October 8, 2026, with 78 upvotes.

Ashley: The title of the paper is 'Long-WAM: Scaling the Context of World-Action Models.'

Evan: It's authored by Wei Huang and Bohan Zhang, with the corresponding author being Wei Huang from NVIDIA.

Ashley: Great, let's dive into the Introduction section.

Ashley: The paper starts by highlighting the importance of understanding scene changes in fast robot control.

A single image can reveal an object's position but provide ambiguous information about its motion and interaction progress.

Evan: Right, and world-action models, or WAMs, aim to bridge this gap by bringing video prediction into what's known as closed-loop control.

But Ashley, what’s the challenge here?

Ashley: The challenge lies in processing the visual history quickly enough to inform real-time decisions.

More history can enhance control accuracy but also introduce delays in action execution.

Evan: So it's not just about having access to more historical data, right?

It’s also about how effectively the model can use it to make predictions.

Ashley: Exactly.

Recent WAMs often retain history using causal caches and persistent or selected memories.

However, there’s a distinction between merely accessing this history and truly learning from it to make accurate predictions.

Evan: The Introduction mentions certain existing models.

Can you give us a brief overview of those?

Ashley: Sure.

There’s DreamZero and LingBot-VA, both of which adapt bidirectionally pretrained video generators for causal video-action prediction.

Meanwhile, LingBot-VA 2.0 pretrains both causal video and learned latent actions.

Evan: Interesting.

So what approach do Wei Huang and colleagues propose with Long-WAM?

Ashley: They propose first learning autoregressive, or AR, video prediction, then adapting to actions while preserving the history-to-future structure.

Essentially, AR video pretraining learns causal temporal structures that are retained during adaptation to real-time action generation.

Evan: This leads us to their key contribution, right?

Ashley: Exactly.

Their central finding is that longer visual context substantially boosts performance, but this effect heavily depends on how the video model is pretrained.

For instance, extending the context window from zero to 19.2 seconds on the RoboCasa GR-1 setting shows a notable increase in task success.

Evan: So AR-pretrained foundations see significant gains, while those pretrained bidirectionally do not.

Can you give us some numbers to illustrate?

Ashley: Of course.

On the RoboCasa GR-1, increasing the history window from zero to 19.2 seconds raised the success rate from 63.3% to 78.7%.

In contrast, a bidirectional pretrained initialization showed no net gain in the same scenario.

Evan: And how does this AR pretraining benefit real-world performance?

Ashley: The AR model already factors in causal temporal relationships, so it's better at using historical data to inform future actions.

Long-WAM, as a result, preserves the learned structure during action adaptation, enhancing performance on tasks requiring extended context.

Evan: But longer context only helps if the system can still respond quickly, right?

Ashley: Right.

That's why Long-WAM combines asynchronous execution and edge acceleration to maintain responsiveness.

Predictive conditioning from the AR video foundation supports smooth action transitions without the need for blending or prefix guidance, keeping the action handoffs seamless.

Evan: How exactly do the mechanisms like streaming VAE encoding and hardware-specific acceleration contribute to this?

Ashley: Streaming VAE encodes incoming observations in real-time while the robot executes actions, thus reducing post-trigger computation time.

The use of NVFP4 quantization along with kernel tuning accelerates the whole video-action pipeline on hardware like RTX 5090, DGX Spark, and Jetson AGX Thor.

Evan: So all these optimizations make it possible for the system to handle long context without compromising real-time performance.

Got it.

Ashley: Yes, exactly.

For instance, on the RTX 5090, each action chunk, including future-video latent prediction, takes about 107.4 milliseconds, which is considerably fast.

Evan: And what about their real-world deployments?

Any performance metrics there?

Ashley: Sure, Long-WAM demonstrated a 95% success rate on dynamic cup stacking tasks with the Unitree G1 robot, outperforming other models that failed to succeed in any of the 20 trials.

It also supports longer task durations, showing 81.7% success on YAM tasks lasting over 40 seconds on average.

Evan: That's impressive.

It sounds like Long-WAM strikes a balance between leveraging extensive historical context and maintaining real-time performance.

Ashley: Correct.

And that wraps up the Introduction section of the paper.

Evan: Alright, let's move on to the meat of the paper: the Method section.

Ashley: This section outlines the core components and steps involved in their proposed model, Long-WAM, for scaling the context of world-action models in real-time robot control.

Evan: Where do they start with this?

Ashley: The authors begin with the pretraining of what they call LongLive2.0-Robot, which is the foundation for their autoregressive video pretraining.

Evan: And what does this pretraining involve?

Ashley: This involves training on approximately 10,000 hours of robot and egocentric human videos across five different datasets.

They utilize a specific method called sequence-parallel autoregressive training, or AR training.

Evan: Sequence-parallel autoregressive training?

Could you explain that a bit more?

Ashley: Certainly.

This method involves breaking down long video sequences across multiple GPUs, which allows the model to handle extended motion and interaction histories efficiently.

Essentially, it helps the model learn from long sequences while keeping the computational load manageable.

Evan: How exactly do they structure this training?

Ashley: They follow a teacher-forcing formulation, which means the model is trained to predict the next step in a sequence based on previous steps, with noisy chunks supervised by their clean ground-truth prefixes.

This helps in learning predictive dynamics very effectively.

Evan: Got it.

What happens after the pretraining?

Ashley: Post-pretraining, they adapt the model for action generation.

This involves what they call causal-to-causal world-action adaptation.

Essentially, they maintain the causal video dependencies learned during pretraining while conditioning the denoising of action chunks on observed history.

Evan: What does this adaptation process look like?

Ashley: The visual information from observed history is used to predict future video latents.

This predicted future, along with the historical data, is then used to generate action sequences.

The action model thus inherits a visual dynamics prior explicitly trained for predicting the future.

Evan: So it’s like the model is imagining the future before taking its actions?

Ashley: Exactly.

They refer to this as inverse dynamics modeling, where the action expert generates actions based on a predicted future scenario, which allows for more accurate and context-aware decision-making.

Evan: Interesting.

And how do they handle the computational aspect, especially considering the real-time constraints?

Ashley: Great question.

They designed the system to use asynchronous execution combined with edge-specific accelerations.

For instance, streaming VAE encodes observation frames as they come in, which spreads out the computational load and reduces latency.

Evan: That sounds very efficient.

What hardware do they deploy it on?

Ashley: They deploy the model across several devices, including the RTX 5090, DGX Spark, and Jetson AGX Thor.

All these optimizations enable the system to perform real-time control on these edge devices without sacrificing accuracy.

Evan: Fantastic.

Does the paper include any metrics or figures about this efficiency?

Ashley: Yes, they report that on the RTX 5090, each action chunk, which includes future-video latent prediction, takes about 107.4 milliseconds.

This makes it viable for real-time deployment.

Evan: And how do they validate their approach?

Do they run specific experiments?

Ashley: They run experiments on several benchmarks, including LIBERO-Long, RoboTwin 2.0, DOMINO, and RoboCasa.

They compare their method with various baselines and show substantial improvement in success rates across different conditions.

Evan: Can you give us some specific results?

Ashley: Sure.

For instance, on RoboCasa GR-1, extending the history window from zero to 19.2 seconds raised the success rate from 63.3% to 78.7%.

Similarly, on LIBERO-Long, they achieved a 5% increase in success rate, peaking at 99.5%.

Evan: That's quite the jump.

It underscores the value of longer historical context.

Ashley: 'Precisely.

They also show that the impact of context scaling is more significant when the video backbone is pretrained autoregressively.

This makes a strong case for AR video pretraining as a valuable step in WAMs.

Evan: How about their real-world tests?

Do they include results from those as well?

Ashley: Yes, they do.

For example, Long-WAM achieved about 95% success on dynamic cup stacking with the Unitree G1, a notable feat considering no other models succeeded in any of the trials.

It also achieved 81.7% success on longer-horizon tasks with the YAM robot, indicating robust performance extending over 40 seconds.

Evan: So, the methods section really dives into the nitty-gritty of how they built Long-WAM and validated it both in simulated environments and real-world scenarios.

Ashley: Exactly, and it provides a comprehensive look at how scaling the context and carefully designed real-time execution can significantly improve the performance of world-action models.

Evan: That covers the Method section of Long-WAM.

Evan: Let's jump into the Experiments and Results, Ashley.

How did the authors validate Long-WAM?

Ashley: They conducted extensive evaluations across several simulation benchmarks and real-world tasks to demonstrate the effectiveness of their approach.

Evan: Sounds comprehensive.

What are the key benchmarks they used?

Ashley: The main benchmarks include LIBERO, RoboTwin 2.0, DOMINO, and RoboCasa GR-1.

Each benchmark is designed to test different aspects of robot control and task execution.

Evan: And what sort of tasks were these benchmarks focused on?

Ashley: These tasks ranged from single-arm to bimanual manipulation, dynamic environments with moving objects, and complex household tasks.

For instance, LIBERO focused on single-arm manipulation, while RoboCasa GR-1 included multi-stage tasks with several object transfers.

Evan: That’s quite a variety.

How did Long-WAM perform in these benchmarks?

Ashley: Long-WAM showed remarkable performance.

On LIBERO-Long, which features longer tasks, Long-WAM achieved the highest average success rate of 99.5%, outperforming all other compared methods.

Evan: 99.5% is impressive.

How about RoboTwin 2.0?

Ashley: On RoboTwin 2.0, Long-WAM (IDM) led with an average success rate of 94.4%, holding strong in both clean and randomized settings.

Evan: And what about the more dynamic situations, like DOMINO?

Ashley: For DOMINO, after fine-tuning on dynamic data, Long-WAM achieved a 34.9% success rate and a manipulation score of 45.1, significantly surpassing the strongest baseline models by 15.0 percentage points in success rate and 10.1 points in manipulation score.

Evan: It seems Long-WAM handles both static and dynamic environments well.

Were any longer-context studies conducted as part of this evaluation?

Ashley: They performed a context scaling study on the RoboCasa GR-1 benchmark.

Extending the history window from 2.4 to 19.2 seconds increased the success rate from 66.3% to 78.7%.

This indicates the importance of extended historical context in performing multi-stage tasks.

Evan: Interesting.

Extending context seems to have a tangible impact.

Did these improvements plateau or continue scaling with longer context?

Ashley: There was a drop-off at about 38.4 seconds, where success decreased to around 75.2%.

They hypothesize that this decline is due to the dataset having sparse real history, leading to a large portion of the inputs being padding.

Evan: So there is a practical upper limit to how much historical context is useful, at least with current datasets?

Ashley: Exactly.

The results suggest that improving these datasets with more dense historical data could further enhance performance.

Evan: What about real-world tests?

How did Long-WAM perform outside of simulations?

Ashley: In real-world deployment, Long-WAM excelled in several challenging tasks.

For instance, on the Unitree G1 robot, it achieved a 95% success rate on dynamic cup stacking, whereas models like Fast-WAM and π0.5 failed to succeed in any trials.

Evan: 95% on dynamic tasks is quite impressive.

How did it perform on longer-duration tasks?

Ashley: On YAM, Long-WAM succeeded in 81.7% of tasks involving complex, long-duration manipulations lasting over 40 seconds.

This included tasks like brick sorting by color, placing dumplings in a pan, and stacking bowls.

Evan: Over 40 seconds!

It seems like the system is quite robust.

Ashley: Indeed.

The real-world deployments validate not just the accuracy but also the robustness and adaptability of Long-WAM in dynamic and extended scenarios.

Evan: Did the authors provide any visualization or detailed rollouts of these tasks?

Ashley: Yes, they included detailed rollouts.

For instance, they provided visual evidence of Long-WAM completing a series of tasks with both overview and wrist views to illustrate the step-by-step execution and transitions.

Evan: That’s very helpful for understanding the system's performance in real-world scenarios.

Anything else noteworthy in terms of results?

Ashley: One noteworthy point is that they found lower-latency, asynchronous action generation to be particularly effective.

By overlapping model inference with ongoing robot execution, they managed to keep the system highly responsive.

Evan: Thanks for breaking down the experiments and results, Ashley.

That definitely gives us a clearer picture of Long-WAM's capabilities in both simulated and real-world settings.

Ashley: You're welcome.

That wraps up the Experiment section of the paper.

Evan: Alright, Ashley, let’s delve into the Related Work section.

How does Long-WAM position itself within the existing body of research?

Ashley: The authors provide a comprehensive overview of the existing landscape in world-action modeling, focusing on several key areas including video generators, memory and context scaling, and generative control for real-time deployment.

Evan: Let's start with world-action modeling from video generators.

Which models do they benchmark against?

Ashley: They compare Long-WAM with methods like Motus, DreamZero, and LingBot-VA.

Motus integrates pretrained video, vision-language, and action experts, whereas DreamZero and LingBot-VA adapt bidirectional video backbones to causal control.

Evan: So, what distinguishes Long-WAM from these models?

Ashley: The key difference lies in the pretraining approach.

Long-WAM undergoes autoregressive, or AR, video pretraining specifically for history-to-future predictions.

This pretraining strategy is then retained during action adaptation.

Evan: That's interesting.

And how about the models that jointly pretrain causal video with learned latent actions?

Ashley: For instance, LingBot-VA 2.0 does this joint pretraining.

However, Long-WAM opts to first learn from AR video pretraining, which separates the predictive pretraining task from subsequent action adaptation.

This staged approach sets it apart from such models.

Evan: Got it.

Now, the paper mentions something about memory and context scaling.

What can you tell us about that?

Ashley: Memory and context scaling involve managing the amount of historical data a model uses and how it retrieves this data for action generation.

Long-WAM leverages causal caches, compressed histories, and event retrieval to handle different memory interfaces.

Evan: And how does this benefit their model?

Ashley: This allows Long-WAM to scale the duration of historical observations effectively, improving decision-making without significant computational overhead.

The paper explains that their method enhances predictive power by using a longer context compared to other models.

Evan: Are there any specific studies that they mention which look at memory for video generation or context scaling?

Ashley: Yes, they reference studies like Echo-Memory, which explores memory for camera-conditioned video generation, and RoboTTT, focusing on context scaling in robot policies.

These studies provide the groundwork for understanding how to manage historical data efficiently.

Evan: And how does their work fit into the bigger picture of efficient and deployable generative control?

Ashley: The authors discuss the cost-intensive nature of online WAM control with large video backbones.

Systems like DreamZero and LingBot-VA mitigate costs with various optimizations, such as caching and few-step generation.

Evan: So, what’s the approach taken by Long-WAM for this challenge?

Ashley: Long-WAM enhances efficiency through NVFP4 quantization, CUDA Graph replay, PyTorch compilation, and asynchronous VAE decoding.

These optimizations ensure fast, real-time action generation even with large historical context windows.

Evan: Do they mention any other techniques or methods that inspired their approach?

Ashley: Yes, they draw on complementary works in compilation, kernel orchestration, and video quantization.

The need to balance execution speed and maintaining temporal alignment in transitions is a recurring theme in many studies.

Evan: It sounds like Long-WAM is quite comprehensive in its approach.

How do they ensure consistency in asynchronous deployments?

Ashley: They emphasize the importance of consistency between consecutive action chunks.

Long-WAM achieves this through streaming causal VAE encoding and edge-specific kernel tuning, which maintain continuity and speed concurrently.

Evan: Given all these enhancements, how does Long-WAM compare with its contemporaries in terms of results?

Ashley: In their results, Long-WAM outperforms other models like Fast-WAM and π0.5 in dynamic manipulation tasks.

They achieve higher success rates and better temporal smoothness, validating the efficiency and effectiveness of their enhancements.

Evan: That's pretty significant.

It sounds like Long-WAM sets a new benchmark in this space.

Ashley: Indeed.

It proves that a well-pretrained AR video foundation combined with smart context management and edge-specific optimizations can achieve impressive results in real-time robotic control.

Evan: Thanks for that detailed walkthrough, Ashley.

That concludes the Related Work section.

Evan: We've covered a lot today, so let's take a moment to summarize the key contributions and takeaways from this paper.

Ashley: Long-WAM's core innovation is its approach to scaling the context of world-action models through autoregressive video pretraining.

By focusing on the causal temporal relationships, the model can leverage extensive visual history to improve prediction and decision accuracy.

Evan: One major takeaway here is the importance of how the visual foundation is pretrained.

Models with AR pretraining see substantial benefits from longer historical contexts, while bidirectionally pretrained models do not.

Ashley: Exactly.

This shift highlights the significance of predictive learning over merely accessing historical data.

Additionally, the efficiency mechanisms, including asynchronous execution and hardware optimizations, ensure that Long-WAM maintains real-time performance.

Evan: This balance between leveraging long contexts and maintaining low latency is key for real-world applications, as demonstrated by their impressive results on both simulated benchmarks and actual robots.

Ashley: Correct.

With deployment on devices like RTX 5090 and Jetson AGX Thor, Long-WAM achieves fast action generation without sacrificing accuracy.

Real-world deployments with high success rates in dynamic and long-duration tasks further validate its robustness.

Evan: In essence, Long-WAM sets a new standard for real-time robot control by intelligently scaling context and optimizing execution.

It shows that a thoughtful combination of pretraining, context management, and execution efficiency can lead to significant performance improvements.

Ashley: That's right, Evan.

And with that, we conclude our episode on Long-WAM.

Evan: Thank you all for tuning in to Daily Paper Cast.

We hope you found today’s discussion insightful.

Ashley: Be sure to join us tomorrow for more in-depth looks at cutting-edge research in AI and robotics.

Evan: Until next time, this is Evan and Ashley signing off.

Have a great day!