GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation

🤗 Upvotes: 37 | cs.RO, cs.CV

Authors:
AgiBot Research Team, Renhang Liu, Wenzhi Zhao, Zhuo Yang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Guanghui Ren, Youlun Peng, Rongjun Jin, Nan Wang, Sukai Wang, Xindong He, Jinyuan Feng, Ziyu Xiong, Linqing Zhong, Yifei Wei, Feng Han, Long Zhang, Da Huang, Nanshu Zhao, Chenghao Yin, Mo Wu, Zhaodong Yan, Kongtao Hu, Yuxiang Yan, Aogelijiang Niyazi, Yu Fang, Jia Zeng, Lizhu Meng, Daizhen Lv, Haoyu Cao, Zhiwen Hou, Lianjin Ye, Yuehan Niu, Zhikai Cai, Xuan Hu, Hui Min, Xiongfeng Cai, Yue Liao, Jing Wu, Soujanya Poria, Ye Li, Sanping Zhou, Maoqing Yao

Title:
GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation

Arxiv:
http://arxiv.org/abs/2609.05588v1

Abstract:
World-action models (WAM) predict future states to guide robot actions, enabling learning from both action-free video and action-labeled interaction. Most inherit pretrained video generators, leaving WAM pretraining and scaling underexplored. We introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model whose trainable generative and action components are all initialized from scratch on manipulation data. It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM). CoAE retains action- and instruction-relevant information under aggressive compression, while SVP produces a complete future state in one differentiable pass, so visual planning and inverse dynamics can be pretrained separately on complementary data. The components are then jointly trained with knowledge-aligned selective optimization (KASO), which reduces mismatched supervision by selecting only predicted futures judged behaviorally compatible with the recorded action. We evaluate pretrained checkpoints directly, without per-task fine-tuning, on 100 tasks across 20 manipulation skill groups with held-out scenes, backgrounds, lighting, and object instances. Scaling co-training data from 300 to 30,000 hours raises success from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D; despite comprising less than 2% of the co-training data, G2-90D improves by 17.7 points, suggesting cross-embodiment transfer. Gains span 19/20 and 18/20 skill groups, and skill-specific coverage strongly correlates with zero-shot out-of-distribution (OOD) success (Pearson r=0.80; Spearman rho=0.85). Under the same protocol, the model grounds object, color, shape, and position references in at least 90% of trials and follows explicit instructions even when they conflict with an already-committed behavior or a conventional scene association.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast, your favorite podcast for the latest in AI research.

Ashley: Today, we're diving into a paper from the Hugging Face daily paper list of September 9, 2026, which has received 37 upvotes.

Evan: The paper is titled 'GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation'.

Ashley: It's authored by Renhang Liu and Wenzhi Zhao, with the corresponding author being Shengcong Chen from the AgiBot Research Team.

Evan: Alright, let's dive into the introduction.

The rise of large-scale pretraining has significantly advanced fields like language modeling, multimodal understanding, and visual generation.

Ashley: In robotic manipulation, vision-language-action (VLA) policies have made strides by mapping observations and instructions directly to motor commands.

However, they often overlook the modeling of physical dynamics in interactions.

Evan: That's where world-action models, or WAMs, come in.

These models predict how a scene will unfold and use those predictions to generate actions.

This more comprehensive approach allows better understanding and generation of actions in robotic systems.

Ashley: Exactly.

Current research in WAMs typically builds on pre-trained video generators.

But there's a gap in understanding how to pre-train and scale these models specifically for robotic manipulation from scratch.

Evan: This paper introduces Genie Envisioner Act 2.0, or GE-Act 2.0.

Unlike existing models, GE-Act 2.0 initializes every component from scratch using manipulation data.

Ashley: It uses a control-oriented autoencoder, or CoAE, to retain crucial information under heavy compression.

It also features a single-step visual planner, SVP, which produces a completed future state in one differentiable pass, and an inverse dynamics model, or IDM.

Evan: These components are then jointly trained using a method called Knowledge-Aligned Selective Optimization, or KASO.

KASO aims to reduce mismatched supervision by selecting only those predicted future states that align behaviorally with recorded actions.

Ashley: The paper's authors evaluate these pre-trained checkpoints on 100 tasks across 20 manipulation skill groups.

They find significant improvements in zero-shot out-of-distribution performance as they scale the co-training data from 300 to 30,000 hours.

Evan: For example, success rates increased from 17.1% to 44.1% on one set of tasks and from 13.4% to 31.1% on another.

These gains were observed across almost all skill groups and were strongly correlated with skill-specific coverage.

Ashley: The authors also noted that the model could ground object, color, shape, and position references in at least 90% of trials.

Qualitative stress tests showed that it followed explicit instructions even when they conflicted with already committed behaviors or conventional scene associations.

Evan: So, to summarize, this paper addresses two main questions: how to pre-train visual generation and inverse dynamics models on their respective data before connecting them, and how WAM capability scales with manipulation data when all components are initialized from scratch.

Ashley: Concretely, they contribute a new world-action model, GE-Act 2.0, that focuses on scalable pre-training, identifies the validity gap in supervision, and demonstrates systematic scaling improvements.

Evan: That's the end of the introduction section.

Next, we'll dive into the methods and key components that make up GE-Act 2.0.

Evan: Let's dive into the core of GE-Act 2.0, starting with its method and architecture.

Ashley: GE-Act 2.0 is designed around a decoupled two-stage architecture featuring three main components: the control-oriented autoencoder, a single-step visual planner, and an inverse dynamics model.

Evan: So, let's start with the control-oriented autoencoder, or CoAE.

What's its purpose in the system?

Ashley: The CoAE aims to create a compact latent space that retains essential information needed for action recovery and prediction.

Unlike typical video autoencoders optimized for reconstruction fidelity, CoAE aggressively compresses the latent space for manipulation tasks.

Evan: That's intriguing.

How does it manage this aggressive compression while retaining crucial information?

Ashley: It uses a combination of pixel reconstruction, perceptual losses, and adversarial losses during training.

Additionally, it aligns its latents with three frozen visual teachers to incorporate semantic and spatiotemporal structures.

Evan: And these visual teachers are?

Ashley: They are SigLIP 2, V-JEPA 2.1, and DINOv3.

Each provides a unique aspect like language-aligned semantics, spatiotemporal structures, and dense visual features to the autoencoder.

Evan: Got it.

Now let's move on to the single-step visual planner, or SVP.

What's unique about it?

Ashley: The SVP stands out because it creates a complete future visual state in one differentiable pass.

This single-step generation allows the visual planner and the inverse dynamics model to be pre-trained separately on different data before being connected.

Evan: How exactly does this single-step generation work?

Ashley: The SVP is essentially a flow generator that uses the multi-view diffusion transformer pattern.

It grounds the instruction in the current observation, and then fully denoises dense and sparse future visual latents in one go.

Evan: Why is this single-step important for the overall architecture?

Ashley: The single-step flow generator reduces the computational and memory overhead associated with multi-step video generation.

It enables efficient end-to-end co-training by ensuring that the gradients can propagate through the entire generation sequence in one pass.

Evan: That makes sense.

Finally, let's talk about the inverse dynamics model, or IDM.

How does it fit into this system?

Ashley: The IDM predicts actions from the future visual states generated by the SVP.

During separate pre-training, it learns from robot trajectories without requiring successful trial labels or action annotations.

Evan: And during co-training, it's connected to the SVP.

How does this pairing happen?

Ashley: Exactly.

During co-training, the IDM utilizes the future states generated by the SVP to predict actions.

The key here is that this co-training is guided by Knowledge-Aligned Selective Optimization, or KASO, to reduce mismatched supervision.

Evan: Talk to me about KASO.

How does it improve the co-training process?

Ashley: KASO works by sampling multiple future states generated by the SVP and then training the system only on those states that align behaviorally with the recorded actions.

This selective process minimizes the validity gap where the predicted future state does not match the action label.

Evan: So, it ensures that the IDM is trained on compatible state-action pairs?

Ashley: Precisely.

By focusing only on action-compatible futures, KASO preserves broader action-space coverage which is crucial for subsequent reinforcement learning or fine-tuning steps.

Evan: What about the evaluation of these models?

I understand they tested on 100 tasks across 20 skill groups.

Ashley: Yes, each skill group represents a distinct manipulation category, ranging from basic tasks like picking up objects to more complex ones like folding towels.

The evaluation excludes task-specific fine-tuning, which reflects the pre-trained model's zero-shot capability.

Evan: How did the scaling of training data from 300 to 30,000 hours impact the model's performance?

Ashley: The results show substantial improvements.

For instance, zero-shot success rates increased from 17.1% to 44.1% on one set of tasks, and from 13.4% to 31.1% on another.

Despite G2-90D tasks constituting less than 2% of the training data, success rates for these rarely seen tasks significantly improved.

Evan: That's impressive.

Did any specific skill groups show notable gains?

Ashley: Yes, improvements spanned nearly all skill groups.

Notably, tasks involving wiping, pouring, and picking objects saw some of the most significant gains, reflecting a broad enhancement in the model's manipulation capabilities.

Evan: It sounds like GE-Act 2.0 made some significant strides.

Any qualitative assessments?

Ashley: Indeed.

Qualitative tests demonstrated that GE-Act 2.0 follows explicit instructions even when those instructions conflict with an ongoing behavior or conventional scene associations.

This was evident in complex scenarios like behavioral or semantic conflicts where the model adhered to the given commands despite the presence of strong priors.

Evan: It looks like GE-Act 2.0 sets a new standard in world-action models for robotic manipulation.

That's the end of the Method section.

After this break, we'll explore the experiments and results in more detail.

Evan: Let's dive into the experiments and results of GE-Act 2.0 to see how well this model actually performs.

Ashley: The primary evaluation of GE-Act 2.0 is its performance across 100 atomic tasks spread over 20 detailed skill groups.

These include picking, placing, wiping, stacking, passing, folding, and many other complex manipulations.

Evan: It sounds like a comprehensive suite of tasks.

How were these tasks evaluated?

Ashley: Each task was run in a zero-shot manner, meaning the model did not receive any task-specific fine-tuning.

The tasks were performed with two different embodiments – G1-OP, which dominates the co-training data, and G2-90D, a sparsely represented embodiment.

Evan: How was the data scaled to examine the model's performance?

Ashley: They evaluated four different scales of training data: 300, 1,200, 5,000, and 30,000 hours.

Each scale added nested increments, maintaining the same mixture of data sources.

Evan: Let's talk numbers.

What were the success rates at different scales?

Ashley: G1-OP had success rates rise from 17.1% with 300 hours of training data to 44.1% with 30,000 hours.

Similarly, G2-90D showed improvements from 13.4% to 31.1% with the same scaling.

Evan: That's a significant jump.

Were the gains uniform across all tasks and skill groups?

Ashley: Not entirely uniform, but broad.

Out of 20 skill groups, 19 in G1-OP and 18 in G2-90D showed improved performance.

This scaling effect boosted the model's capabilities across a wide range of tasks.

Evan: Can you give us some specific examples of skill groups that showed notable improvements?

Ashley: Of course.

For G1-OP, the 'Wipe' skill group saw an increase of 46.7 percentage points, 'Separate' improved by 43.3 points, and 'Pick' by 39.2 points.

These gains were substantial and indicated broad enhancement in the model's ability.

Evan: What about the sparsely represented G2-90D embodiment?

How did it fare?

Ashley: Interestingly, even with less than 2% of the co-training data, G2-90D saw meaningful improvements.

For example, tasks involving 'Flip' and 'Separate' saw their success rates rise to 20% and 16.7% respectively, even with minimal data.

Evan: It's impressive that the model adapted well to such minimal data.

Ashley: This suggests that GE-Act 2.0 learned useful cross-embodiment skills from the broader training corpus, highlighting its adaptability.

Evan: That's the quantitative side.

Were there any qualitative evaluations or stress tests?

Ashley: Yes, there were qualitative tests that showed the model's robustness in following explicit instructions even under conflicting scenarios.

For instance, the model successfully followed new commands during behavioral conflicts like target switches or arm switches midway through the task.

Evan: And were there any interesting observations from these tests?

Ashley: Yes, the model demonstrated the ability to override conventional associations, such as placing a cup into a shoebox when instructed, even though it typically wouldn't do so based on scene priors.

This highlights the model's capacity for nuanced understanding and task execution based on explicit instructions.

Evan: That's quite remarkable.

Any final takeaways from the experiment results?

Ashley: The key takeaway is that scaling the amount of training data significantly enhances the model's performance across a wide range of tasks, and the model shows strong cross-embodiment adaptability.

GE-Act 2.0 sets a solid foundation for future work in scalable pre-training and world-action modeling for robotic manipulation.

Evan: That's the end of the Experiment section.

Stay tuned, as we'll explore the paper's broader implications and related work next.

Evan: Alright, let's shift gears and talk about how GE-Act 2.0 fits into the broader landscape of robotic manipulation research.

Ashley: Sure thing.

The paper delves into related work across several domains: generalist policies, world-action models, video and control representations, fast video generation, and action alignment.

Evan: Let's start with generalist policies.

How does GE-Act 2.0 compare to existing research in this area?

Ashley: Generalist robot policies aim to scale language-conditioned control across diverse task collections.

Prominent examples include π0, RT-1, and the Generalist AI Team's GEN series.

GE-Act 2.0 distinguishes itself with a decoupled two-stage design, allowing for standalone pretraining of visual generation and action models.

Evan: I see.

And how does it align with world-action models specifically?

Ashley: World-action models link visual prediction with control through various interfaces.

Some designs, like Jang et al.'s DreamGen, generate future visual states first and then infer actions.

Others predict video and actions within a shared backbone, such as Cheang et al.'s GR2.

GE-Act 2.0's innovation lies in producing a single complete future visual state that interfaces with an independently pretrained IDM.

Evan: What about representations for robot video and control?

How does GE-Act 2.0 differ here?

Ashley: Most prior latent video generators are based on autoencoders optimized for content reconstruction.

Systems like Genie, LAPA, and IGOR focus on linking visual tokens to robot actions.

GE-Act 2.0's CoAE addresses the need for an aggressively compressed, control-oriented latent space to directly support downstream visual planning and action prediction.

Evan: The paper mentions fast video generation too.

How does GE-Act 2.0 improve on these methods?

Ashley: In robot world models, video generation often builds on pretrained video generators adapted from natural video.

For instance, systems like MeanFlow and Seaweed-APT focus on fast-one-step video generation through distillation or adversarial post-training.

GE-Act 2.0's single-step visual planner represents a conditional extension of these techniques, producing complete future visual states efficiently.

Evan: That’s quite an improvement in efficiency.

Let’s talk about action alignment.

What did the paper have to say on that front?

Ashley: Action alignment ensures that the generated future states match the recorded actions for coherent training.

Systems like LingBot-VA use teacher forcing to match recorded and generated states, whereas other approaches detach the sampling path.

Here, GE-Act 2.0’s KASO addresses mismatches by training on selected futures that align behaviorally with the recorded actions.

Evan: That's quite comprehensive.

Any notable comparisons or references to make sense of GE-Act 2.0's innovations?

Ashley: Yes, the paper references a range of contemporary works, including Jang et al., Wu et al., and Ye et al., highlighting both methodological similarities and distinct improvements in scalability and efficiency.

Furthermore, the CoAE's aggressive compression is compared against methods like DC-AE and standard video autoencoders, showing significant advantages in preserving control-relevant information.

Evan: So, GE-Act 2.0 not only leverages existing methodologies but significantly enhances them with innovations in pretraining and efficiency.

Ashley: Exactly.

By targeting key bottlenecks like the validity gap and integrating advances in visual and action model pretraining, GE-Act 2.0 positions itself as a robust framework in the evolution of world-action models for robotic manipulation.

Evan: That’s the end of the Related Work section.

Next, we’ll discuss the conclusions and implications of GE-Act 2.0’s findings.

Evan: Now, let's wrap up by summarizing the key contributions and takeaways of the GE-Act 2.0 paper.

Ashley: Certainly.

GE-Act 2.0 makes several significant contributions to the field of robotic manipulation.

Firstly, it introduces a scalable world-action model designed from scratch using manipulation data.

This model integrates a control-oriented autoencoder, a single-step visual planner, and an inverse dynamics model.

Evan: The control-oriented autoencoder, or CoAE, creates a highly compressed latent space that retains crucial action-relevant information, making future predictions more efficient.

Ashley: Then there's the single-step visual planner, which generates complete future visual states in one pass.

This efficient generation process allows the visual planner and the inverse dynamics model to be pretrained separately on complementary data.

Evan: We also have the Knowledge-Aligned Selective Optimization, or KASO, method.

It reduces mismatched supervision by selecting action-compatible future states, which preserves broader action-space coverage.

Ashley: In terms of performance, GE-Act 2.0 showcases significant improvements in zero-shot out-of-distribution tasks, with substantial gains observed across almost all skill groups as training data scaled from 300 to 30,000 hours.

Evan: Overall, GE-Act 2.0 stands out as a robust and scalable framework for advancing robotic manipulation through efficient pretraining and innovative action alignment strategies.

Ashley: It's an exciting development and positions GE-Act 2.0 as a key player in future robotics research.

We look forward to seeing how it influences further advancements in this field.

Evan: That concludes today's episode of Daily Paper Cast.

Thank you for joining us as we explored the GE-Act 2.0 paper.

We hope you found it insightful.

Ashley: As always, we appreciate your continued support and interest.

Be sure to join us again for more deep dives into the latest AI and robotics research.

Evan: Until next time, stay curious and keep exploring.

Ashley: Goodbye for now!