Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction

🤗 Upvotes: 36 | cs.CV, cs.AI

Authors:
Dahyun Chung, Siyoon Jin, Hyunwook Choi, Honggyu An, Junyoung Seo, Hyunsung Kim, Seung Wook Kim, Seungryong Kim

Title:
Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction

Arxiv:
http://arxiv.org/abs/2610.12299v1

Abstract:
Egocentric world models predict first-person observations conditioned on an agent's actions, but most focus on a single agent. Real embodied settings often involve multiple agents that act and interact within a shared environment. Existing multi-agent world models rely on coarse actions like locomotion, camera control, or discrete commands, leaving fine-grained embodied interactions underexplored. We formulate multi-agent egocentric world modeling as synchronized ego-stream generation for multiple agents interacting through fine-grained actions in a shared world. This requires cross-view action consistency, shared-environment consistency, and consistent propagation of interaction-induced state updates. We propose Multi-agent Egocentric World Model (ME-World), which jointly denoises multiple ego streams in a shared token sequence, conditions each stream on all agents' target-view poses, and grounds generation with shared environment memory. We train and evaluate on real and synthetic multi-agent data and introduce shared-world consistency metrics for environment, update, and identity consistency. Experiments show ME-World improves shared-world consistency, action control, identity preservation, and video quality over existing methods.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Ashley: Today's paper is from the Hugging Face daily paper list of October 9, 2026, with 36 upvotes.

Evan: The title is 'Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction.'

Ashley: First two authors are Dahyun Chung and Siyoon Jin, and the co-corresponding author is Seung Wook Kim from KAIST AI.

Evan: Let's dive into the Introduction.

Ashley: Egocentric world models predict first-person observations conditioned on an agent’s actions, primarily focusing on single agents.

However, real-world applications often involve multiple agents interacting in a shared environment.

Evan: So, essentially, it's moving from single to multi-agent scenarios?

Ashley: Yes, exactly.

Existing multi-agent world models typically rely on broad actions like locomotion, camera control, or discrete commands.

This leaves out the more detailed, embodied interactions.

Evan: How does this paper propose to deal with these detailed interactions?

Ashley: The authors propose a new approach termed Multi-Agent Egocentric World Model, or ME-World.

This model aims to generate synchronized ego streams for multiple agents interacting through fine-grained actions in a shared world.

Evan: Can you break down what 'synchronized ego streams' mean in this context?

Ashley: Sure.

It means that each agent’s view should be consistent with the others, maintaining character identity, scene appearance, and interaction outcomes across all perspectives.

Evan: Got it.

So, how does ME-World achieve this synchronization?

Ashley: ME-World uses a combination of techniques.

First, it denoises multiple ego streams jointly within a shared token sequence.

Second, it conditions each stream on all agents’ target-view poses.

Lastly, it grounds the generation with shared environment memory.

Evan: And how are these techniques beneficial for the consistency of interactions?

Ashley: These techniques ensure cross-view action consistency, environmental consistency, and consistent propagation of interaction-induced state updates.

To evaluate whether their approach works, they introduce shared-world consistency metrics.

Evan: What are these shared-world consistency metrics?

Ashley: These metrics include environment consistency, update consistency, and identity consistency.

For instance, environment consistency checks if the environment looks the same across all ego streams, while update consistency ensures that interaction-induced changes are reflected correctly across all views.

Evan: So, ME-World not only improves shared-world consistency but also enhances action control and identity preservation.

Interesting.

Ashley: Yes, the experiments showcased in the paper indicate that ME-World outperforms existing methods on both real and synthetic multi-agent datasets.

It shows better shared-world consistency, action control, identity preservation, and overall video quality.

Evan: So to recap, the introduction clearly outlines the challenges in multi-agent egocentric world modeling and presents ME-World as a robust solution to synchronize multiple ego streams and maintain consistency across interactions.

Ashley: Yes, and that's the end of the Introduction section.

Evan: Let's now move on to the Method section, where the authors detail how they developed their proposed model, ME-World.

Ashley: Sure.

The core idea of ME-World is to generate multiple synchronized ego streams for agents interacting in a shared environment.

To achieve this, the model uses three main components: joint multi-agent generation, shared action conditioning, and shared environment memory.

Evan: Let's break these components down one by one.

Can you start with joint multi-agent generation?

Ashley: Joint multi-agent generation means that instead of generating each agent's ego stream independently, ME-World generates all streams together.

They use a video diffusion transformer (DiT) model to achieve this.

Evan: Okay, and how does the DiT handle multiple streams simultaneously?

Ashley: The DiT processes joint sequences with self-attention, allowing each agent's stream to inform the others.

This way, all streams are treated as mutually informed observations of the same shared world.

Evan: Interesting approach.

What about identifying which stream each token belongs to?

Ashley: For that, they use rotary positional embeddings and a learnable agent embedding for each agent.

This helps the model distinguish which stream each token belongs to and which conditions it should follow.

Evan: Got it.

Now, let's move on to shared action conditioning.

How does that work?

Ashley: Shared action conditioning ensures that each agent's actions are visible from other agents' viewpoints.

For every frame, each agent's body and head motion are included in the target agent's pose condition sequence.

Evan: And how do they encode these motions?

Ashley: The body motion is encoded as a self-action skeleton map, while head motion is treated as an action in egocentric generation, represented with Plücker ray maps.

This approach provides a geometric encoding of camera rays induced by head movements.

Evan: Great.

So the pose condition sequence includes both the agent's own motion and the motions of others?

Ashley: Exactly.

By integrating these actions into the target agent's pose conditions, the model explicitly includes cross-agent motion cues, which are crucial for maintaining consistent embodied interactions.

Evan: Let's talk about the third component: shared environment memory.

Ashley: Shared environment memory provides consistent scene information by aggregating observation histories of all agents.

It uses stream-aligned geometric memory and cross-stream anchor memory.

Evan: Can you explain how these memory paths work?

Ashley: Stream-aligned geometric memory ensures that the observations are geometrically aligned with the target view, providing direct visual evidence for each generated frame.

On the other hand, cross-stream anchor memory selects reference images for their complementary scene coverage, helping to maintain the global appearance.

Evan: And how are these incorporated into the generation process?

Ashley: The reference images and geometric memory are both encoded and concatenated with the noisy latents during the denoising process, ensuring all streams can attend to the same anchors and geometric evidence.

Evan: So it sounds like shared environment memory helps synchronize the appearance and state of the environment across all streams?

Ashley: Exactly.

It helps the model to avoid inconsistencies in the scene layout, object states, and overall environment, ensuring all agents share the same evolving world.

Evan: This sounds very comprehensive.

How does the training process work for this model?

Ashley: ME-World is trained using a synchronized dataset consisting of both real and synthetic multi-agent data.

The model uses a rectified flow objective, which helps in processing and denoising the latent sequences.

Evan: And what kind of datasets were used for the training?

Ashley: They combined the real CoMind dataset with synthesized multi-agent interactions.

The synthetic dataset was generated by applying motion data to avatars in various scenes and rendering synchronized ego streams.

Evan: That should provide a good variety of interaction scenarios.

What about evaluation?

Ashley: For evaluation, they introduced shared-world consistency metrics like environment consistency, update consistency, and identity consistency.

Additionally, they used existing metrics for video quality and action control.

Evan: So the Method section provides a detailed explanation of how ME-World generates synchronized multi-agent ego streams using joint generation, shared action conditioning, and shared environment memory.

It also covers how the model is trained and evaluated using both real and synthetic data.

Ashley: Yes, and that wraps up the Method section of the paper.

Evan: Let's now delve into the Experiment and Results section, where the authors validate the performance of ME-World against existing methodologies.

Ashley: The authors ran extensive evaluations on both real and synthetic multi-agent datasets.

For real-world testing, they curated synchronized two-ego pairs from the CoMind dataset, which involves collaborative kitchen activities captured from multiple perspectives.

Evan: And what about the synthetic dataset?

How was that constructed?

Ashley: The synthetic dataset was generated by retargeting paired motion sequences to diverse 3D avatars and placing them in various shared environments.

They rendered synchronized ego streams for these avatars, providing complementary scene coverage and detailed motion data.

Evan: What metrics did they use to evaluate the models?

Ashley: They proposed new metrics specifically for shared-world consistency: environment consistency, update consistency, and identity consistency.

They also used standard metrics to assess video quality and action control, such as PSNR, SSIM, LPIPS, FVD, Hand-F1, Hand-mIoU, and Full-Body-PCK.

Evan: How did ME-World perform compared to other methods?

Ashley: ME-World outperformed the methods across multiple criteria.

For example, in shared-world consistency, ME-World achieved higher environment consistency and state update consistency compared to models like GEN3C and AnyView-DVS.

For identity consistency on synthetic data, ME-World also showed marked improvements.

Evan: Do you have specific figures for these improvements?

Ashley: Yes, on the real dataset, ME-World achieved an environment consistency score of 0.468 and a state update consistency score of 0.466, significantly higher than GEN3C’s 0.417 and 0.412, respectively.

On the synthetic dataset, ME-World’s identity consistency score was 0.561 versus 0.433 for GEN3C.

Evan: What about the video quality metrics?

How does ME-World fare in these areas?

Ashley: ME-World also excels in video quality metrics.

For instance, on the real benchmark, it scored 19.849 for PSNR, 0.656 for SSIM, and 637.9 for FVD, which are higher than all competing models.

The visual fidelity and perceptual quality were significantly better.

Evan: So, this model could very well be a game-changer for applications requiring high-fidelity video generation combined with multi-agent consistency.

Ashley: Precisely.

The improved shared-world consistency means more reliable interaction modeling, while the enhanced video quality metrics signal better overall user experience, making ME-World valuable in VR simulations, AI-driven collaborative tasks, and multi-user gaming environments.

Evan: I noticed there were also qualitative comparisons made.

What did they look like?

Ashley: The qualitative comparisons showed stark differences.

ME-World maintained consistent environment and interaction outcomes across all agent views, whereas others often generated conflicting or visually inconsistent streams.

Figures in the paper display synchronized ego streams across timestamps, highlighting how ME-World manages coherency even in complex scenes.

Evan: That's impressive.

How were the ablation studies scoped in the paper?

Ashley: The authors conducted component ablation studies to evaluate the impact of each of ME-World's core components.

Removing any of these components, like joint multi-agent generation or shared action conditioning, led to notable declines in shared-world consistency metrics and video quality.

This underscores the necessity of each part in ME-World’s framework.

Evan: So overall, the experimental results robustly demonstrate that ME-World brings substantial improvements in multi-agent egocentric video generation.

That's the end of the Experiment section.

Evan: Next, let's explore the Related Work section to understand the backdrop of research that ME-World builds upon.

Ashley: To start, the paper classifies related work into two major categories: Egocentric World Models and Multi-Agent World Models.

Evan: Okay, let's begin with Egocentric World Models.

What do we need to know about them?

Ashley: Egocentric world models generate future first-person observations based on an agent’s actions.

This line of work has various applications, including autonomous driving, robotics, open-domain simulations, and video games.

Evan: So these models help an agent predict how a scene will evolve from its viewpoint, right?

Ashley: Exactly.

They enable agents to imagine future scenarios without acting in the real world.

Earlier research like work by Ha and Schmidhuber (2018) paved the way for many such advancements within this field.

Evan: How have these egocentric models evolved?

Ashley: Recent work has explored various dimensions such as whole-body motion, hand and camera control, joint body motion, and dexterous manipulation.

Examples include the works by Bai et al., Tu et al., and Pallotta et al., among others.

Evan: And what about human-world interactions?

How are they being modeled?

Ashley: They're modeled through explicit 3D memory systems, goal-directed world evolution, and manipulation tasks.

Researchers like Kim et al., Goswami et al., and Hao et al.

have significantly contributed to these aspects.

Evan: It seems there's a lot of nuance in how egocentric world models operate.

Let's move on to the second category: Multi-Agent World Models.

Ashley: Multi-agent world models extend the concept to multiple agents interacting in a shared environment.

This typically involves coupling actions and observations across different agents through a common world state.

Evan: How do these models maintain coherence in a multi-agent setup?

Ashley: Maintaining coherence involves strategies such as joint cross-agent attention, shared scene representations, and explicit memory states.

This area has seen developments in multiplayer game settings, robotics, driving simulations, and general open-domain videos.

Evan: Can you mention some notable contributions in this field?

Ashley: Sure.

Significant works include those by Savva et al.

in the context of multiplayer games, Wu et al.

focusing on consistent video stream generation, and Liu et al.

who look into robot control in manipulation tasks.

Evan: What are the main challenges addressed by these multi-agent models?

Ashley: The core challenges include ensuring synchronized cross-view actions, maintaining a consistent shared environment, and accurately reflecting interaction-induced state changes across all agent observations.

Evan: How does ME-World advance these established lines of research?

Ashley: ME-World progresses these lines by offering a robust framework for generating synchronized ego streams as coupled observations of a shared environment.

It integrates joint generation, shared action conditioning, and shared environment memory, addressing specific nuances of embodied interactions in a multi-agent context.

Evan: So ME-World doesn't just build on previous work but addresses their gaps, particularly in fine-grained actions and consistency across multiple agents.

Ashley: Precisely.

By combining the strengths of existing egocentric and multi-agent models, ME-World ensures nuanced, coherent interactions and smooth synchronization across diverse agent views.

Evan: That wraps up the Related Work section.

Evan: We've covered the introduction, methodology, experiments, and related work of the ME-World paper.

Now, let's summarize the key contributions and takeaways.

Ashley: One of the major contributions of the paper is the introduction of ME-World, a new model framework for multi-agent egocentric world modeling.

It focuses on generating synchronized ego streams for multiple agents interacting through fine-grained actions in a shared environment.

Evan: Right.

ME-World stands out by addressing three critical aspects: joint multi-agent generation, shared action conditioning, and shared environment memory.

These components ensure consistency in cross-view actions, environmental layout, and interaction-induced state updates.

Ashley: Exactly.

The joint generation approach means all agents’ ego streams are created together, keeping them in sync.

Shared action conditioning allows each agent to consider the motions of others, and shared environment memory helps maintain a cohesive scene across different views.

Evan: The innovative shared-world consistency metrics introduced in this paper, like environment consistency, update consistency, and identity consistency, provide a robust framework for evaluating multi-agent interactions.

Ashley: Yes, and the experimental results clearly highlight how ME-World outperforms existing models with significant improvements in these metrics, as well as in video quality and action control.

Evan: The practical implications are broad, impacting areas like VR simulations, AI-driven collaborative tasks, and multi-user gaming environments, where synchronized interactions are crucial.

Ashley: Precisely.

This paper pushes the boundaries in egocentric and multi-agent world modeling by offering a comprehensive solution to synchronize multi-agent streams and maintain interaction consistency.

Evan: And that brings us to the end of today's episode.

Thank you for tuning in to Daily Paper Cast.

Ashley: We hope you found the discussion insightful.

Be sure to join us again tomorrow for more deep dives into cutting-edge research in AI, NLP, CV, and related areas.

Evan: Until next time, keep exploring and stay curious!

Ashley: Goodbye for now!

Evan: Goodbye!