Marionette: Predicting World States, Rendering Geometry, Painting Appearance

🤗 Upvotes: 23 | cs.CV, cs.AI

Authors:
Zian Meng, Zhen Li, Chuanhao Li, Qiang Li, Kaipeng Zhang

Title:
Marionette: Predicting World States, Rendering Geometry, Painting Appearance

Arxiv:
http://arxiv.org/abs/2608.14530v1

Abstract:
Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. We explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. We instantiate this idea as Marionette, a world model for interactive games with articulated characters. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls. Our experiments establish two properties of Marionette. First, the predicted world state is directly controllable. Forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behaviour is determined in the state, and can be repaired there. Left free, the two generated characters drift to 21.2 m apart (recorded sessions stay near 5 m) and a third of frames show ground penetration. Two rules imposed on the explicit state, a terrain collider and a separation cap, cut penetration by 66% and keep the pair engaged, with no change to the observation model. Routing appearance through the predicted state costs no fidelity we can detect, at an FVD of 831 against 799 for recorded pose.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast!

Evan: Today’s paper is from the Hugging Face daily paper list of August 17, 2026, and it’s already gathered 23 upvotes.

Ashley: The title of the paper is Marionette: Predicting World States, Rendering Geometry, Painting Appearance.

Evan: It’s authored by Zian Meng, Zhen Li, Chuanhao Li, Qiang Li, and Kaipeng Zhang.

Zhen Li and Zian Meng contributed equally, and Kaipeng Zhang is the corresponding author.

Ashley: They are affiliated with Alaya Lab, Shanghai Innovation Institute, and Huazhong University of Science and Technology.

Evan: Ashley, could you give us an overview of the problem this paper addresses?

Ashley: Evan.

Traditional interactive game world models typically autoregress visual observations directly in pixel or latent space.

This approach forces the model to maintain structured properties implicitly, like pose, geometry, and occlusion, which often leads to errors accumulating over long time horizons, making consistency and controllability fragile.

Evan: That sounds quite challenging.

What's the proposed solution in this paper?

Ashley: The authors propose a new approach by explicitly modeling the evolving world state.

They delegate exact geometric computation to a fixed, zero-parameter renderer and leave the neural model to synthesize the appearance.

Their system, Marionette, is designed specifically for interactive games with articulated characters.

Evan: Interesting!

How does Marionette achieve this?

Ashley: Marionette consists of three main components.

First, it has a two-stage autoregressive dynamics model that predicts an explicit and interpretable 276-dimensional three-dimensional world state.

This state includes multi-entity articulated skeletons, metric root trajectories, and rotations.

Evan: And what's the role of the zero-parameter graphics bridge?

Ashley: The zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form.

This is where exact geometric computations happen, ensuring everything aligns correctly in a physically plausible way.

Evan: Got it.

And how does the model handle the visual synthesis?

Ashley: The third component is a control-conditioned video-diffusion observation model.

This model synthesizes photorealistic RGB observations from the structured controls generated by the previous stages.

Essentially, it takes over the appearance generation, ensuring high-quality visuals.

Evan: So Marionette separates the generation task into structured state prediction and appearance synthesis.

What are the key findings from their experiments?

Ashley: The experiments highlight two main properties.

First, the predicted world state is directly controllable.

They found that forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments, demonstrating that the articulated dynamics effectively respond to the actions.

Evan: And how about the behavior over long horizons?

Ashley: For long horizons, the behavior is determined within the state itself and can be repaired there.

In an unconstrained situation, two generated characters might drift up to 21.2 meters apart.

By imposing rules like a terrain collider and a separation cap, the errors were significantly reduced, cutting ground penetration by 66% and keeping the characters engaged.

Evan: Does routing the appearance generation through the predicted state affect the visual fidelity?

Ashley: No, it doesn’t.

Their results showed that routing appearance through the predicted state did not incur a loss in fidelity that is detectable.

They reported an FVD, Fréchet Video Distance, of 831 against 799 for recorded pose.

Evan: That’s fascinating.

Using a structured world state seems to address the fragility in consistency and controllability that other models face over long sequences.

Ashley: Indeed, Evan.

This structured approach provides an explicit and controllable world representation while preserving high-quality visual generation, which is crucial for interactive games.

Evan: That wraps up the introduction to Marionette.

Evan: Let’s dive into the methodology behind Marionette.

Ashley, can you break down the structure of their approach?

Ashley: Sure, Evan.

Marionette’s approach is broken down into three core components.

First, they have a two-stage autoregressive dynamics model.

The second component is a zero-parameter deterministic graphics bridge, and the third is a control-conditioned video-diffusion observation model.

Evan: Alright, let's start with the two-stage dynamics model.

How does it work?

Ashley: The two-stage dynamics model is designed to predict future states in two steps: decision and animation.

The decision stage employs a compact causal transformer called ActionGPT to autoregressively predict one discrete action token per frame per entity.

Evan: What exactly does ActionGPT take into account for these predictions?

Ashley: ActionGPT is conditioned on a low-dimensional summary of recent state, which includes the root displacement, rotation stream, learned action embeddings, animation progress, and weapon sub-state.

It also regresses the next root motion and orientation.

Evan: So ActionGPT decides what actions the characters should take.

What happens next in the animation stage?

Ashley: Next, PoseGPT, a larger causal transformer model, maps the chosen action tokens and recent body state to the next-frame 258-dimensional body pose.

Essentially, PoseGPT translates these action tokens into articulated body poses for the game characters.

Evan: And this is all done autoregressively?

Ashley: Yes.

The process repeats itself step by step.

ActionGPT proposes actions and root motion at each step.

If necessary, these action tokens can be overridden.

PoseGPT then animates the body based on those actions, assembling the next 276-dimensional state.

Evan: How does Marionette ensure the state remains consistent with the environment, like terrain?

Ashley: Great question.

Marionette integrates terrain information into the dynamics model.

It uses an egocentric height patch rooted and aligned with the entity.

This patch along with per-joint ground clearances help ensure that predicted motions respect local ground surfaces.

Evan: So the terrain data actually influences the predicted actions and animations?

Ashley: Exactly.

It feeds this terrain data into both stages of the dynamics model.

This ensures that the model can predict feasible motions that interact correctly with the environment.

Evan: How does the deterministic graphics bridge fit into this framework?

Ashley: The deterministic graphics bridge tackles the conversion of the state into pose-control video.

This bridge has zero learnable parameters and ensures exact geometry within the scene.

For each frame, it calculates the 6D root rotations, integrates root motions into the world frame, and reconstructs metric world-space skeletons.

Evan: What happens after this skeleton reconstruction?

Ashley: The reconstructed skeletons are viewed with a camera, and the scene is rasterized into the pose-control frame that the observation model consumes.

This ensures that geometry, occlusion, and metric scale are accurately maintained by construction.

Evan: And this process is done entirely through geometric operations?

Ashley: Yes.

It uses closed-form geometric operations, resulting in exact control over world-space consistency.

The only potential for drift occurs if the predicted state itself diverges.

This bridge can also enforce feasibility by projecting infeasible root positions back onto the ground surface.

Evan: What role does the video-diffusion observation model play?

Ashley: The observation model maps the pose-control videos to photorealistic RGB frames.

It's built on a modern DiT video backbone and handles appearance generation.

For longer time horizons, it uses a chunk-relay rollout where generated frames from one chunk seed the next.

Evan: So, to summarize, this three-part system translates high-level action tokens into detailed, controllable animations that interact accurately with their environment, and then renders them photorealistically.

Can we now dive into how they evaluated their system’s performance?

Ashley: Definitely.

Their evaluation centers on two main questions: whether a control input has authority over the generated world and what determines long-horizon behavior.

Evan: Let's start with the first question.

How do they test control over the generated world?

Ashley: They test this by rolling out 48 held-out segments under three action stream conditions: Free, Force-GT, and Force-Shuf.

They measure the root-aligned mean per-joint position error, RA-MPJPE, against the recorded pose.

The results show that a mismatched stream increases pose error by 31%, showing that control inputs do indeed change what the body does.

Evan: And the second question, about long-horizon behavior?

Ashley: For long-horizon behavior, they fixed the observation model and varied the state driving it.

They compared the generated and recorded states on three metrics: ground penetration, foot-skate, and entity drift.

They found that ground penetration was significantly reduced with a terrain collider rule, and drift was controlled with a separation cap.

Meanwhile, their observation model maintained high visual quality with only minimal fidelity degradation.

Evan: It sounds like Marionette provides a robust framework for maintaining consistency over extended sequences, critical for interactive game worlds.

Ashley: By splitting the job between exact geometry handling and neural model-driven appearance, they ensure both controllability and high-quality visuals.

Evan: And that’s the end of the Method section.

Evan: So, how did Marionette perform in their experiments?

Can you share some insights, Ashley?

Ashley: Of course, Evan.

The experiments conducted focus mainly on two questions: whether a control input has authority over the generated world and how long-horizon behavior is determined and can be repaired.

Evan: Alright, let's start with the first question.

How did they test control inputs?

Ashley: To test control, the authors rolled out 48 held-out segments using different action streams — Free, Force-GT, and Force-Shuf.

They measured the Root-Aligned Mean Per-Joint Position Error, or RA-MPJPE, which quantifies pose accuracy excluding global translation.

Evan: And what did they find?

Ashley: They found that forcing a mismatched action stream increased pose error by 31%, from 0.272 to 0.357 meters, compared to the free-running model.

This demonstrates that the forced action tokens do have causal authority over the generated poses.

Interestingly, forcing the correct action tokens yielded a similar error to the free-run condition, 0.272 versus 0.281 meters, indicating that the model's own predictions are generally quite accurate.

Evan: That's compelling.

How did they examine long-horizon behavior?

Ashley: For long-horizon behavior, they fixed the observation model and manipulated how the state driving it was generated.

They evaluated different rules imposed on the state and compared the results using metrics like ground penetration, foot-skate, and entity drift.

Evan: Could you explain what these metrics mean and what the results showed?

Ashley: Certainly.

Ground penetration measures the extent to which any of the character’s joints go below the terrain surface, which should ideally be minimal.

Foot-skate detects horizontal foot movement when a foot is supposed to be in contact with the ground, indicating unnatural movement.

Entity drift measures how far apart characters like the hunter and monster move from each other over time, which can affect gameplay realism.

Evan: What were the main findings regarding these metrics?

Ashley: Without any corrective rules, the generated characters could drift up to 21.2 meters apart, significantly more than the recorded state, which was around 4.8 meters.

By applying a terrain collider rule, they reduced the ground-penetration collision-frame ratio from 0.337 to 0.114 and maintained the drift at 21.2 meters.

Adding a separation cap further reduced the drift to 5.1 meters, but at the expense of slightly increased foot-skate from 1.14 to 1.22 meters per second.

Evan: So, they were able to address these issues by imposing rules on the state itself?

Ashley: Exactly.

These imposed rules drastically improved stability and controllability without affecting the visual quality of the output.

According to the FVD or Fréchet Video Distance, all states landed close to the recorded state, with only minor differences.

This confirms that the visual fidelity remained high across various conditions.

Evan: What about their baseline comparisons?

How did Marionette perform against other models?

Ashley: They compared Marionette against an end-to-end pixel-autoregressive world model trained on the same footage.

Marionette outperformed the baseline across various metrics, achieving an FVD of 831, lower than the baseline’s 975, indicating better perceptual and temporal quality.

Evan: Did they provide any qualitative evaluations?

Ashley: Yes, qualitative evaluations showed that the commercial general-purpose video generators, while producing visually appealing outputs, lacked the structured control required in an interactive game setting.

The side-by-side comparison highlighted Marionette’s strength in providing entity-level control and accurate action execution.

Evan: So, it sounds like Marionette offers a robust solution for interactive game world modeling by providing structured, controllable outputs while maintaining high visual quality.

Ashley: Indeed.

The experiments strongly support Marionette’s approach of separating structured state prediction from appearance synthesis, validating their design choice.

Evan: And that brings us to the end of our discussion on the Experiment section of Marionette.

Evan: Now that we've covered the methodology and experiments, let's move on to the related work.

Ashley, can you give us an overview of the background and context provided in this section?

Ashley: Evan.

The related work section of Marionette is quite comprehensive.

It touches on various domains, including generative game world models, the separation of structure from appearance in video generation, and controllable articulated motion.

Evan: Let's start with generative game world models.

What did they discuss here?

Ashley: The authors point out that a large family of interactive world models generates appearance directly in pixel or latent space conditioned on user actions.

They recounted how these models range from Atari action-conditional prediction to GAN-based game simulators and beyond, including foundation-scale systems targeting open worlds and real-time streaming.

Evan: What are the limitations of these models?

Ashley: These systems often work well for rendering, dynamics, and control all in one sequence model but struggle with consistency and controllability over long horizons, as all structured properties are maintained implicitly within the network’s activations.

As a result, consistency and controllability are emergent properties rather than guaranteed.

Evan: So, Marionette's approach of separating these properties explicitly addresses these limitations?

Ashley: Exactly.

Marionette moves the exact bookkeeping out of the generator, making it more stable over long horizons by keeping the structured state explicit and leveraging deterministic components for exact computations.

Evan: Alright.

How about separating structure from appearance in video generation?

What did they highlight here?

Ashley: The authors discussed approaches that ground video generation in explicit 3D or memory, such as caching generated content and retrieving past context.

They also touched on specifying camera trajectories explicitly and how benchmarks probe the preservation of off-screen state and the response to actions.

Evan: So, there’s already work looking at splitting different aspects of video generation?

Ashley: Exactly.

However, these methods often derive 3D structures or consistency priors from pixels that have already been generated, while the forward dynamics stay within the video latent space.

Marionette, on the other hand, predicts structure before rendering, explicitly modeling the physical state as articulated entities.

Evan: That’s a key distinction.

What did they mention about controllable articulated motion?

Ashley: The paper references a few lines of research in character animation, like generating articulated motion under text and kinematic constraints using models that convert text descriptions into motion clips.

They also talk about models that drive characters through physics simulations and use combined text plus other constraints to generate human-like motion.

Evan: And how does Marionette fit into this context?

Ashley: Marionette's two-stage dynamics model isolates the problem of translating high-level control signals into detailed articulated motion.

It leverages existing advances in text-driven motion generation and physics-grounded controllers while focusing on how to structure and control the world state explicitly.

Evan: Any other notable areas they touch on?

Ashley: Yes, they also discuss various uses of world models.

For instance, in control-oriented settings where latent models predict short-horizon futures for agent planning, and in driving where they predict explicit 3D scene states from visual inputs.

However, Marionette's objective is consistency and controllability within an interactive and persistent game world.

Evan: So Marionette benefits from these existing research areas but focuses specifically on interactive games with articulated characters?

Ashley: Exactly.

By leveraging the strengths of explicit state modeling and combining them with deterministic rendering and high-quality neural appearance synthesis, Marionette aims to provide a robust solution for game world modeling that ensures both visual fidelity and interactive controllability.

Evan: And that’s the end of the Related Work section.

Evan: Alright, let's wrap things up by summarizing the key contributions and takeaways from today’s paper.

Ashley: Marionette represents a significant step forward in interactive game world modeling by explicitly separating the world state from appearance synthesis.

It achieves this through three main components: the two-stage autoregressive dynamics model, a zero-parameter deterministic graphics bridge, and a control-conditioned video-diffusion observation model.

Evan: Correct.

By predicting an explicit 276-dimensional 3D world state and using deterministic rendering to ensure geometric correctness, Marionette maintains stability and controllability over long sequences.

Ashley: Their experiments showed that the predicted state is directly controllable and that errors accumulated over long horizons can be repaired by imposing rules directly on the state.

Importantly, these controls did not degrade visual fidelity, demonstrating Marionette's ability to create high-quality, interactive game worlds.

Evan: In terms of practical application, Marionette outperformed end-to-end pixel-autoregressive models by providing more consistent and controllable outputs, essential for interactive gaming environments.

It also showcased how structured state prediction combined with high-quality neural appearance synthesis can preserve both consistency and visual detail.

Ashley: This structured approach leverages existing research in generative models, articulated motion, and rendering, but integrates them in a unique way that addresses key limitations faced by prior systems, particularly in maintaining long-horizon consistency and control.

Evan: That’s a thorough look at Marionette.

For those interested in learning more, you can find the full paper and additional resources on its project page linked in the show notes.

Ashley: Thank you for joining us today.

If you enjoyed this episode and want to stay updated on the latest in AI and machine learning research, be sure to subscribe to Daily Paper Cast.

Evan: We’ll be back with more insights on cutting-edge research papers.

Until then, stay curious and keep exploring!

Ashley: Goodbye and see you next time!