Programmable World Model

🤗 Upvotes: 32 | cs.CV

Authors:
Zheng-Hui Huang, Guixu Lin, Jiacheng Lin, Yi-Chuan Huang, Ruihan Yu, Muyao Niu, Siqi Yang, Yu-Lun Liu, Yung-Yu Chuang, Kaipeng Zhang, Zhixiang Wang

Title:
Programmable World Model

Arxiv:
http://arxiv.org/abs/2609.10540v1

Abstract:
Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome back to Daily Paper Cast.

Ashley: Today's paper is from the Hugging Face daily paper list of September 10, 2026, and it has 32 upvotes.

Evan: The title is 'Programmable World Model'.

Ashley: The first two authors are Zheng-Hui Huang and Guixu Lin.

The corresponding author is Zhixiang Wang from Alaya Lab.

Evan: Alright Ashley, let's dive right into the introduction.

What’s the background and main objective of this paper?

Ashley: Sure, Evan.

The paper addresses some of the current limitations in video world models, specifically their inability to maintain a persistent world state and implement programmable rules over longer interactions.

Evan: Interesting.

So what have they proposed to tackle these issues?

Ashley: They introduce something called the 'Programmable World Model'.

This framework decouples the evolution of the world state from the generation of visual observations.

Essentially, an agent converts natural-language instructions into executable programs that define entity states and interaction rules.

Evan: So how does that help in terms of maintaining the world state and programmability?

Ashley: By breaking these two tasks apart, they ensure that the global world state remains explicit and persistent.

This state not only includes visible elements but also off-screen entities and non-visual attributes, which are managed by a lightweight execution engine.

Evan: And how do they handle the visual generation part then?

Ashley: For visual generation, they use a state-augmented 3D oriented bounding box, or OBB, as an intermediate representation.

The engine compiles these OBBs into pixel-aligned spatiotemporal conditioning signals, which are then used by a pretrained video model to render the visuals.

This approach ensures that the visuals stay consistent with the world state.

Evan: I see.

And what are the main contributions of this paper?

Ashley: The paper claims three main contributions.

Firstly, they introduce the Programmable World Model framework, which allows entity-level control, persistent state management, and user-programmable world rules.

Secondly, they develop a data curation pipeline that extracts spatial and semantic supervision from videos, making it practical to scale training data for such generative worlds.

Lastly, they propose CombatStateBench, a controlled benchmark to evaluate the consistency between the generated videos and the engine-maintained world states, specifically focusing on character counts and state realization under various conditions.

Evan: CombatStateBench certainly sounds like an interesting benchmark.

And it seems they achieve very high accuracy rates with their new framework.

Can you elaborate on that?

Ashley: Indeed.

On CombatStateBench, their method achieved 94% Count Accuracy and 98% State Accuracy.

These numbers significantly outperform existing interactive video world models, showing that their approach effectively separates state evolution from visual generation and supports coherent long-horizon generation.

Evan: That’s impressive.

So to sum it up, the Programmable World Model decouples state evolution from visual rendering, maintains persistent global states, uses OBBs for intermediate representation, and has shown high performance on the CombatStateBench benchmark?

Ashley: Exactly, Evan.

This wraps up the Introduction section of the paper.

Evan: Alright Ashley, now let's move into the core of today's paper.

Can you unpack the methodology behind the Programmable World Model?

Ashley: Certainly, Evan.

The methodology in this paper is structured into three primary components: World Programming, Control Compilation, and Generative Rendering.

Each of these components plays a crucial role in enabling the framework to maintain persistent world states while generating consistent visuals.

Evan: Let's start with World Programming.

How does that work?

Ashley: World Programming essentially involves defining the initial state of the world and the rules that govern how it evolves.

This is done through an agent-orchestrated process where natural-language specifications from the user are translated into executable programs.

These programs specify entity states, relationships, and interaction rules.

For instance, it could define how a character’s health changes when they take damage or what conditions lead to the end of a game session.

Evan: So, these programs are then executed by the framework's engine?

Ashley: Exactly.

The engine maintains and updates the world state by utilizing these programs.

It handles player interactions and updates entity states based on predefined rules.

The key idea is that the world state is persistent and explicit, meaning that it is not reliant on continuous visual observations but is maintained structurally within the engine.

Evan: Interesting.

So what's next in the sequence?

Ashley: The next step is Control Compilation.

This involves transforming the updated world states into a form that can be used by the generative renderer to create visuals.

Specifically, the state compiler projects state-augmented 3D oriented bounding boxes, or OBBs, into the camera’s coordinate system, producing what they call 'spatial controls'.

These controls include identity, semantics, and motion direction attributes for each entity.

Evan: And how are these spatial controls used?

Ashley: These spatial controls are crucial because they provide a clear and structured way to align the actual world state with the generated visuals.

The identity map ensures that entities are consistently represented across frames, the semantic map gives information about the categories of entities, and the direction map details how these entities move relative to the camera.

All of these factors collectively guide the renderer to generate consistent and accurate visuals.

Evan: Got it.

Now, how does the Generative Rendering part come into play?

Ashley: In Generative Rendering, they use a pretrained video generation model called LingBot-World-v1.

This model is conditioned on the camera trajectory and the structured spatial controls generated from the state compiler.

They introduced a Structured Spatial ControlNet, which processes these controls, and injects control features into the main video generation model.

Evan: That's a lot to integrate.

How do they ensure long-horizon rendering remains consistent?

Ashley: They extend the renderer in a chunk-autoregressive manner to support long-horizon generation.

Essentially, the video is divided into chunks, and information from completed chunks is propagated to generate subsequent chunks.

Notably, they maintain both a temporal history and a geometry-aligned spatial memory.

The temporal history retains recent latent context, while the spatial memory uses estimated depth and camera parameters to preserve the visual consistency across longer sequences.

Evan: And how do they handle the training data for such a complex system?

Ashley: Good question.

They developed an automatic data engine to extract the structured annotations necessary for training.

This engine recovers camera parameters, semantic labels, instance tracks, and object trajectories from unlabeled videos, constructing the camera-aligned conditioning maps required for training the system.

Evan: Impressive.

How do they evaluate the performance of their framework?

Ashley: They introduced a specific benchmark called CombatStateBench.

This benchmark is used to evaluate whether the generated videos are consistent with the underlying world state maintained by the engine.

The evaluation focuses on metrics such as Count Accuracy and State Accuracy.

Count Accuracy measures if the number of visible characters in generated frames matches the engine’s records, while State Accuracy checks if state changes, like character deaths, are visually realized in the generated video.

Evan: And how does their method perform on this benchmark?

Ashley: Their method achieves high performance, with a 94% Count Accuracy and a 98% State Accuracy.

This indicates that the framework reliably maintains visual and state consistency, even over longer interaction sequences.

Evan: So to recap the methodology, the paper introduces World Programming to handle the state and rules, Control Compilation to translate these states into spatial controls, and Generative Rendering to produce the visual output.

All of this is supported by a robust evaluation framework in CombatStateBench.

This wraps up the Method section of the paper.

Evan: Alright Ashley, let’s dive into the experiments and results section.

How do they set up their experiments to evaluate the Programmable World Model?

Ashley: The authors set up their experiments by collecting HUD-free gameplay videos from Cyberpunk 2077, Forza Horizon 6, and Grand Theft Auto V.

These videos were processed by their data engine to construct paired video-control training data.

This setup allows them to evaluate how well their model can generate consistent and realistic game environments.

Evan: So, walk us through their primary evaluation benchmark, CombatStateBench.

Ashley: CombatStateBench is designed to evaluate the consistency between generated videos and the engine-maintained world state.

It comprises 50 clips featuring diverse combinations of camera and entity motions, including interactions where entities initially lie outside the camera view.

The benchmark assesses both Count Accuracy and State Accuracy, measuring if the generated videos accurately reflect the number of visible alive characters and the visual realization of state transitions like character deaths.

Evan: And how did their model perform on this benchmark?

Ashley: Their model performed exceptionally well, achieving a 94% Count Accuracy and a 98% State Accuracy, which substantially outperforms existing interactive video world models.

This demonstrates the framework’s capacity to maintain a reliable world state and generate coherent long-duration sequences.

Evan: Those are impressive numbers!

How do they compare to other models in the benchmarks?

Ashley: The authors compared their method with two representative interactive video world models: LingBot-World-V2 and YUME.

On the CombatStateBench, their model significantly outperformed both, with improvements in visual fidelity and state consistency.

For example, while LingBot-World-V2 scored 40.75% in State Accuracy and YUME scored 58.00%, their method reached a remarkable 98.00%.

Evan: That’s a huge gap.

What about other evaluation metrics?

Ashley: They used several metrics from the VBench suite to assess perceptual and temporal quality.

Specifically, they measured Imaging Quality, Subject Consistency, Background Consistency, and Temporal Stability.

Their method outperformed both LingBot-World-V2 and YUME in all these metrics, achieving the best overall performance.

Evan: Can you give us a closer look at the qualitative results?

Ashley: Of course.

The authors provided qualitative comparisons showing entity-death interactions and longer sequences under different conditions.

Their method reliably preserves entity counts and state transitions, even under dynamic camera and entity motions.

They also tested their model on scenes unseen during training, including novel environments and racing scenarios, demonstrating the framework's adaptability and robustness.

Evan: What are some key highlights from these qualitative tests?

Ashley: One significant highlight is their model’s ability to handle large-angle camera rotations while keeping the scene consistent.

It can correctly reveal characters and entities initially placed behind the first-frame view as the camera moves, maintaining the integrity of the scene.

Another example shows the framework supporting a racing game scenario with complex camera and vehicle motions.

The generated videos closely followed the specified camera motion, spatial layout, and entity states.

Evan: It sounds like their framework can potentially generalize across various types of games and scenarios.

Anything else notable in the results?

Ashley: Yes, they also demonstrated the joint rendering of heterogeneous object categories, such as humans and vehicles, within the same scene.

Even in longer sequences where multiple non-playable characters enter the scene progressively, the model maintained visual and temporal stability, suggesting it can support more complex and evolving worlds.

Evan: To wrap up, the experiments and results show that the Programmable World Model excels in maintaining world-state consistency and visual accuracy, significantly outperforming existing models.

This wraps up the Experiment section of the paper.

Evan: Now, Ashley, let's delve into the Related Work section.

How does this paper fit into the context of existing research?

Ashley: Glad you asked, Evan.

The authors categorize their related work into three primary areas: interactive video world models, explicit-state world modeling, and generative rendering.

Evan: Let's start with interactive video world models.

What's the background there?

Ashley: Interactive video world models have made significant progress in creating realistic and responsive visual environments. These models, such as those referenced in works [2, 16, 17, 18, 20], predict follow-up observations based on visual histories and user actions to synthesize immersive environments. They aim to provide action-conditioned generation and controllable camera motion, but they often lack a structured, persistent world state.

Evan: And why is maintaining a persistent world state so critical?

Ashley: A persistent world state is essential for coherent long-horizon interactions.

Without it, you risk inconsistencies across scenes, especially when dealing with occlusion, camera motion, or extended sequences.

The current models primarily optimize for visual plausibility rather than maintaining an executable world state.

This paper separates the tasks of state maintenance and visual generation to address this gap.

Evan: Got it.

So what about explicit-state world modeling?

How does it differ?

Ashley: Explicit-state world modeling involves explicitly representing the internal states of interactive environments. For instance, StatePlay [14] predicts both visual observations and game-state variables to guide visual generation and ensure consistency in game mechanics. However, these models often suffer from state prediction errors, which can accumulate over time. Some systems, like MASS [3], introduce an authoritative shared state for multiplayer scenarios, but still depend on learned transition dynamics that could lead to errors.

Evan: So the main issue with these systems is their reliance on predicted states rather than maintaining an explicit state?

Ashley: Precisely.

The framework in this paper aims to overcome these issues by keeping an explicit and verifiable canonical world state, executed by a lightweight engine based on clear, programmable rules.

This approach minimizes the errors associated with state predictions over long interactions.

Evan: And finally, generative rendering.

How does this paper build on what's been done in that area?

Ashley: Generative rendering leverages learned models to produce photorealistic visuals from structured representations. Traditional approaches, like DiffusionRenderer [12], use high-dimensional geometric datasets to generate forward and inverse renderings. The AlayaRenderer series [11, 13] pushes this further by rendering dynamic worlds efficiently with real-time capabilities. However, these methods still require rich structural supervision and complex inference mechanisms.

Evan: How does the Programmable World Model differ in its approach?

Ashley: The Programmable World Model provides a more streamlined approach by utilizing state-augmented 3D oriented bounding boxes as an intermediate representation.

This reduces the need for detailed geometry while still allowing for accurate visual rendering.

Instead of relying solely on generative priors, the structured controls and state compilers in their framework offer a balanced way to guide the generative model, ensuring spatial-temporal consistency and reducing the computational burden.

Evan: So they achieve a balance between explicit control and generative flexibility?

Ashley: Exactly.

This balance allows their model to be both flexible in generating rich visual content and reliable in maintaining a consistent world state, a combination that addresses the shortcomings of previous methods in these related areas.

Evan: Got it.

Anything else notable from the related work section worth mentioning?

Ashley: They also talk about various other interactive world models and rendering techniques, but those are the main points.

Overall, the Programmable World Model stands out by effectively combining the strengths of these approaches while mitigating their individual limitations.

Evan: That’s really insightful.

This wraps up the Related Work section of the paper.

Evan: Alright Ashley, let’s summarize the key contributions and takeaways from this paper.

Ashley: Evan.

The key contributions of the Programmable World Model are quite impactful.

Firstly, the framework decouples world-state evolution from visual observation generation.

By doing so, it ensures entity-level control, persistent state management, and user-programmable world rules.

Evan: And that’s achieved through their innovative use of state-augmented 3D oriented bounding boxes, right?

Ashley: Exactly.

These OBBs act as an intermediate representation that facilitates the connection between maintaining a consistent world state and generating accurate visuals.

Evan: The CombatStateBench benchmark also seems like a significant contribution.

What can you tell us about that?

Ashley: CombatStateBench is critical for evaluating the consistency between the generated videos and the underlying world state, especially in terms of visible alive-character counts and the visual realization of state changes like character deaths.

Their method showed notable improvements, achieving 94% Count Accuracy and 98% State Accuracy.

Evan: So overall, the paper provides a robust framework that addresses current limitations in video world models by ensuring persistent state management and effective visual rendering through a novel use of OBBs and a deterministic state compiler.

Ashley: Precisely.

It’s a significant step forward in building interactive, persistent, and programmable virtual environments that can reliably generate long-horizon experiences.

Evan: Well, that wraps up today's episode of Daily Paper Cast.

Thank you all for tuning in.

Ashley: We hope you found today's discussion insightful.

Don't forget to join us again for our next episode, where we’ll dive into another fascinating paper from the world of AI and machine learning.

Evan: Until next time, keep exploring and stay curious!