Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

🤗 Upvotes: 29 | cs.CV

Authors:
Kang Liao, Yihang Luo, Xiao-Ming Wu, Linyi Jin, Size Wu, Chunyu Lin, Yao Zhao, Fei Wang, Wei Li, Chen Change Loy

Title:
Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

Arxiv:
http://arxiv.org/abs/2609.04196v1

Abstract:
We propose Puffin-World, a unified multimodal architecture that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules. To reliably construct and interact with 3D worlds, our framework jointly models three native world states: physics (gravity field and latitude), geometry (depth), and appearance (image), together with a unified Omni-Camera representation that supports diverse tasks and flexible motions. Beyond modeling these states, we introduce a strategy for propagating physical dynamics across future frames. By grounding absolute camera properties in the real world, Puffin-World enables physically consistent and visually stable world generation. We further couple appearance and geometry within a single generative process, jointly synthesizing each future view and reconstructing its underlying geometry. This unified paradigm enables interleaved closed-loop applications requiring synergy across multiple tasks, including mimic and self-calibrated world exploration. To scale Puffin-World to complex scenarios, we construct Puffin-16M, comprising 15 million vision-language-camera triplets and 1 million trajectories featuring various and challenging motions. To foster further research in this area, we released the code, models, and datasets.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast!

Ashley: Today we're diving into a paper from the Hugging Face daily paper list of September 4, 2026, which has garnered 29 upvotes.

Evan: The paper we're exploring is titled 'Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States.'

Ashley: The first authors of this paper are Kang Liao and Yihang Luo, with correspondence from Size Wu, all from Nanyang Technological University.

Evan: Let's dive straight into the core of this paper.

Ashley, can you start us off with the background and objectives?

Ashley: Of course.

The motivation behind Puffin-World comes from the central goal of multimodal spatial intelligence and physical AI, which is to develop a model capable of perceiving, generating, and reconstructing the world from arbitrary visual observations.

Evan: So, it's not just about generating images, but understanding the camera's position and the scene's geometry as well?

Ashley: Exactly.

A capable world model should do more than generate plausible images.

It should know where the camera or viewpoint is located in the real world, reason about the scene's geometry, and synthesize what the world looks like from new viewpoints in a manner consistent with physical laws.

Evan: That sounds ambitious.

How does Puffin-World aim to achieve this?

Ashley: Puffin-World integrates physical understanding, spatial simulation, and 3D world generation and reconstruction into a unified framework without relying on external offline modules.

It models three native world states: physics, geometry, and appearance, using a unified Omni-Camera representation that supports diverse tasks and flexible motions.

Evan: Three native world states—can you explain more about what these entail?

Ashley: Yes, the three states are physics, which includes gravity and latitude, geometry, which deals with depth, and appearance, which relates to the visual content of the images.

The physics state grounds an observation in the absolute physical world, the geometry state describes the underlying 3D spatial structure, and the appearance state represents what we see in images and sequences.

Evan: Got it.

And this Omni-Camera representation—how does it function within the model?

Ashley: The Omni-Camera combines gravity-anchored absolute field with a relative ray field to support continuous single-view and cross-view control.

This helps Puffin-World simulate physical-grounded spatial simulation and 3D scene generation while maintaining consistency with real-world physical constraints.

Evan: And beyond modeling these states, what novel strategies does the paper introduce?

Ashley: The paper introduces a strategy for propagating physical dynamics across future frames, ensuring physically consistent and visually stable world generation.

This involves coupling appearance and geometry within a single generative process to jointly synthesize each future view and reconstruct its underlying geometry.

Evan: Interesting.

So essentially, Puffin-World doesn't just generate isolated frames, but an integrated sequence that remains consistent over time and space?

Ashley: Exactly.

This unified approach enables interleaved closed-loop applications requiring multiple tasks to work synergistically, such as mimic and self-calibrated world exploration.

It ensures that the generated world aligns with physical laws, providing reliable and stable simulations.

Evan: To scale to complex scenarios, what datasets or benchmarks does Puffin-World employ?

Ashley: The team constructed Puffin-16M, which includes 15 million vision–language–camera triplets and 1 million trajectories featuring diverse and challenging motions.

This dataset provides extensive supervision for scaling the model's capabilities in handling complex and dynamic scenes.

Evan: That's a significant dataset.

Any closing thoughts on the major contributions of this paper?

Ashley: Yes, the main contributions include proposing Puffin-World as a unified multimodal model that perceives, generates, and reconstructs 3D worlds; introducing the Omni-Camera representation and physics propagation mechanism; constructing the large-scale Puffin-16M dataset; and open-sourcing their code, models, and datasets to foster further research and development in this area.

Evan: Excellent summary, Ashley.

That wraps up the Introduction section.

Evan: Alright Ashley, let's get into the core of Puffin-World's architecture.

Can you walk us through the methods they proposed?

Ashley: Puffin-World’s approach is built on the foundation of modeling three native world states: physics, geometry, and appearance.

Let's start with 3D native world states.

Evan: Sure, what exactly are 3D native world states and how are they represented in Puffin-World?

Ashley: Most existing models focus primarily on appearance-level prediction, using RGB images or videos.

In contrast, Puffin-World models multiple levels: physics for global physical grounding, geometry to describe 3D spatial structure, and appearance to represent observable visual content.

This joint modeling is crucial for generating stable, spatially consistent simulations and interactions.

Evan: Got it.

And how does Puffin-World unify camera representations?

Ashley: The camera acts as a core interface between 3D and 2D data.

Puffin-World uses the Omni-Camera representation, combining absolute physical grounding with continuous spatial modeling.

This includes an absolute camera field and a relative ray field, ensuring both absolute orientation and cross-view spatial relationships are maintained.

Evan: Explain more about the Omni-Camera representation.

How does it work?

Ashley: The Omni-Camera representation integrates gravity-aware absolute orientation with ray-based relative geometry.

It merges the absolute up-vector and latitude angle with the relative ray map, which includes the ray origin and direction.

This allows detailed physical grounding and supports versatile applications like physical-world understanding and continuous scene generation.

Evan: That's interesting.

Moving onto the main framework of Puffin-World, how does it actually function?

Ashley: Puffin-World unifies three core functions: physical-world perception, free-viewpoint spatial simulation, and 3D world generation and reconstruction.

Let's break this down: representation unification, modality unification, and task unification.

Evan: Alright, let's start with representation unification.

What does that involve?

Ashley: Representation unification involves sharing a common latent space for appearance and geometry.

For instance, it uses RGB images and depth maps encoded by the same Variational Autoencoder, while all camera configurations, like rotations and translations, are represented using the Omni-Camera framework.

Evan: And what about modality unification?

Ashley: For modality unification, a single backbone perceives and generates the world.

The geometry-aligned vision encoder and large language model produce understanding outputs, while learnable queries and connectors transform these into conditioning signals for the diffusion generator.

This setup allows perception and generation to work as two sides of the same model.

Evan: And task unification?

Ashley: Task unification is determined by the available inputs—text, target cameras, reference views, and so on—with a role mask indicating the function of each input.

This single framework supports camera-to-world understanding, camera-controllable text-to-image generation, 3D world generation, and joint appearance-geometry reconstruction.

Evan: That sounds comprehensive.

Can you shed light on the physics perception aspect?

Ashley: Sure.

Given a single image, Puffin-World estimates the camera’s absolute physical state, including roll, pitch, vertical field-of-view, and a semantic scene description.

This is done through autoregressive multimodal sequence modeling rather than direct regression, leveraging the model's sequence modeling capabilities and holistic scene understanding.

Evan: And how does this transition into free-viewpoint spatial simulation?

Ashley: For free-viewpoint spatial simulation, given a text prompt and a desired Omni-Camera map, the model synthesizes an image adhering to the specified viewpoint and intrinsics.

This involves using the camera condition within the diffusion latent space, allowing precise and stable spatial control.

Evan: So how does the 3D world modeling come into play?

Ashley: 3D world modeling extends single-view generation to multi-view trajectories while supporting geometry reconstruction.

It processes a set of views within a unified joint-attention sequence, with reference views providing clean latents and remaining views as targets to be denoised.

This setup ensures cross-view consistency and physical grounding.

Evan: And what specific technique do they use to maintain this consistency across views?

Ashley: They use a technique called physics propagation.

The absolute state of the reference view is perceived and then propagated across future frames, ensuring that each view retains a coherent gravity-anchored frame while exhibiting individual relative motions.

This guarantees physically and spatially stable generation across diverse trajectories.

Evan: Does this approach also include geometry?

Ashley: Yes, joint appearance-geometry modeling is implemented by predicting appearance and geometry together.

Depth is encoded as an RGB image using a color mapping, enabling both to share the same latent representation.

This design supports seamless multi-view modeling and geometry prediction without additional encoders or decoders.

Evan: It sounds like a complex model.

How did they train Puffin-World?

Ashley: They employed a four-stage training strategy for the entire framework.

Stage I focuses on cross-modal alignment and single-view understanding and generation.

Stage II involves fine-tuning the full framework.

In Stage III, the model is extended to cross-view 3D world modeling.

Finally, Stage IV improves geometry reconstruction by jointly modeling appearance and depth.

Evan: What kind of data did they use for training?

Ashley: The Puffin-16M dataset forms a major part of their training data, along with other public datasets like RealEstate10K and ScanNet.

Puffin-16M provides extensive vision–language–camera triplets and challenging motion trajectories, ensuring diverse and comprehensive training.

Evan: That's a robust dataset.

What about their training specifics and key challenges?

Ashley: They used various techniques: physics propagation pre-computation for training efficiency, multi-source sampling for balanced training data, and classifier-free guidance through condition dropout.

One challenge was maintaining physical consistency across extended trajectories, which they addressed using the Omni-Camera and physics propagation mechanisms.

Evan: It sounds like a massive undertaking.

How about we explore the datasets used and the evaluation metrics next?

Ashley: That's a great idea, but for now, that wraps up the Method section.

Evan: Alright Ashley, let's dive into the experiments and results.

How was Puffin-World evaluated?

Ashley: Puffin-World’s performance was evaluated across several tasks: camera-to-world understanding, camera-controllable image generation, and 3D world modeling.

Let's break it down, starting with camera-to-world understanding.

Evan: How did they measure the effectiveness of camera-to-world understanding?

Ashley: They evaluated Puffin-World against a diverse set of methods on four widely adopted benchmarks: MegaDepth, TartanAir, LaMAR, and Stanford2D3D.

They looked at angular errors for roll and pitch, and the error in vertical field-of-view for camera intrinsics.

The metrics reported include median error and AUC at thresholds of 1, 5, and 10 degrees.

Evan: And how did Puffin-World perform on these metrics?

Ashley: Puffin-World outperformed all competing methods, achieving the best median error across all benchmarks and most AUC metrics.

For instance, on MegaDepth, Puffin-World achieved a median roll error of 0.28 degrees and a pitch error of 1.01 degrees, significantly better than previous methods.

Evan: That's impressive.

What about camera-controllable image generation?

Ashley: For camera-controllable image generation, they assessed Puffin-World against recent state-of-the-art models on a benchmark they introduced, Puffin-Cam-Bench.

They measured angular errors for up vector, latitude, gravity direction, and visual fidelity using Frechet Inception Distance.

Evan: And how did Puffin-World fare in this area?

Ashley: Puffin-World consistently outperformed other models.

For instance, it achieved a mean up vector error of 0.96 degrees and a gravity direction error of 1.32 degrees, both significantly lower than other methods.

It also scored the best FID with 75.93, indicating high visual fidelity and realism.

Evan: Great.

Now let's discuss the results in 3D world modeling.

Ashley: For 3D world modeling, Puffin-World demonstrated its capabilities across image-to-3D generation, text-driven 3D generation, flexible rotation control, and compound motion control.

Visual fidelity was measured using PSNR, SSIM, and LPIPS, while camera control accuracy was evaluated by comparing generated frames’ estimated parameters against ground truth.

Evan: And how did the performance stand compared to other methods?

Ashley: On the RealEstate10K and Puffin-Traj-Bench, Puffin-World achieved the highest PSNR at 17.22 and 18.00 respectively.

It also scored best in LPIPS with 0.288 on RealEstate10K and 0.288 on Puffin-Traj-Bench, indicating excellent perceptual quality.

For camera control accuracy, it outperformed all baselines across roll, pitch, and field-of-view metrics.

Evan: It seems like Puffin-World is robust across a variety of scenarios.

Were there any ablation studies or additional experiments mentioned?

Ashley: Yes, they conducted an ablation study on the physics propagation mechanism, comparing performance with and without it.

Results showed significant improvements in visual fidelity and physical consistency with physics propagation, especially under challenging rotational motions.

Evan: Interesting.

How did they approach long-horizon exploration?

Ashley: They extended generation using an autoregressive approach, ensuring all chunks share a coherent gravity-aligned frame, thus preventing orientation drift.

This was achieved by conditioning on the denoised target latent of the carried-over view from one chunk to the next.

Evan: To sum up, it seems like Puffin-World has set a new benchmark across multiple tasks.

Anything else noteworthy from the experiments?

Ashley: The results emphasize the model's robust performance in handling diverse tasks with high fidelity and stability.

Puffin-World exhibits promising multi-task synergy, beneficial for complex applications like virtual reality and embodied intelligence.

Evan: Indeed.

That wraps up the Experiment section.

Evan: Alright Ashley, let's round up our discussion with the related work.

How does Puffin-World compare and build upon previous efforts in this field?

Ashley: Puffin-World situates itself within a broad landscape of multimodal models, generative world models, and unified approaches.

Let's start with camera-to-world understanding.

Historically, recovering physical camera parameters from images has been a longstanding problem in 3D vision.

Evan: Right.

Can you elaborate on the evolution of approaches in this area?

Ashley: Certainly.

Early methods regressed camera parameters directly from a single image.

However, recent approaches like dense pixel-wise geometric representations have provided greater robustness.

Notable examples include distortion maps, pixel displacement fields, and incidence fields.

Puffin-World builds on this by using a language-modeling perspective for camera-centric tasks.

Evan: Interesting approach.

And what about unified multimodal models?

Ashley: Unified multimodal models aim to integrate visual understanding and generation within a single framework.

Traditional models focused on 2D semantics, such as autoregressive modeling over discrete or continuous visual tokens.

More advanced methods used diffusion decoders.

Puffin-World extends this to 3D by seamlessly integrating physical, geometric, and appearance states.

Evan: How does this compare specifically to previous models?

Ashley: While previous efforts like GeoCalib and MSCC have made strides in camera-to-world understanding, and models like Vision-ST3D and MVGenMaster excel in 3D scene synthesis, Puffin-World uniquely combines these capabilities into a single model.

This unification supports seamless transitions between perception, simulation, and reconstruction.

Evan: And generative world models—how does Puffin-World fit in there?

Ashley: Generative world models, such as video and multi-view diffusion frameworks, have been impressive but often focus solely on appearance.

Models like CAT3D and VD3D have aimed to condition generation on detailed camera information.

Puffin-World, however, anchors generation to absolute physical frames and integrates perception, simulation, and reconstruction within one framework.

Evan: What major gaps does Puffin-World address compared to these existing models?

Ashley: There are a few key gaps Puffin-World addresses: First, the combination of appearance generation with explicit physical grounding.

Second, maintaining physical consistency over long trajectories, which models based on relative camera motions often struggle with due to orientation drift.

Lastly, Puffin-World's closed-loop applications demonstrate multi-task synergy that's rare in other models.

Evan: Can you give a bit more detail about the closed-loop applications?

Ashley: Certainly.

Closed-loop applications involve continuous interaction between the model and the environment, such as mimic world exploration and self-calibrated world exploration.

These applications highlight Puffin-World's ability to adapt and maintain consistency across dynamic scenarios, setting it apart from more static generation-focused models.

Evan: It sounds like Puffin-World is setting new benchmarks in several areas.

Are there any specific studies or models that directly influenced its development?

Ashley: Yes, definitely.

Models like GeoCalib and AnyCalib were instrumental in refining camera parameter estimation.

Unified multimodal frameworks like those explored in works on visual language models have influenced its integrative approach.

Generative frameworks like Stable Diffusion and Vision-ST3D provided insights into scaling and fidelity improvements for world modeling.

Evan: So, Puffin-World is both a continuation and an enhancement of these previous efforts?

Ashley: Exactly.

Puffin-World stands on the shoulders of many pioneering works, while pushing forward with innovations in unified modeling of 3D world states and physically grounded scene generation.

Evan: A very comprehensive overview.

That’s the end of the Related Work section.

Evan: As we wrap up, Ashley, could you summarize the key contributions and takeaways from Puffin-World?

Ashley: Of course.

To summarize, Puffin-World is a groundbreaking unified multimodal model that perceives, generates, and reconstructs 3D world states.

Its core innovation lies in modeling appearance, geometry, and physics seamlessly within a single framework.

Evan: Right, and this was enabled by their Omni-Camera representation and the physics propagation mechanism.

These approaches ensure that the model maintains physically consistent and spatially stable world generation.

Ashley: Exactly.

Another major contribution is the creation of Puffin-16M, a massive dataset comprising 15 million vision–language–camera triplets and 1 million trajectories showcasing diverse and challenging motions.

This dataset is fundamental for scaling and testing the model’s capabilities.

Evan: The breadth and depth of their approach are truly impressive.

They evaluated Puffin-World across various benchmarks, demonstrating superior performance in camera-to-world understanding, camera-controllable image generation, and 3D world modeling.

Ashley: And let’s not forget about the practical implications.

Puffin-World shows promising applications in closed-loop tasks like mimic world exploration and self-calibrated world exploration, further cementing its utility in virtual reality and embodied AI fields.

Evan: It's clear that Puffin-World is setting a new paradigm for unified multimodal models.

They’ve also fostered the community by open-sourcing their code, models, and datasets, thereby promoting further research and development.

Ashley: Indeed.

This paper marks a significant step forward in the field of spatial intelligence and physical AI.

I'm excited to see how the research community builds upon these contributions.

Evan: That concludes our discussion on Puffin-World.

Thank you all for tuning in to today's episode of Daily Paper Cast.

Don’t forget to subscribe and join us tomorrow for more insights into the latest research in AI, NLP, CV, and related fields.

See you next time!

Ashley: Thanks for listening, everyone!

Take care and see you next episode!