SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution

🤗 Upvotes: 26 | cs.CV

Authors:
Xingjian Ran, Xiaoye Mo, Sihao Liu, Jianyu Zhang, Li Luo, Bo Dai

Title:
SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution

Arxiv:
http://arxiv.org/abs/2609.05594v1

Abstract:
Diverse and simulation-ready indoor scenes are essential for interactive entertainment and embodied AI, yet their scalable generation remains challenging. Recent agentic text-to-3D scene pipelines that rely on vision-language models (VLMs) can generate scenes of high fidelity but require costly iterative object placement and refinement. Another mainstream paradigm, parametric image-to-3D scene models, produces scenes efficiently from strong priors learned from 2D images but often leads to imprecise and physically invalid scenes. More importantly, both paradigms struggle to output diverse scenes for a single input, making it hard for them to reflect the dynamically changing nature of real scenes. In this paper we propose \textbf{SceneMosaic}, a framework that combines the merits of both paradigms. It obtains the initial candidate from the learned image-based prior, and subsequently evolves the result through VLM agents, ensuring both efficiency and physical validity. Within the evolution process, SceneMosaic exploits the locality of natural scenes and decomposes a scene into independent local units, allowing separate evolution within each unit before composing the global scene via Cartesian product. On SceneEval-100, SceneMosaic matches the strongest agentic baseline in semantic layout quality with a 24x speedup, substantially reduces physical violations, and receives the highest human ratings. Our code is publicly available at https://github.com/rxjfighting/SceneMosaic.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Ashley: Today we’re discussing a paper from the Hugging Face daily paper list, September 9, 2026.

It’s garnered 26 upvotes.

Evan: The title is quite intriguing: 'SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution'.

Ashley: It's authored by Xingjian Ran and Xiaoye Mo, with the corresponding author Bo Dai, and the team is from The University of Hong Kong and the University of Electronic Science and Technology of China.

Evan: Jumping right into the Introduction, the paper addresses the scalable generation of simulation-ready indoor environments.

This is crucial for fields like interactive entertainment and embodied AI.

Ashley, could you explain why scalability in scene generation is so challenging?

Ashley: Creating diverse, detailed indoor scenes that are also ready for simulation involves balancing high fidelity with efficiency.

Traditional methods, like agentic text-to-3D scene pipelines, are very detailed but computationally expensive and slow.

On the other hand, parametric image-to-3D models can quickly generate scenes using learned 2D image priors but often produce physically invalid results.

Plus, both struggle to adapt scenes dynamically, which is critical for simulating real-world environments.

Evan: So, what solution does this paper propose to overcome these obstacles?

Ashley: The authors propose SceneMosaic, a framework that combines the strengths of both agentic text-to-3D and parametric image-to-3D models.

It starts with a rapidly generated initial layout from an image-based prior.

This coarse layout is then refined by vision-language model agents iteratively to ensure both efficiency and physical validity.

Evan: That sounds promising.

How does SceneMosaic handle the diversity and dynamic nature of real-world scenes?

Ashley: Great question.

SceneMosaic decomposes scenes into independent local units, allowing separate evolution within each unit.

This essentially means that one part of the scene can evolve independently from another, and then these are combined in multiple ways to create diverse scene variants using a Cartesian product.

This method captures the structured rearrangements that are common in real-world settings, making the simulations more realistic and useful for training and evaluation.

Evan: And what specific contributions does this paper make in terms of results and improvements?

Ashley: SceneMosaic achieves several key improvements.

On the SceneEval-100 dataset, it matches the best agentic baseline in semantic layout quality but is 24 times faster.

It also significantly reduces physical violations like object collisions and instability commonly seen in image-to-3D models.

According to human ratings, the scenes generated by SceneMosaic are also rated highest in both semantic and physical plausibility.

Evan: That's quite a leap in both performance and quality.

So, to sum it up, SceneMosaic leverages a hybrid approach to tackle the issues of efficiency, accuracy, and diversity in simulated scene generation, making it highly suitable for applications in interactive entertainment and embodied AI.

This brings us to the end of the Introduction section of the paper.

Evan: Let's move on to the methods proposed in SceneMosaic.

Ashley: Right, the Method section of SceneMosaic is quite comprehensive.

It starts by formulating the problem: given a single reference input, like an image or a text-to-image prompt, the goal is to generate a base 3D scene and multiple diverse variants.

Evan: And how do they construct these 3D scenes from just a single reference input?

Ashley: They use a hybrid approach.

First, they perform object-centric scene reconstruction.

This involves unifying all input signals into a scene image.

A perception agent registers objects in the image, producing a stable object inventory with semantic labels, descriptions, and bounding boxes.

Using these, each object is then refined and segmented, ensuring a consistent representation across the scene.

Evan: So, this sets up the initial scene.

How do they ensure the detailed layout and structure of the scene?

Ashley: Next, they convert the reconstructed objects into a structured scene representation, inferring room structures, object relations, and physical properties.

This process is organized as a directed acyclic graph or DAG.

Specialized task agents within this framework tackle support reasoning, containment analysis, wall recovery, and semantic refinement, ensuring consistent information flow.

Evan: Interesting approach.

Once they have this structured model, what comes next?

Ashley: Then, they move on to the physics-based layout stabilization.

This involves correcting any containment issues and conducting gravity-based simulations.

Essentially, they place the layout into a physical simulation environment to resolve collisions and ensure stability.

This creates a physically grounded base scene.

Evan: How do they handle the actual evolution and refinement of the layout?

Ashley: This is handled through what they call 'Critic-Actor Layout Evolution.' Starting from the physically stable layouts, they further improve these through iterations.

In each iteration, a Critic agent assesses the layout's validity based on visual evidence and simulation tools.

It provides diagnostic feedback and suggests optimizations.

An Actor agent then adjusts the layout parameters based on this feedback.

Evan: So the Critic-Actor loop iteratively refines each local layout.

Does this approach ensure diversity as well?

Ashley: Yes, precisely.

They use agent memory and spatial abstraction to stabilize updates and avoid optimization deadlocks.

Each local layout evolves independently, capturing significant scene diversity.

Finally, they assemble various local layouts combinatorially to generate a large pool of candidates.

Evan: How do they filter and select the most novel among these candidates?

Ashley: Great point.

They employ a 'Novelty-Aware Scene Distance' metric that considers relative position, absolute distance, and rotation differences.

They then use a dynamic greedy search to select the most representative and diverse subset of scenes from the candidate pool.

Evan: This approach seems to cover both the quality and the diversity aspects comprehensively.

What about their evaluation metrics and datasets?

Ashley: SceneMosaic's performance was evaluated on SceneEval-100, a dataset with 100 manually designed indoor scene prompts.

They benchmarked their method against both agentic text-to-3D and image-to-3D models.

The evaluation considered semantic quality and physical validity measures, like positional correctness and collision rates.

Evan: And how did SceneMosaic perform compared to these benchmarks?

Ashley: Their results are impressive.

SceneMosaic matches the semantic layout quality of the strongest agentic baselines but does so at a fraction of the computational time.

Specifically, it is 24 times faster.

It also significantly reduces physical violations like collisions, compared to the image-to-3D models.

Evan: Were there any user studies to validate these findings?

Ashley: Yes, they conducted a user study with 48 participants who rated scenes on semantic and physical plausibility using a 5-point Likert scale.

SceneMosaic outperformed other methods with the highest ratings and the lowest standard deviations, ensuring consistency across evaluations.

Evan: To sum up, SceneMosaic's hybrid method of using both image-based and agentic systems allows for accurate, diverse, and physically valid indoor scene generation, well-suited for real-world applications.

This brings us to the end of the Method section of SceneMosaic.

Evan: Moving on to the Experiments and Results section of the SceneMosaic paper, what methods were used to validate the proposed framework, Ashley?

Ashley: The authors conducted extensive experiments on the SceneEval-100 benchmark, which includes 100 manually designed indoor scene prompts.

They evaluated SceneMosaic against both agentic text-to-3D and image-to-3D generation paradigms.

Evan: Right, and this evaluation covers both the efficiency and the quality of the generated scenes.

Can you explain what specific metrics they used?

Ashley: Sure.

They used several metrics to assess semantic quality and physical plausibility.

For semantic quality, they evaluated positional correctness (POS) and rotational correctness (ROT).

For physical plausibility, they measured navigability (NAV), collision rate (COL), and out-of-bounds rate (OOB).

Evan: Starting with semantic quality, how did SceneMosaic perform compared to the baselines?

Ashley: SceneMosaic achieved high semantic quality, with positional correctness (POS) and rotational correctness (ROT) scores matching the strongest agentic baseline, SceneSmith.

This is a significant accomplishment because SceneSmith is known for its high semantic fidelity but at the cost of long generation times.

Evan: Interesting.

How about the physical plausibility measurements like navigability, collision rate, and out-of-bounds rate?

Ashley: SceneMosaic shines in physical plausibility.

It achieved a near-perfect navigability score (99.2%) and had zero collision and out-of-bounds instances.

This is a critical improvement over image-to-3D models like SAM3D, which often suffer from high collision rates and boundary violations.

Evan: What about the efficiency of SceneMosaic?

Did it manage to outperform the other methods in terms of compute time?

Ashley: SceneMosaic demonstrated exceptional efficiency with an average generation time of just 0.14 hours per base scene, compared to 3.43 hours for SceneSmith, which is the fastest traditional agentic method.

When generating diverse variant scenes, the time dropped further to 0.03 hours per variant, thanks to their combinatorial approach.

Evan: How about user feedback?

Did they run any user studies to gather qualitative data on the generated scenes?

Ashley: Yes, they conducted a user study with 48 participants who rated scenes on semantic and physical plausibility using a 5-point Likert scale.

The results were compelling.

SceneMosaic scenes were rated the highest on both semantic and physical plausibility, with means of 4.33 and 4.47 respectively.

Furthermore, the low standard deviations indicate consistent quality in the generated scenes.

Evan: It’s impressive that they balanced both performance and quality.

Were there any ablation studies to understand the impact of individual components of SceneMosaic?

Ashley: Yes, they performed comprehensive ablation studies to validate the key designs.

For instance, removing image-based initialization drastically increased generation latency and slightly degraded semantic alignment scores.

Omitting physics-based processing and tools led to notable degradation in physical validity metrics, emphasizing the importance of each component.

Evan: That clearly shows how each part of the framework contributes to the overall performance.

What about diversity in the generated scenes?

How did they evaluate that?

Ashley: They used two diversity metrics: Div-IoU and Div-N.

Div-IoU calculates the pairwise intersection-over-union of occupancy maps, while Div-N uses the novelty-aware scene distance.

SceneMosaic showed high diversity in generated layouts, outperforming even repeated sampling methods.

This high diversity was achieved without sacrificing quality, which is quite remarkable.

Evan: This is a thorough piece of work with a well-rounded evaluation.

Anything unique they found that’s particularly worth mentioning?

Ashley: One intriguing finding was the capability to generate diverse and functional layouts quickly.

The combinatorial variant assembly allowed SceneMosaic to produce a wide range of plausible scene variants with minimal computation, demonstrating scalability and flexibility crucial for real-world applications.

Evan: Great insights, Ashley.

That wraps up the Method and Experiment sections of SceneMosaic.

Evan: Let's now discuss the Related Work section of the SceneMosaic paper.

Ashley: Sure, Evan.

This section is crucial as it places SceneMosaic within the broader context of existing research in 3D indoor scene generation.

The authors review three primary areas: 3D indoor scene synthesis, language models and vision-language models driven scene generation, and image-to-3D generation.

Evan: Let's start with 3D indoor scene synthesis.

What does the paper highlight in this area?

Ashley: Early work in this area primarily focused on learning object arrangements from annotated layout datasets.

These methods used autoregressive transformers, diffusion models, and explicit layout priors with constraint graphs to place furniture within a room.

However, these approaches had limitations in semantic controllability and often relied on category-specific training data.

Evan: And how does SceneMosaic compare?

Ashley: SceneMosaic advances this by coupling strong visual layout priors with agentic refinement.

This combination allows it to produce dense, physically valid scenes without the need for category-specific training data.

It represents an evolution in approach, focusing on both efficiency and detailed scene semantics.

Evan: What about the use of language and vision-language models in scene generation?

Ashley: Recent advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have significantly influenced scene generation.

These models can guide scene synthesis by predicting numerical layouts or structured scene descriptions, often incorporating program synthesis over object databases.

Agentic pipelines further extend this by managing perception, placement, and iterative correction through tool use and multimodal feedback.

Evan: Can you give an example of how these pipelines work?

Ashley: For instance, SceneSmith orchestrates multiple steps including perception and object placement iteratively, achieving high semantic fidelity but at a considerable computational cost.

SceneMosaic, however, retains the flexibility of agentic reasoning while grounding it in image-based initialization.

It decomposes scenes into locally independent units, creating a more efficient and diverse evolution process.

Evan: And finally, what insights does the paper provide on image-to-3D generation?

Ashley: Advances in feed-forward 3D asset generation have enabled methods to infer both object geometry and coarse layouts from single images.

These methods utilize strong visual priors for rapid initialization, but often result in physically invalid poses and collisions.

SceneMosaic leverages these image-based priors for a quick start and refines the scene using agentic layout evolution to ensure physical plausibility.

Evan: It sounds like SceneMosaic successfully integrates and improves upon various existing methodologies.

Would you agree?

Ashley: Indeed.

SceneMosaic builds on the strengths of previous work by combining image-based rapid initialization with the detailed refinement capabilities of agentic models.

This hybrid approach allows it to produce high-quality, diverse, and physically valid scenes efficiently.

Evan: Did the authors mention any specific methods that directly influenced SceneMosaic’s design?

Ashley: Yes, aside from the general methods, the authors referenced specific models such as SAM3D for object reconstruction, and SceneSmith and SAGE for agentic scene synthesis.

These models provided a foundation upon which SceneMosaic expands by adding a layer of combined local and global refinement.

Evan: What would you say is the unique value proposition of SceneMosaic compared to these models?

Ashley: The unique value lies in its balanced approach.

It not only generates semantically rich and physically plausible scenes but also does so with remarkable efficiency and diversity.

The use of locality to decompose scenes into manageable units and the Cartesian product for diverse scene generation are key innovations that set SceneMosaic apart.

Evan: That makes SceneMosaic quite a comprehensive and forward-thinking solution.

Alright, that brings us to the end of the Related Work section.

Evan: As we wrap up this episode, let's summarize the key contributions and takeaways from the SceneMosaic paper.

Ashley: Definitely.

SceneMosaic's primary contribution is its hybrid approach that combines the strengths of both agentic text-to-3D and parametric image-to-3D models.

This allows for rapid initialization of scenes with high semantic and physical fidelity.

Evan: This innovative combination significantly bridges the efficiency-fidelity gap observed in traditional methods, enhancing both the speed and realism of generated scenes.

Ashley: That's right.

The decomposition of scenes into independent local units is another key feature.

This enables individual evolution within each unit and then combines them using a Cartesian product to generate diverse scene variants.

Evan: Lastly, SceneMosaic's results on the SceneEval-100 dataset showcased superior performance in terms of semantic quality, physical validity, and computational efficiency, compared to state-of-the-art agentic and image-based methods.

Ashley: In summary, SceneMosaic stands out by efficiently synthesizing diverse, simulation-ready indoor scenes that are both semantically rich and physically plausible.

It achieves this through a well-thought-out hybrid approach, setting a new benchmark in the field.

Evan: This paper certainly pushes the envelope when it comes to generating realistic and varied 3D scenes.

That brings us to the end of today's episode.

Ashley: We hope you enjoyed diving into SceneMosaic with us.

Join us again tomorrow for more insights from the world of AI research here on Daily Paper Cast.

Evan: Thanks for tuning in, everyone.

See you next time!

Ashley: Take care and keep exploring!