PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models

🤗 Upvotes: 58 | cs.CV, cs.RO

Authors:
DeepCybo Team, Yu Bin, Haipeng Cao, Zheng Chang, Kai Chen, Youning Chen, Kailin Deng, Yichao Du, Xiaotong Fu, Haoyang Ge, Yunlong Guo, Chenliu Hao, Jiyan He, Xuguo He, Yakun Hou, Kai Hu, Cong Huang, Tuopusen Huang, Yu Huang, Hong Li, Peize Li, Shijie Lian, Xiaopeng Lin, Yun Lin, Haibao Liu, Haochen Liu, Qiuzhi Liu, Shengcai Liu, Zhiqiang Liu, Tao Luo, Peng Ren, Shuo Ren, Chaoyi Ruan, Zhaolong Shen, Yukun Shi, Qiyuan Su, Yuxuan Tian, Yining Wang, Changti Wu, Hao Wu, Xueyin Xu, Ruoqi Yang, Zhaoyang Yang, Hang Yuan, Zhaoyang Zeng, Hanwen Zhang, Ruimeng Zhang, Yao Zhang, Yibo Zhang, Yuxiang Zhang, Zhirui Zhang, Ziyi Zhang, Zubin Zheng, Zishen Zhuang

Title:
PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models

Arxiv:
http://arxiv.org/abs/2609.14973v1

Abstract:
We present PhysBrain 1.5, a unified model for understanding physical environments, generating actions, and predicting future states. Motivated by the physical loop of observation, interaction, and environmental change, we bring these capabilities into a common learning framework. Starting from a general vision--language model, we encode language responses, end-effector motion, and dense visual targets as discrete sequences and jointly optimize them with autoregressive next-token prediction. Pre-training draws its embodied supervision entirely from human interaction videos, using task-centered episodes to pair semantic and spatial context with recovered motion and subsequent observations. We then adapt the model through supervised fine-tuning on a mixture of human demonstrations, robot trajectories, and simulated experience. Across 28 embodied understanding benchmarks, our 8B model achieves an average score of 72.5, setting a new open-source state of the art and performing on par with leading proprietary models such as GPT-6-Astra and Gemini 3.6 Flash. It achieves the best open-source results on 14 benchmarks while retaining general multimodal capabilities. Beyond these understanding evaluations, qualitative examples show the model's ability to produce end-effector trajectories and predict future scenes through spatially aligned RGB, depth, and robot-mask outputs.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to the Daily Paper Cast!

Ashley: Today's paper is from the Hugging Face daily paper list, dated September 15, 2026, and it has garnered 58 upvotes.

Evan: The paper we're discussing is titled 'PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models.'

Ashley: The first two authors are Yu Bin and Haipeng Cao, with Zheng Chang as the corresponding author, all from DeepCybo Team.

Evan: Alright Ashley, let’s dive into the Introduction section.

What does this paper aim to address?

Ashley: Sure, Evan.

The paper centers around the concept of 'physical intelligence,' which depends on continuous interaction between an agent and its environment.

The authors call this interaction the 'physical loop'—where the agent observes its surroundings, interprets spatial relationships and task goals, anticipates action outcomes, acts on the environment, and uses the changed world to inform its subsequent behavior.

Evan: So, they're proposing a model that encompasses all these steps within a unified framework?

Ashley: Exactly.

They argue that a physical foundation model should provide reusable capabilities for understanding, acting, and predicting within this loop.

In this context, existing models typically specialized in just one aspect—be it visual understanding or robot behavior.

For example, general vision-language models give us semantic knowledge and instruction following, while specialized systems focus on spatial reasoning and planning.

Evan: Can you give us a bit more detail on these specialized systems?

Ashley: Sure.

Some models address spatial reasoning for grounding and affordance, while others connect language reasoning with robot capabilities through skills or spatial control representations.

Recent advancements have started integrating these capabilities, forming more comprehensive embodied foundation models that broaden spatial understanding, interaction reasoning, autonomous decision-making, and planning.

Evan: Are there any examples of these embodied foundation models that have influenced the design of PhysBrain 1.5?

Ashley: Models like HY-Embodied, RynnBrain 1.1, and MiMo-Embodied have contributed significantly.

They explore combined spatial understanding, interaction reasoning, and planning.

SayCan and VoxPoser, for instance, connect language reasoning with robot skills, providing more integrated physical capabilities.

Building on these approaches, PhysBrain 1.5 proposes jointly learning embodied understanding, end-effector motion, and future-state prediction.

Evan: That sounds like a significant leap.

What methodology do they outline for achieving this unified learning?

Ashley: The authors focus on joint learning, accommodating differences in supervision from various tasks.

Understanding tasks produce language, spatial coordinates, and interaction targets.

Action generation demands temporally structured motion representations, while future-state prediction involves dense visual outputs.

They introduce shared learning that accommodates these differences while supporting physical generation.

Their model, PhysBrain 1.5, extends the language vocabulary with discrete action and visual tokens, creating a connected system for understanding, acting, and predicting.

Evan: Fascinating.

How is the data organized to support this kind of learning?

Ashley: The data is sourced from human interaction videos, providing a foundation of tasks, spatial annotations, motion trajectories, and subsequent observations.

Their pre-training corpus includes egocentric and synchronized exocentric recordings, supplemented with panoramic video, to construct task-centered episodes.

These episodes combine semantic and spatial context with recovered human motion and subsequent visual states.

Fine-tuning further integrates diverse sources—human demonstrations, real-robot trajectories, and simulated interactions—to adapt the learned model to various environments and tasks.

Evan: And what achievements are highlighted in their evaluations?

Ashley: The evaluations span 28 benchmarks that cover perception, spatial and multi-view understanding, embodied reasoning and planning, grounding and affordance, and visual trajectory reasoning.

PhysBrain 1.5 achieves an average score of 72.5, setting a new open-source state of the art.

It ranks first on 14 benchmarks and second on 10, performing on par with leading proprietary models such as GPT-6-Astra and Gemini 3.6 Flash.

The qualitative results further demonstrate the model’s ability to generate end-effector trajectories and predict future scenes through spatially aligned RGB, depth, and robot-mask outputs.

Evan: Indeed impressive.

So, to summarize, their unified model brings together various aspects of physical intelligence under one learning framework to achieve state-of-the-art performance across multiple benchmarks.

Ashley: Exactly.

That's the end of the Introduction section, Evan.

Ready to dive into the specifics next?

Evan: Let's dissect the model architecture and methods now.

Evan: Alright, Ashley.

Let's dive into the specifics of the PhysBrain 1.5 model architecture and methods.

How do they structure this unified model?

Ashley: PhysBrain 1.5 builds on the Qwen3-VL Instruct family and extends a pretrained vision-language model into a unified model of physical perception, interaction, and state transition.

The central design is to express language, end-effector motion, and visual-state targets as discrete sequences, which they learn through an autoregressive interface.

The unification happens in the generated representations, which share a language-model backbone, token embedding, and output head.

Evan: That sounds comprehensive.

Can you break down the different components of this model?

Ashley: Sure, Evan.

The model architecture consists of several key components.

First, there's the unified vocabulary that covers language, action, and visual-state tokens.

They denote this as V comprising Vlang for language tokens, Vact for action tokens, and Vvis for visual tokens.

This shared vocabulary supports joint training for understanding and generation across different tasks.

Evan: Interesting.

How do they handle embodied understanding and action generation?

Ashley: Embodied understanding is integrated within the general image and video understanding framework of the base vision-language model.

Physical and spatial perception, multi-view understanding, and embodied reasoning and planning are all enriched through autoregressive supervision.

The model produces either natural-language answers or structured spatial targets, ensuring semantic recognition, spatial localization, and temporal reasoning through a common interface.

Evan: And what about action generation?

Ashley: For action generation, human motion is converted into robot action representations using the Human-as-Humanoid pipeline.

This retains positions and orientations of both wrists while excluding finger motions.

The model predicts end-effector trajectories using ActionPiece, which encodes these trajectories into sequences of discrete action tokens.

Each action chunk consists of 16 steps, predicting relative orientation and translation, and models the physical units consistently.

Evan: How do they ensure the model's actions are adaptable across different sources and embodiments?

Ashley: Adaptability is achieved through the homogeneous action continuation mechanism.

This involves using recent action history as a local motion context, allowing the model to infer local action conventions, motion trends, and short-term response patterns.

This way, the model can continue trajectories naturally without requiring global motion conventions.

Evan: Great.

Let's move on to visual-state prediction.

How do they handle that?

Ashley: For visual-state prediction, the model generates RGB images, depth maps, and robot masks as the future physical state.

These are spatially aligned and tokenized using a shared VQ-VAE with a target resolution of 128 by 128.

Each modality—RGB, depth, and mask—is tokenized and interleaved into a single sequence, which the model then generates autoregressively.

At inference time, these tokens are re-mapped to codebook indices and de-interleaved to produce the corresponding outputs using the same frozen VQ decoder for reconstruction.

Evan: That’s quite a comprehensive process.

How do they train this unified model?

Ashley: Training PhysBrain 1.5 involves two main stages: physical-aware pre-training and embodied supervised fine-tuning.

The physical-aware pre-training data come from human interaction videos, covering various tasks, human motions, and subsequent observations.

This stage uses egocentric recordings, synchronized egocentric-exocentric views, and panoramic videos.

For fine-tuning, they incorporate diverse data sources including human demonstrations, real-robot trajectories, and simulated interactions.

The fine-tuning data also include language and multimodal instruction data to retain broad capabilities.

Evan: What kind of data do they specifically use in these stages?

Ashley: During pre-training, they use structured annotations, captions, and question-answer pairs derived from task interactions in human videos.

This includes object detection, counting, 3D object detection, and more.

For human motion, they extract end-effector trajectories aligned with task execution.

Future-state data pairs initial and resulting visual states to help the model predict scene changes.

In the fine-tuning stage, high-quality interaction data, real-robot trajectories, and additional multimodal instruction data further refine the model’s capabilities.

Evan: And the metrics and evaluation protocols they use?

Ashley: They evaluate on 28 benchmarks across visual-spatial perception, spatial and multi-view understanding, embodied reasoning, spatial grounding and affordance, and visual trajectory reasoning.

The evaluations involve tasks like object counting, spatial relationship understanding, reasoning about task progress, and generating motion trajectories.

They use metrics such as accuracy, normalized RMSE for trajectory predictions, and F1 scores for point-localization tasks to ensure comprehensive performance assessment across different capabilities.

Evan: Sounds like the authors have put together a robust and versatile framework.

That's the end of the Model Architecture and Methods section.

What's next on our agenda, Ashley?

Ashley: Next, we'll dive into the experiments and results to see how PhysBrain 1.5 performs across these various benchmarks.

Stay tuned!

Evan: Alright, Ashley.

Now let's dive into the experiments and results.

How does PhysBrain 1.5 perform across the benchmarks they've set up?

Ashley: The authors evaluate PhysBrain 1.5 on a robust set of 28 benchmarks, which encompass a wide range of tasks, including visual-spatial perception, spatial and multi-view understanding, embodied cognition and planning, spatial grounding and affordance, and visual trace and trajectory reasoning.

Let's start with the foundational visual-spatial perception benchmarks.

Evan: Sure, what do these benchmarks cover?

Ashley: These benchmarks focus on basic visual recognition and spatial discrimination tasks using datasets such as BLINK and CV-Bench.

PhysBrain 1.5 attains remarkable accuracy levels, achieving notable scores on object counting and depth comparison tasks, which are crucial for foundational scene understanding.

Evan: Interesting.

And how does it fare in spatial and multi-view understanding?

Ashley: Spatial and multi-view understanding benchmarks test the model's ability to interpret geometric relations, scene structure, viewpoint changes, and cross-view reasoning.

Using datasets such as 3DSRBench, EmbSpatial-Bench, and MindCube, PhysBrain 1.5 excels in tasks involving spatial relationships and reasoning about scene geometry, maintaining top performance in complex 3D spatial tasks and multi-view understanding.

Evan: How about its capability in embodied cognition and planning?

Ashley: These benchmarks assess goal-directed reasoning, procedural understanding, and planning, with tasks sourced from benchmarks like COSMOS and EgoPlan-Bench2.

PhysBrain 1.5 showcases strong performance in understanding task sequences, assessing progress, and planning next actions.

It effectively handles both short-term and long-term planning scenarios, ranking at the top in several embodied cognition tasks.

Evan: And what about spatial grounding, pointing, and affordance tasks?

Ashley: For spatial grounding, pointing, and affordance, the benchmarks test object localization, referring, affordance recognition, and placement reasoning using datasets like Part-Affordance-2K and PIOBench.

PhysBrain 1.5 consistently delivers precise point annotations and spatial predictions, achieving high F1 scores in grounding and affordance understanding tasks.

Its performance in pointing and placement tasks underscores its robust spatial reasoning capability.

Evan: That sounds comprehensive.

How does it perform in visual trace and trajectory reasoning?

Ashley: Visual trace and trajectory reasoning benchmarks evaluate the model’s ability to understand and predict motion trajectories using datasets such as ShareRobot-Trajectory and VABench-Visual-Trace.

PhysBrain 1.5 achieves impressive results, effectively predicting future motion traces and interaction trajectories.

It aligns predicted waypoints closely with actual motion paths, ensuring high accuracy and reliability in trajectory reasoning tasks.

Evan: That’s quite impressive across all benchmarks.

Were there any qualitative results to showcase the model’s capabilities?

Ashley: Definitely, Evan.

The qualitative results include diverse visualizations that highlight the model's ability to produce end-effector trajectories and predict future scenes through spatially aligned RGB, depth, and robot-mask outputs.

These examples demonstrate realistic scene changes and motion predictions, reinforcing the model’s proficiency in physical-world interaction simulations.

Evan: It's clear that the model performs exceptionally well across various complex tasks.

Were there any notable comparisons with other models?

Ashley: Yes, PhysBrain 1.5 is compared against several proprietary models like GPT-6-Astra and Gemini 3.6 Flash, as well as other strong open-source embodied models like HY-Embodied-VLM-1.0 and RynnBrain1.1.

Despite its open-source nature and compact size of 8 billion parameters, PhysBrain 1.5 holds its own, often outperforming or closely matching these leading models, which underscores its advanced capabilities.

Evan: That's very compelling.

Summarizing these evaluations, PhysBrain 1.5 has set a new open-source state of the art with an average score of 72.5 and numerous first-place rankings across benchmarks.

It's a testament to the efficacy and versatility of their unified learning framework.

Ashley: Exactly, Evan.

That's the end of the Experiment section.

Next, we'll take a closer look at the broader implications and potential applications of PhysBrain 1.5 in various fields.

Evan: Okay, Ashley.

Let’s delve into the Related Work section.

How does PhysBrain 1.5 build upon existing models and research?

What context do the authors provide about the current landscape?

Ashley: The authors outline a rich landscape of prior research that sets the foundation for PhysBrain 1.5.

They begin by highlighting the capabilities of general vision-language models, which are pivotal for visual understanding, semantic knowledge, and instruction following.

Models such as InternVL3 and PixMo have set high standards in these areas.

Evan: So these vision-language models form the basis for understanding visual and semantic content.

How do specialized models fit into this landscape?

Ashley: Exactly.

Specialized models are crucial for addressing specific aspects of physical intelligence.

For instance, spatial reasoning models, such as SpatialReasoner and VSI-Train-10K, excel in geometric understanding, scene structure, and viewpoint reasoning.

Pixel-level tasks, interactive affordances, and spatial understanding are significantly enhanced through these models.

Evan: Can we delve into models that connect language with robotic actions?

Ashley: Certainly.

Vision-language-action models, or VLAs, bridge the gap between language understanding and robot behavior.

Prominent examples in this category are the π series, Qwen-VLA, and GR00T.

These models focus on learning diverse robotic actions, encompassing both short-term and long-term behaviors.

Evan: Interesting.

How do models that incorporate predictive world modeling enhance physical interactions?

Ashley: Predictive world modeling is a significant advancement that provides robots with the ability to anticipate future states and plan actions accordingly.

Models like DreamZero and Qwen-RobotWorld serve as great examples.

They integrate predictive capabilities to improve decision-making and action planning in dynamic environments.

Evan: That’s quite comprehensive.

What systemic approaches do the authors mention for combining language and robotic skills?

Ashley: Systemic models like SayCan and VoxPoser approach this by connecting language reasoning with robotic skills through programmatic or skill-based frameworks.

They leverage language models to generate executable robotic programs or skills that control robot actions, effectively integrating natural language with physical operations.

Evan: How do more recent embodied foundation models contribute to the design of PhysBrain 1.5?

Ashley: Recent embodied foundation models are pivotal for the development of PhysBrain 1.5.

For instance, HY-Embodied, MiMo-Embodied, and RynnBrain 1.1 advance spatial understanding, interaction reasoning, and planning.

These models demonstrate how combining multiple capabilities into a single framework can enhance overall physical intelligence.

Evan: Can you highlight any unified learning approaches from existing models that are referenced?

Ashley: Of course.

Unified learning approaches like those seen in UniVLA and RynnVLA-002 are crucial.

These models focus on shared learning for understanding, action generation, and visual prediction.

Cosmos 3, another notable model, explores multimodal understanding and generation in physical environments and has influenced the integrated learning approach in PhysBrain 1.5.

Evan: It sounds like PhysBrain 1.5 combines various elements from these models to create a comprehensive framework that supports embodied understanding, action generation, and future-state prediction.

They seem to have taken the best aspects from these individual models and integrated them.

Ashley: Precisely, Evan.

The authors have synthesized advancements across general vision-language models, spatial reasoning systems, VLA approaches, predictive world models, and systemic language-robot integration methods to develop PhysBrain 1.5.

This synthesis allows PhysBrain 1.5 to provide a unified framework for physical intelligence.

Evan: That’s the end of the Related Work section.

Next, we'll explore the conclusion and broader implications of PhysBrain 1.5’s developments.

Evan: Alright, Ashley, we’ve covered quite a bit today.

Let’s summarize the key contributions and takeaways of the PhysBrain 1.5 paper.

Ashley: Definitely, Evan.

PhysBrain 1.5 presents a significant advancement in the field of physical intelligence.

It unifies embodied understanding, action generation, and future-state prediction within a single framework.

This approach is rooted in the 'physical loop' concept, emphasizing continuous interaction and adaptation between an agent and its environment.

Evan: Rather than focusing on isolated capabilities like traditional models, PhysBrain 1.5 integrates these aspects through a shared autoregressive backbone, extending a vision-language model with discrete action and visual tokens.

It's a comprehensive method that brings semantic understanding, end-effector motion, and visual-state prediction under one roof.

Ashley: Exactly.

The authors utilized extensive human interaction videos for pre-training, from egocentric and panoramic perspectives to synchronized exocentric views.

This rich dataset enabled them to capture and reproduce complex physical interactions.

Fine-tuning further enhanced the model with diverse data sources, including real-robot trajectories and simulated experience.

Evan: And, their performance evaluations are quite compelling, achieving an average score of 72.5 across 28 benchmarks, setting a new open-source state of the art, and competing closely with leading proprietary models like GPT-6-Astra and Gemini 3.6 Flash.

The model demonstrated top-tier capabilities in perception, spatial reasoning, planning, grounding, and trajectory reasoning.

Ashley: The integrated learning framework allows PhysBrain 1.5 to handle a broad range of tasks effectively, showcasing realistic scene changes, precise point annotations, structured motion continuations, and coherent visual predictions.

This makes it a highly versatile model for applications requiring intricate physical interactions.

Evan: That wraps up our discussion on PhysBrain 1.5, a powerful step forward in embodied AI.

Thank you, Ashley, for unpacking the details for us.

Ashley: It was a pleasure, Evan.

And thank you to our listeners for tuning in to today's episode of the Daily Paper Cast.

We hope you found it insightful.

Evan: Remember to join us next time as we continue to explore groundbreaking research in the world of AI.

Until then, stay curious and keep learning!

Ashley: Goodbye everyone!