Show-Harness: Just a VLM Agent Can Play Robots

🤗 Upvotes: 43 | cs.RO, cs.AI, cs.CV, cs.MM

Authors:
Yanzhe Chen, Zechen Bai, Zhijun Cao, Wenzheng Zeng, Kevin Qinghong Lin, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Qiming Huang, Mike Zheng Shou

Title:
Show-Harness: Just a VLM Agent Can Play Robots

Arxiv:
http://arxiv.org/abs/2609.10522v1

Abstract:
Foundation vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an Embodied Harness that enables VLMs to "play" robots through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for fine-grained physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Ashley: Today's paper comes from Hugging Face's daily paper list of September 10, 2026, and has garnered 43 upvotes.

Evan: The title of the paper is 'Show-Harness: Just a VLM Agent Can Play Robots'.

Ashley: The first two authors are Yanzhe Chen and Zechen Bai, and the corresponding author is Mike Zheng Shou from the Show Lab, National University of Singapore.

Evan: Ashley, let's dive into the Introduction.

Can you give us some background on what this paper is addressing?

Ashley: Evan.

So, foundation vision-language models, or VLMs, already show a lot of broad intelligence about the world.

They can recognize and understand objects and spatial relations, and even break down long-term goals.

The challenge, however, is translating this intelligence into robot control, which isn't straightforward.

Evan: Right, I've seen these models used effectively in various applications, but robot manipulation seems particularly complex.

Why is that?

Ashley: It boils down to the need for precise physical control.

Vision-language-action models, or VLAs, usually try to address this by fine-tuning VLMs for specific robot actions, but these often collapse the rich semantic understanding into an opaque mapping from pixels to actions.

This means you have to repeatedly adapt the model for different tasks, environments, and robot embodiments.

Evan: So, how does Show-Harness address this issue?

Ashley: Show-Harness aims to bridge this gap with a model-agnostic Embodied Harness.

The key is its use of a compact semantic action interface which links intent to action more intuitively.

The idea is to expose discrete semantic action units that the VLMs can reason over naturally, while specific interpreters ground these units into local, embodiment-specific robot actions in a deterministic manner.

Evan: That sounds intriguing.

What are these semantic action units like?

Ashley: Each semantic action unit specifies a single movement or a gripper action.

For example, movements could be 'move forward', 'move up', or 'rotate clockwise'.

These units are directly interpretable by the VLM, making it more straightforward for the model to execute fine-grained physical decisions.

Evan: And what problem does this solve in practical terms?

Ashley: Good question.

Practically, Show-Harness makes it possible to control robots directly using VLMs without extensive retraining or model adjustments.

This means you can deploy closed-source VLMs for zero-shot robot control or adapt smaller, open-source models for low-cost deployment with just a few GPU-hours of fine-tuning.

Evan: Interesting.

What about the adaptability across different embodiments and environments?

Ashley: Show-Harness excels in this aspect as well.

It generalizes robustly across tasks, embodiments, and environments by translating high-level semantic decisions into precise actions.

This has been shown to outperform other agentic and VLA paradigms in extensive experiments.

Evan: So is Show-Harness limited to robot manipulation tasks, or is it more versatile?

Ashley: It's quite versatile.

The researchers also developed GUMI, which stands for GUI Manipulation Interface.

This extends Show-Harness's semantic action space to graphical user interface-based demonstration collection.

Essentially, this allows both humans and agents to 'play' robots across various embodiments without needing specialized teleoperation hardware.

Evan: It sounds like a transformational approach.

What are some of the key contributions the paper highlights?

Ashley: The paper emphasizes four main contributions.

First, it introduces Show-Harness itself, which operates robots via a compact semantic action interface.

Second, it demonstrates the feasibility of using both closed-source frontier VLMs for zero-shot control and smaller-scale models for low-cost deployment.

Third, it showcases the strong generalization capabilities of Show-Harness-equipped VLM agents across various tasks and environments.

Lastly, it presents GUMI, enabling flexible and reusable data collection for robot demonstrations.

Evan: Indeed, those sound like significant steps forward in the field of robot manipulation using VLMs.

And that wraps up the Introduction section.

Evan: Alright, Ashley, let's delve into the methods detailed in the paper.

How is Show-Harness structured and how does it function?

Ashley: Evan.

The authors provide a comprehensive breakdown of Show-Harness, starting with an overview of its architecture.

Essentially, Show-Harness is designed to translate the intelligence of foundation VLMs into physically grounded robot behavior by putting the VLM inside an iterative perception–reasoning–action loop.

Evan: Can you explain what happens at each interaction step in this loop?

Ashley: Yes, of course.

At each interaction step, the VLM receives perceptual inputs and interaction history, reasons about the next action, expresses this intent as a semantic action decision, and observes the outcomes of this decision when it's physically executed by the robot.

Evan: How do these perceptual inputs and interaction history work together in practice?

Ashley: Good question.

The system captures multi-view visual observations and the robot’s proprioceptive state — which includes things like joint angles and velocities.

These inputs are processed through a set of reasoning plugins, generating what they call a reasoning-refined context.

This context helps the VLM make an informed decision.

Evan: I see.

And what specifically are these reasoning plugins?

Ashley: The reasoning plugins are really the core of the system's adaptability.

They include components for subtask planning, situated planning, action chunking, adaptive step sizing, and visual prompts.

Each plugin serves a specific purpose within the perception–reasoning–action loop to refine the decision-making context for the VLM.

Evan: Could you give an example of how one of these plugins works?

Ashley: Sure.

Let’s take subtask planning.

This plugin breaks down the task instruction into an ordered sequence of subtasks, each with completion criteria that are visually checkable.

So, instead of solving the entire task at once, the VLM tackles manageable chunks, ensuring each subtask is completed before moving to the next.

Evan: That makes a lot of sense.

Now, how are these semantic action units grounded in physical control?

Ashley: Each semantic action decision made by the VLM is passed to an embodiment-specific interpreter.

This interpreter translates the semantic action into a specific robot control action.

For instance, a semantic unit like 'move forward' might be converted into a precise movement command suitable for the particular robot being used.

Evan: So, the interpreter is what makes the action physically happen.

What about ensuring the action is safe and effective?

Ashley: Right.

Safety bounds and limits are configured within the interpreter to block actions that would violate workspace constraints, ensuring safe physical interaction.

And feedback on execution is given after each action unit, allowing the VLM to make real-time adjustments.

Evan: Interesting.

What about the hardware setups used for these experiments?

Ashley: The paper outlines two main hardware platforms used for experiments: a 7 degrees of freedom Franka Research 3 robot arm and a dual-arm AgileX setup.

Both platforms are equipped with parallel-jaw grippers and observed by multiple cameras, providing both global and detailed views necessary for precise manipulation tasks.

Evan: And what tasks did they test on these platforms?

Ashley: They designed ten real-robot manipulation tasks involving different objects and target placements, such as placing a banana on a plate or a teddy bear in a bowl.

The tasks cover diverse physical properties, like rigid geometry, rolling dynamics, and deformability, to test the system’s versatility.

Evan: Were there any tasks or objects that were held out for evaluation?

Ashley: Yes, they held out a teddy bear and a chess piece for out-of-distribution evaluation to test the generalization capabilities of the fine-tuned VLMs.

Evan: How did they measure success in these tasks?

Ashley: The main metrics reported were the success rate and the average number of steps per episode.

Each episode was capped at 50 steps, with any timeout being counted as a failure.

Evan: What about the comparison with baseline methods?

Ashley: The paper compares Show-Harness with three representative families of robot-control systems: direct VLA models, VLA-centric agents, and code-as-policy agents.

Examples include models like GR00T and π0.5, as well as agentic systems like Harness VLA and Goal-VLA.

Evan: How does Show-Harness perform in comparison?

Ashley: Show-Harness-equipped VLM agents consistently outperformed these baselines across tasks, environments, and embodiments.

This highlights the robustness and adaptability of their semantic-action-based approach.

Evan: That’s impressive.

How about the GUI-based Manipulation Interface, GUMI, they developed?

Ashley: GUMI is quite a significant part of this research.

It's a GUI-based interface allowing both humans and agents to control robots using the same semantic action units.

This facilitates easy collection of demonstration data and supports mixed human–agent data collection without specialized teleoperation hardware.

Evan: How did they collect the demonstration data for their experiments?

Ashley: They used GUMI to collect demonstrations from both real robots and simulated environments.

This data was then used to fine-tune the smaller VLMs in their experiments.

The demonstrations included a range of tasks with several thousand decision steps recorded, ensuring a comprehensive dataset for training and evaluation.

Evan: That's a thorough approach to ensuring robust data collection and model evaluation.

And that brings us to the end of the Method section.

Evan: Now let’s dive into the Experiment and Results section of this paper.

How did they evaluate Show-Harness?

Ashley: To evaluate Show-Harness, the authors conducted extensive experiments across different levels of generalization: tasks, environments, and embodiment shifts.

Specifically, they tested ten real-robot manipulation tasks combining various objects with two target receptacles, a plate, and a bowl.

Evan: That's a diverse set of tasks.

How did they measure the generalization capabilities of the system?

Ashley: They evaluated the system in three primary generalization scenarios.

First, cross-task generalization involved ten object-receptacle tasks.

Second, cross-environment generalization tested the system under different background settings, lighting conditions, viewpoints, distractors, and even sim-to-real transfers.

Finally, cross-embodiment generalization assessed how the system transferred between different robot arms, specifically the Franka and AgileX setups.

Evan: How did Show-Harness perform in these scenarios compared to the baselines?

Ashley: Show-Harness outperformed several baseline systems across all three generalization levels.

For example, in cross-task scenarios, Show-Harness achieved an average success rate of 89 percent in zero-shot mode and 86 percent in fine-tuned mode.

This was significantly better than baselines like GR00T and π0.5.

Evan: Those are impressive numbers.

What about the system’s performance in different environments?

Ashley: In cross-environment tests, Show-Harness maintained high performance even under varied conditions.

It scored a perfect 100 percent in zero-shot mode and 88 percent in fine-tuned mode despite background changes, different lighting conditions, varied viewpoints, and additional distractors.

Even for sim-to-real transfers, which are notoriously challenging, the system succeeded 13 out of 20 times using only simulated demonstrations.

Evan: And how did it handle different robot embodiments?

Ashley: For cross-embodiment evaluations, the results were similarly strong.

Show-Harness demonstrated remarkable adaptability, achieving 93 percent success with the zero-shot frontier models and 87 percent with the fine-tuned smaller models on different robot configurations.

This highlights the effectiveness of its semantic action interface in facilitating smooth transitions between different robot arms.

Evan: Did they also analyze the specific capabilities of the system?

Ashley: Yes, they ran capability analyses focusing on physical and semantic adaptability.

For instance, they tested finer motion control tasks like block stacking and peg insertion by simply adjusting the interpreter step size from 2 cm to 1 cm.

Zero-shot performance improved from 60 percent to 82 percent, and fine-tuned models from 40 percent to 65 percent, without retraining.

Evan: What about tasks requiring coordination between two arms or those that needed more reasoning?

Ashley: They evaluated multi-arm coordination in tasks like tidying a table and passing a banana, showing that a joint policy can prevent collisions and improve task success significantly.

For reasoning tasks, like arranging scattered letters to spell 'SHOW' or performing actions based on a hidden object's location, the system achieved high success rates with the help of plugins like Situated Planning.

Evan: And what were the results of their ablation studies?

Ashley: The ablation studies highlighted the importance of the harness plugins.

For example, removing multi-view guidance dropped success rates from 96 percent to 58 percent.

Similarly, turning off proprioception, subtask planning, or action chunking notably reduced performance, confirming the benefits of these components.

Evan: What did they conclude from the evaluation of different backbones and thinking efforts?

Ashley: They tested multiple VLM backbones like Gemini-3.1 Pro and GPT-5.6-sol.

Stronger models correlated broadly with better performance, but increasing the 'thinking effort' generally improved efficiency more than success rates.

Errors were mostly linked to fine-grained grasping and placement, suggesting areas for further refinement.

Evan: So overall, the experiments provided strong evidence for the robustness and adaptability of Show-Harness across multiple dimensions.

Ashley: Evan.

The extensive testing and detailed analysis showed that Show-Harness could significantly improve robot manipulation tasks by leveraging a semantic action space and structured perceptual feedback.

And that concludes the Experiment section.

Evan: Alright, Ashley, let's now explore the Related Work section of this paper.

How does Show-Harness fit into the broader landscape of existing research?

Ashley: The paper provides a comprehensive look at previous work across several areas, starting with foundation models for robot manipulation.

Vision-language-action, or VLA models, have been a significant focus.

These models essentially couple VLMs with learned action generation to predict robot controls based on visual and language inputs.

Evan: How do these VLA models typically operate?

Ashley: VLA models typically function by regressing continuous actions from visual and language inputs, which often means creating a direct pixel-to-actuation mapping.

This can be done through techniques like regression, diffusion, or flow matching to generate continuous action chunks.

Some use discretized motor tokens or task-space keyframes, while others employ latent action codes from robot trajectories.

Evan: And what are the limitations of these methods?

Ashley: These methods usually trade semantic richness for control precision.

By fine-tuning models to specific motor signals and actions, the broader semantic knowledge encoded in VLMs is often collapsed into an opaque mapping.

This requires fresh adaptations for new tasks or robotic embodiments, making them less versatile.

Evan: What alternatives do researchers explore to overcome these limitations?

Ashley: An alternative is keeping the VLMs above low-level control, using their pretrained semantic knowledge to decompose tasks into subgoals or keypoints instead.

So, the VLM sets the intent, and downstream controllers handle the physical realization.

This preserves the model’s semantic capabilities but surrenders direct physical control.

Evan: How does Show-Harness differentiate itself from these approaches?

Ashley: Show-Harness takes a middle path by proposing a compact set of semantic action units that are both meaningful to the VLM and sufficiently fine-grained for direct execution.

This approach aims to provide the semantic richness missing in low-level methods while retaining direct control, unlike high-level alternatives which relinquish this aspect.

Evan: What about agentic robot systems?

How do they come into play?

Ashley: Agentic systems keep the VLM intact within a harness that translates its high-level semantic decisions into specific robot actions, ensuring a feedback loop between perception, decision, and execution.

These systems can use various strategies for execution, like composing programs, selecting predefined skills, or employing symbolic planners.

Evan: How does Show-Harness utilize these concepts?

Ashley: Show-Harness employs an interface that exposes fine-grained semantic actions directly executable by the robot.

The harness architecture makes the VLM directly responsible for precise movements and manipulation tasks, thus ensuring the semantic intent is accurately realized physically.

Evan: That’s a novel way to combine various strategies.

What does the paper say about digital interfaces and physical manipulation?

Ashley: The paper draws parallels between digital and physical domains, suggesting that exposing compact, interpretable action spaces can be beneficial.

For example, computer-use and game agents operate through discrete commands like keyboard inputs, while similar principles can apply to robot manipulation using a set of semantic actions.

Evan: Could you give an example from the paper?

Ashley: Certainly.

In digital navigation, agents use directional primitives native to simulators, and recent general-purpose agents perform well with them.

Show-Harness applies this principle to physical manipulation, using deterministic grounding of semantic actions for precise control across different embodiments and environments.

Evan: So Show-Harness bridges a gap that other approaches haven't quite succeeded in addressing.

Ashley: Exactly.

By maintaining both semantic richness and physical precision, it aims to unlock the full potential of VLMs in real-world applications.

Evan: That gives a comprehensive understanding of how Show-Harness fits into and advances the current body of work.

And that’s the end of the Related Work section.

Evan: We've covered a lot today.

Ashley, can you summarize the key contributions and takeaways of the paper for our listeners?

Ashley: Of course, Evan.

This paper, 'Show-Harness: Just a VLM Agent Can Play Robots', presents a novel framework that bridges the intelligence of foundation vision-language models with robot control through a compact semantic action interface.

The key contributions include the introduction of Show-Harness, which allows VLMs to directly operate robots by translating high-level semantic decisions into fine-grained actions.

Evan: Right.

It also demonstrates the feasibility of two modes: zero-shot robot control using powerful closed-source VLMs, and efficient fine-tuning of small-scale open-source VLMs for low-cost deployment.

Ashley: Precisely.

Another significant contribution is the creation of GUMI, a GUI-based manipulation interface that enables humans and agents to collect robot demonstrations without specialized teleoperation hardware.

This supports cross-embodiment reuse and human–agent collaborative data collection.

Evan: And their extensive experiments showed that Show-Harness-equipped VLM agents excel in generalizing across tasks, environments, and different robot embodiments, outperforming several representative baseline methods.

Ashley: This highlights the strength and adaptability of their approach, making it a promising direction for future research in robot control using VLMs.

Evan: Absolutely fascinating work.

That's a wrap for today's episode.

We hope you found the discussion on Show-Harness as insightful as we did.

Ashley: Thank you for tuning into Daily Paper Cast.

Don't forget to subscribe and join us again for more deep dives into the latest research papers in AI, NLP, CV, and related fields.

Evan: See you next time!

Bye for now.