UI-Venus-2 Technical Report

🤗 Upvotes: 49 | cs.AI, cs.CL, cs.CV, cs.LG

Authors:
Venus Team, Zhuohan Cai, Haoxing Chen, Jiaxuan Chen, Weizhi Chen, Changlong Gao, Zhangxuan Gu, Yuan Guo, Yusong Hu, Jianrong Jiang, Jianguo Li, Runze Li, Jinzhen Lin, Zhenyu Ma, Changhua Meng, Han Peng, Xinyu Qiu, Shuheng Shen, Zhongyi Shui, Weiqiang Wang, Ming Wen, Zhuoer Xu, Hang Yan, Kaiwen Yang, Ruilin Yao, Nanjun Yu, Zhengwen Zeng, Lianrui Zhang, Yunzhu Zhang, Zhe Zhao, Beitong Zhou

Title:
UI-Venus-2 Technical Report

Arxiv:
http://arxiv.org/abs/2609.00028v1

Abstract:
Multimodal GUI agents have emerged as a promising paradigm for digital task automation, yet transitioning from benchmark-oriented models to dependable real-world applications remains challenging due to limited environment coverage, brittle task construction, and unreliable reward verification. In this work, we present UI-Venus-2, a general-purpose foundation GUI agent designed to operate across mobile, web, and desktop environments through a unified closed-loop reasoning-action framework. To bridge the gap toward practical deployment, we jointly scale three critical dimensions: (1) Environments, expanding coverage to more than 170 multilingual mobile apps and native desktop operating systems; (2) Tasks, employing a deep-research pipeline for function-grounded instruction generation; and (3) Verification, adopting trace-level and sample-level evaluators with visual keypoints and multi-model voting to ensure reliable RL signals for training. Furthermore, we integrate safety-aware mechanisms to ensure controlled execution of consequential actions. By offering a capable, efficient, and open-source foundation, UI-Venus-2 advances the field toward more generalizable, verifiable, and self-reflective agents for real-world applications.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Ashley: Today, we dive into one of the selected papers from Hugging Face's daily paper list of September 2, 2026, with 49 upvotes.

Evan: The paper is titled 'UI-Venus-2 Technical Report'.

Ashley: It's authored by Zhuohan Cai and Haoxing Chen, with Shuheng Shen as the corresponding author from Ant Group.

Evan: Let's dive into their introduction.

What's the background and main objectives of this paper, Ashley?

Ashley: Sure, Evan.

The paper begins by highlighting the significant strides made in creating multimodal Graphical User Interface, or GUI, agents.

These agents represent a promising approach for automating digital tasks.

Essentially, rather than relying on text inputs alone, these agents can interpret and interact with visual elements in applications.

Evan: That sounds quite advanced.

But what challenges do they face with these agents?

Ashley: There are several key challenges.

Transitioning from models that do well on benchmarks to those that are reliable in real-world applications is tough.

They point out issues like limited environment coverage, brittle task construction, and unreliable reward verification.

These need addressing before GUI agents can be widely deployed in practical settings.

Evan: Right, so what does 'UI-Venus-2' propose to tackle these challenges?

Ashley: UI-Venus-2 is quite ambitious.

It's designed as a general-purpose foundation GUI agent.

The main innovation here is its ability to operate across mobile, web, and desktop environments using a unified closed-loop reasoning-action framework.

Evan: Closed-loop reasoning-action framework?

Can you break that down?

Ashley: Basically, the agent observes the current interface, reasons about the task state, executes an action, and then reacts to the environmental feedback for the next decision.

This closed-loop cycle is crucial for it to function effectively across different environments.

Evan: That’s interesting.

How do they propose to bridge the gap towards practical deployment?

Ashley: Their approach scales three critical dimensions simultaneously: environments, tasks, and verification.

They’ve expanded environment coverage to over 170 multilingual mobile apps and native desktop operating systems.

For tasks, they use a research pipeline that grounds instructions in actual app functionalities.

Verification is enhanced with trace- and sample-level evaluators, leveraging visual keypoints and multi-model voting, to ensure the reinforcement learning signals are reliable.

Evan: It's clear they are making comprehensive advancements.

Any notable contributions from their work?

Ashley: Yes, the paper outlines a few key contributions.

Firstly, they have scaled up the multilingual mobile-use environments with reliable task generation.

They’ve added coverage for over 100 Chinese and 70 English language apps, improving task executability even as interfaces change.

Evan: That's quite an achievement.

What else?

Ashley: They’ve also extended capabilities for desktop use from scratch, training the agent to handle tasks across mobile, web, and OS environments in a unified framework.

This means UI-Venus-2 can perceive, reason, and act within desktop environments, efficiently managing a broader range of interactions.

Evan: It sounds like they’ve made significant progress on verification, too.

Ashley: That's correct.

Verification mechanisms have been fine-tuned to evaluate task completion more precisely, using task-relevant visual keypoints and aggregating judgments through model voting.

This makes the reward signal more robust and less prone to exploitation.

Evan: They've put in a lot of effort into making this open-source, right?

Ashley: Indeed.

The code and full parameter weights of UI-Venus-2 are publicly available.

This lowers the barrier for researchers in GUI agent research, facilitating reproducible studies in reinforcement learning and verification.

They hope it accelerates community-driven innovation towards more generalizable and reliable agents.

Evan: Impressive.

And that wraps up our introduction of UI-Venus-2.

Stay tuned as we delve into the methodology and other detailed sections in the following segments.

Evan: Alright Ashley, let's dive into the methods they used for UI-Venus-2.

What's their overall system setup?

Ashley: Great question, Evan.

UI-Venus-2 is essentially a general-purpose foundation GUI agent.

Its standout feature is the combination of visual perception and advanced reasoning within one unified framework.

This allows the agent to operate autonomously across a diverse array of digital environments, including mobile apps, desktop OS, and web platforms.

Evan: Interesting.

So, how do they integrate these diverse capabilities?

Ashley: The key here is their unified reasoning-action paradigm.

The process involves the agent observing the rendered interface images, interpreting the current visual context, translating high-level user intent into executable GUI actions, and continuously adapting its decisions based on the feedback from the environment until the task is accomplished.

Evan: That sounds comprehensive.

Let's break it down.

How was UI-Venus-2 trained?

Ashley: Their training pipeline is divided into three stages: multimodal mid-training, offline reinforcement learning, and multi-teacher on-policy distillation.

Evan: Let's take them one by one.

Start with the multimodal mid-training.

Ashley: In the multimodal mid-training stage, they first trained UI-Venus-2 on a large-scale, heterogeneous mixture of synthetic and interaction-oriented data.

Mobile, web, and OS navigation tasks form the dominant part of their training corpus.

They simulated a variety of interactive environments, spanning different application interfaces and operating systems, collecting executable trajectories within these environments.

Evan: So, what about the realism and quality of these trajectories?

Ashley: Great point.

They balanced realism, linguistic diversity, and task coverage by constructing task queries through a combination of curated query seeds.

The resulting trajectories were then validated using a human–discriminator collaborative process, which filters invalid, ambiguous, or low-quality interactions.

Evan: Got it.

And the second stage, offline reinforcement learning?

Ashley: For offline reinforcement learning, the focus shifts to task-specific supervision, tailored to different interaction scenarios.

They constructed large-scale step-level reinforcement learning trajectories for Mobile, OS, and Web environments.

This helps optimize state-aware action selection, multi-step navigation, transition consistency, and execution reliability.

Evan: That covers Mobile, OS, and Web.

How did they handle CAPTCHA and Grounding tasks?

Ashley: For CAPTCHA and Grounding tasks, they used a programmatic synthesis framework to embed verified CAPTCHA instances and grounding targets into realistic web-page and app interfaces.

This allowed the model to learn robust target identification and precise spatial localization even under visually cluttered conditions.

Evan: They seem to have covered various types of environments and tasks.

What's next in their training process?

Ashley: The final stage is Multi-teacher On-policy Distillation, or MOPD.

Here, they consolidate the capabilities acquired throughout the different stages by integrating multimodal reasoning and specialized skills from the foundation models with those gained from Grounding and CAPTCHA tasks.

Evan: How do they prevent issues when merging these independent expert models?

Ashley: Good question.

During this stage, they use an on-policy distillation paradigm, where teachers score trajectories sampled by the student model.

They focus on structured action-aware supervision to make sure the model learns the importance of different types of tokens, prioritizing executable actions over reasoning traces.

Evan: So, the supervision is tailored to the nature of GUI tasks?

Ashley: Exactly.

They adapt the weighting of the distillation signal based on the correctness of the student’s actions.

For instance, if the entire action is correct, they suppress the distillation signal for that action.

If only the action type is correct but its parameters are not, they strengthen supervision over the action span.

Evan: That sounds very meticulous.

How do they finally unify all these aspects into the model?

Ashley: The structured action-aware distillation essentially helps the model integrate broad multimodal reasoning capabilities with specific skills like precise spatial localization and robust navigation.

By concentrating the distillation signal on critical executable behaviors, they're able to create a unified model that performs effectively across diverse GUI environments.

Evan: This is all very detailed.

It seems like a well-rounded approach to developing a robust GUI agent.

Ashley: Indeed.

By offering a capable, efficient, and open-source foundation, UI-Venus-2 aims to advance the field toward more verifiable and self-reflective agents for real-world applications.

Evan: And that's the end of the Method section.

Stay tuned as we move on to the experimental results and evaluations in our next segment.

Evan: Alright Ashley, let's move on to the experiments and results section.

How did UI-Venus-2 perform across different benchmarks?

Ashley: UI-Venus-2 was evaluated across several benchmarks, covering mobile, desktop, web environments, and CAPTCHA-solving tasks.

Let's dive into them one by one.

Evan: Sounds good.

Let's start with mobile use.

How did UI-Venus-2 perform there?

Ashley: UI-Venus-2 excelled in mobile usage scenarios.

On the MobileGym benchmark, which provides a controlled, browser-based simulation environment, UI-Venus-2-27B achieved a success rate of 60.5%, while the smaller UI-Venus-2-9B model reached 52.7%.

These figures outperform the strongest baseline, Seed2.0 Pro, which stands at 52.0%.

Evan: Impressive.

What about a more realistic setting like VenusBench-Mobile?

Ashley: VenusBench-Mobile, which focuses on realistic, user-centric evaluations, showed UI-Venus-2-27B with a success rate of 48.7% and UI-Venus-2-9B at 46.5%.

Both models significantly outperformed the strongest prior baseline, Claude-Opus-4.6, which had a success rate of 36.5%.

Evan: What about Android-specific benchmarks?

Ashley: On AndroidWorld, which spans a variety of real-world Android applications, UI-Venus-2-27B achieved 84.0%, and UI-Venus-2-9B hit 80.2%.

This edges out the previous best, UI-Venus-1.5-30B-A3B, which had a success rate of 77.6%.

Evan: Nice.

How did they perform on more challenging mobile benchmarks like MobileWorld?

Ashley: MobileWorld is indeed more challenging due to its long-horizon and cross-application task structure.

Here, UI-Venus-2-27B achieved a success rate of 76.1% in the standard 50-step setting, while UI-Venus-2-9B achieved 65.8%.

Under the extended 100-step setting, the performances increased to 82.9% and 75.2%, respectively.

Although they trailed behind the top performer, Qwen-UI-Agent-27B, at 82.1%, they still performed admirably.

Evan: Got it.

What about benchmarks that require personalized assistance and memory retention?

Ashley: For tests like KnowUBench, which evaluates personalized mobile assistance, UI-Venus-2-27B achieved a success rate of 59.7%, while UI-Venus-2-9B scored 56.5%.

Both surpassed the previous best, Seed2.0 Pro, at 51.6%.

For memory-focused benchmarks like MemGUI, UI-Venus-2-27B achieved the highest score of 70.3%, with UI-Venus-2-9B following at 62.6%.

Evan: That’s quite comprehensive.

Let's switch gears to desktop environments.

How did UI-Venus-2 fare there?

Ashley: In desktop environments, UI-Venus-2 also showed robust performance.

On OSWorld-Verified, which evaluates real computer interactions, UI-Venus-2-27B scored 80.5%, close to the leading model Claude-Opus-4.8 at 83.4%.

The smaller 9B model achieved a respectable score of 70.8%.

Evan: What about more complex desktop tasks?

Ashley: For DeskCraft, which focuses on long-horizon professional workflows and collaboration, UI-Venus-2-27B led with a success rate of 55.5%, outperforming the best baseline Kimi-K2.6 at 41.4%.

The 9B model also did well, scoring 48.0%.

Evan: And in long-horizon scenarios like OSWorld 2.0?

Ashley: For OSWorld 2.0, UI-Venus-2's performance was measured by Binary Accuracy and Partial Score.

The UI-Venus-2-27B achieved a Binary Accuracy of 2.8% and a Partial Score of 13.2%.

The 9B model scored 0.0% for Binary Accuracy and 7.5% for Partial Score.

These results highlight areas for possible future improvement.

Evan: Indeed.

Now, moving to web navigation.

How did UI-Venus-2 perform there?

Ashley: On WebVoyager, which evaluates end-to-end web navigation, UI-Venus-2-27B scored 93.4%, the highest among all evaluated models, including the previous leader Fara1.5 at 89.3%.

The 9B variant also did very well, achieving 90.8%.

Evan: That's excellent.

What about benchmarks like Online-Mind2Web that test realistic, dynamic online environments?

Ashley: In Online-Mind2Web, UI-Venus-2-27B again led the performance with 78.3%.

The 9B version was not far behind, scoring 74.0%.

Both outperformed the previous best, Fara1.5-27B, which had a score of 72.3%.

Evan: How did they handle replica environments like REAL?

Ashley: For REAL, which provides deterministic high-fidelity simulations, UI-Venus-2-27B achieved a new state-of-the-art score of 80.2%.

The 9B model attained a strong score of 76.9%, both surpassing the previous best of 74.4% by a fair margin.

Evan: Long-horizon benchmarks like Odysseys must have been challenging.

How did they fare?

Ashley: In Odysseys, a benchmark for long-horizon, multi-step workflows, UI-Venus-2-27B achieved top scores with 80.4 in averaged rubric score and 66.3 in perfect rubric score.

This was distinctly higher than the previous best baseline.

Evan: What about GUI grounding capabilities?

Ashley: For GUI grounding, they evaluated on multiple benchmarks.

For example, on VenusBench-GD, UI-Venus-2-27B achieved 80.1% accuracy, surpassing the previous best.

Similar strong performances were noted on ScreenSpot-Pro, OSWorld-G-R, and UI-Vision benchmarks.

Evan: And how did they handle CAPTCHA solving?

Ashley: On CAPTCHA benchmarks, UI-Venus-2-27B achieved a top score of 79.9% Pass@1 on VenusBench-CAPTCHA, and the 9B model scored 78.1%.

Both models significantly outperformed the previous leaders like Qwen3.6 at 53.0%.

Similar top performances were observed across MCA-Bench, Spatial-CAPTCHA-Bench, and NextGen-CAPTCHAs.

Evan: All in all, UI-Venus-2 seems to demonstrate robust performance across the board.

Ashley: Indeed, Evan.

And that's the end of the Experiments and Results section.

Stay with us as we delve into more insights in the next segment.

Evan: Alright, Ashley, let's move on to the Related Work section.

How does UI-Venus-2 fit into the landscape of existing GUI agents?

Ashley: To understand the context, it's important to note that the field of GUI agents has seen a lot of growth.

Early works primarily focused on platform-specific APIs or handcrafted workflows, which demanded extensive labor and were not easily generalizable.

In recent years, multimodal GUI agents have emerged as an interface that can bridge high-level user intent and concrete digital execution.

Evan: I see.

What were some of the significant advancements leading up to UI-Venus-2?

Ashley: A key shift was the development of Multimodal Large Language Models, or MLLMs, which leverage both visual and textual data to enhance their capabilities.

Researchers like Bai et al.

in 2025 and Liu et al.

in 2024 have contributed significantly in this area.

These models are trained to perceive rendered interfaces and interact through actions like clicking, typing, and scrolling, rather than just text-based commands.

Evan: How has the approach to agent evaluation evolved over time?

Ashley: Initially, most evaluations were benchmark-oriented.

However, recent user-centric evaluations like VenusBench-Mobile in 2026 and OSWorld in 2024 have exposed gaps.

These evaluations showed that while agents might perform well on benchmarks, they often struggle in realistic, dynamic environments.

This gap drove the need for more comprehensive environment coverage, scalable task construction, and reliable reward verification.

Evan: And how does UI-Venus-2 address these needs?

Ashley: UI-Venus-2 builds upon the foundations laid by its predecessors, especially UI-Venus-1.5.

The earlier iterations pioneered a unified, end-to-end framework for GUI grounding and navigation.

They improved the agents through large-scale GUI mid-training and reinforcement learning.

UI-Venus-2 takes this further by systematically scaling environments, tasks, and feedback mechanisms.

Evan: Can you elaborate on the scaling of environments?

Ashley: To expand environmental coverage, UI-Venus-2 includes over 170 multilingual mobile apps and native desktop operating systems.

This broadens the range of scenarios the agent can handle, ensuring it performs reliably in diverse settings.

Evan: Interesting.

What about task construction and data verification?

Ashley: Task construction has become more sophisticated, employing a deep-research pipeline to generate functionally grounded instructions.

This ensures the tasks are not just plausible but executable under varying interface states.

For data verification, they introduced stronger evaluators that assess task completion using visual keypoints and multi-model voting, enhancing the reliability of reinforcement learning signals.

Evan: It's clear they integrated several advanced techniques.

How does their work compare to other state-of-the-art models?

Ashley: UI-Venus-2 compares favorably against various baselines, including Qwen, Claude, and Seed models.

For instance, it consistently outperformed these in mobile-use, desktop-use, web-navigation, and CAPTCHA-solving tasks.

By integrating capabilities from different domains and ensuring robust task completion verification, UI-Venus-2 sets a new benchmark for general-purpose GUI agents.

Evan: It sounds like UI-Venus-2's comprehensive approach has set it apart from its predecessors and peers.

Ashley: Exactly.

The combination of extensive environment coverage, sophisticated task construction, and reliable data verification has allowed UI-Venus-2 to advance the field significantly.

Evan: And that's the end of the Related Work section.

Stay tuned as we move on to more insights in the next segment.

Evan: Alright Ashley, let's wrap up by summarizing the key contributions and takeaways from the UI-Venus-2 paper.

Ashley: Sure, Evan.

To start, UI-Venus-2 represents a significant advancement in the development of general-purpose GUI agents.

It builds on previous efforts by expanding not just the environment coverage but also the depth of task construction and verification.

Evan: Right.

They managed to scale the environment coverage to over 170 multilingual mobile apps and native desktop applications, which is impressive.

Ashley: Exactly.

They also developed a deep-research pipeline to generate grounded instructions for tasks, ensuring high accuracy and executability.

This approach helps the agent perform reliably even as interface states change.

Evan: And for verification, they introduced trace- and sample-level evaluators using visual keypoints and multi-model voting.

This brings more reliability to reinforcement learning signals.

Ashley: Yes, and don't forget their impact on various benchmarks.

UI-Venus-2 showed robust performance across mobile, web, desktop, and CAPTCHA-solving tasks, often outperforming the existing state-of-the-art models.

Evan: Precisely.

It's this comprehensive, well-rounded approach that makes UI-Venus-2 stand out in the field.

Ashley: In summary, by offering a capable, efficient, and open-source foundation, UI-Venus-2 advances the development of more generalizable, verifiable, and self-reflective GUI agents.

Evan: And that's a wrap for today's episode.

Thanks for tuning in to Daily Paper Cast.

Ashley: We hope you found our discussion on UI-Venus-2 insightful.

Be sure to join us again for more deep dives into cutting-edge research.

Evan: And remember, you can find the full paper and more details in our show notes.

Until next time, stay curious and keep exploring!

Ashley: Take care, and see you in the next episode.