Omni Interaction Agent Technical Report

🤗 Upvotes: 57 | eess.AS, cs.AI, cs.LG, cs.MM, cs.SD

Authors:
Orantqing, Shengpeng Ji, Junlong Tong, Jialong Zuo, Dongjie Fu, Di Cao, Yangzhuo Li, Shangda Wu, Franz, Evan, Theron Veyra, Changhao Pan, Jingyu Lu, Dongchao Yang, Zhifei Xie, Yang Tan, Xiaoyu Shen, Xiaoda Yang, Wenfu Wang, Teddysun, Steveyves, Zhou Zhao, Bryanytian

Title:
Omni Interaction Agent Technical Report

Arxiv:
http://arxiv.org/abs/2609.08977v1

Abstract:
In this work, we present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework. In contrast to turn-based conventional paradigms, Gander continuously receives streaming inputs across multiple modalities, including video, speech, and text, enabling natural full-duplex interaction in both everyday conversations and complex workflow-oriented agent scenarios. Users can interrupt the model at any time, while the model can also proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) It employs a Cerebellum-Brain collaborative framework, in which the Cerebellum is responsible for realtime interaction and omni conversational capabilities, while the Brain handles complex reasoning and higher-level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, user inputs and model outputs are further flattened into an ordered token stream at the chunk level, providing a unified representation for low latency, continuous interaction. We conduct comprehensive evaluations of Gander across four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence. Internal human evaluations demonstrate that Gander maintains the natural and expressive spoken dialogue capabilities of SOTA open source models while achieving competitive performance in omni interaction. Gander also demonstrates robustness in challenging real-world scenarios, including background noise interference, multi-party interactions, and backchannel communication. We release Gander together with its models, code, and data to facilitate further research and development in the community.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Ashley: Today’s paper, from Hugging Face's daily list of September 9, 2026, has 57 upvotes.

Evan: The title is 'Omni Interaction Agent Technical Report'.

Ashley: The first two authors are Orantqing and Shengpeng Ji, with Shengpeng Ji as the corresponding author.

They are part of the Hunyuan Speech Team at Tencent.

Evan: Alright, let's dive into the Introduction.

So, Ashley, what kind of context does this paper set for us?

Ashley: Sure, Evan.

The paper starts by contextualizing the evolution of large language models, or LLMs.

Initially, they were mostly passive interfaces for tasks like question answering and basic conversation.

But now, they're transforming into multi-functional agents.

Evan: Interesting.

What kind of functionalities are these LLMs incorporating now?

Ashley: The paper mentions that LLMs are expanding into areas like tool use, environmental interaction, and multi-step planning.

This transition allows them to not just respond to queries but to actively engage in more complex workflows.

Evan: Got it.

So, what exactly does this new paper propose regarding these evolving capabilities?

Ashley: This work introduces 'Gander,' an end-to-end model designed specifically to unify omni perception, real-time interaction, and agentic capabilities within a single framework.

Evan: And how is Gander different from existing systems?

Ashley: Unlike conventional turn-based paradigms, Gander continuously receives streaming inputs across multiple modalities like video, speech, and text, facilitating natural, full-duplex interactions.

Essentially, users can interrupt the model at any time, and the model can proactively provide feedback or ask follow-up questions.

Evan: That sounds pretty advanced.

How do they implement this?

Ashley: They employ a Cerebellum-Brain collaborative framework.

The 'Cerebellum' handles real-time interaction and conversational capabilities, while the 'Brain' manages complex reasoning and higher-level tasks.

The interaction between these two components is continuous and dynamic.

Evan: So they’ve split the responsibilities between two main modules.

What are the benefits of that?

Ashley: Splitting the responsibilities allows each module to specialize.

The Cerebellum handles the immediate, low-latency requirements of interaction, while the Brain can focus on more computationally intensive tasks like reasoning and planning, all without compromising on performance.

Evan: You mentioned continuous and dynamic interaction.

Can you elaborate on that?

Ashley: Gander uses a 'streaming Thinker-Talker' architecture where user inputs and model outputs are turned into an ordered token stream.

This setup supports low-latency, continuous interaction, making back-and-forth communication much more fluid and natural.

Evan: Sounds quite robust.

What aspects of Gander did they evaluate to support their claims?

Ashley: They conducted comprehensive evaluations across four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence.

Internal human evaluations show that Gander not only maintains the natural and expressive spoken dialogue capabilities of state-of-the-art open-source models but also provides strong performance in omni interaction.

Evan: Were there any challenging scenarios they tested this on?

Ashley: Yes, they tested Gander's robustness in real-world scenarios including background noise interference, multi-party interactions, and backchannel communication.

These scenarios are notoriously challenging for interactive systems, making the demonstrated robustness quite impressive.

Evan: Wow, that's a comprehensive evaluation.

Is the system available for public use?

Ashley: Yes, they’ve released Gander along with its models, code, and data to support further research and development in the community.

Evan: It sounds like a significant step forward.

And with that, we've covered the Introduction section of the paper.

Evan: Alright, Ashley, let's move on to the methods behind Gander.

How is the system structured to achieve its objectives?

Ashley: The architecture of Gander is divided into three main components: the front cerebellum, the agent orchestration runtime, and the back brain.

Each of these plays a specialized role in the system.

Evan: Interesting.

Let’s start with the front cerebellum.

What exactly does it do?

Ashley: The front cerebellum acts as a real-time, full-duplex multimodal model.

It handles continuous interaction by processing streaming speech, camera, and screen inputs.

Its job is to determine interaction actions like listening, speaking, and tool invocation, all in real time.

Evan: And how does it process this continuous stream of inputs?

Ashley: The front cerebellum uses a Thinker-Talker architecture.

Essentially, it receives streaming audio-visual input, which is flattened into an ordered chunk stream.

This approach allows it to dynamically decide whether to listen or speak at each chunk, offering a fluid, continuous interaction experience.

Evan: Got it.

So what's the role of the agent orchestration runtime?

Ashley: The agent orchestration runtime is the coordination layer that manages interactions between the cerebellum and the back brain.

It handles the real-time multimodal data transport and asynchronous background task orchestration, acting as a bridge to ensure seamless interaction between the two components.

Evan: How does the front cerebellum know when to delegate tasks to the back brain?

Ashley: Good question.

The front cerebellum dynamically decides based on the current interaction context.

Simple conversational and short-horizon tasks are managed by the cerebellum itself.

For more complex workflows that involve multi-step reasoning or external tools, it delegates to the back brain.

This delegation mechanism is formalized through structured tool calls that follow a specific format.

Evan: What happens once a task is delegated?

Ashley: Once a task is delegated, the cerebellum forwards the transcribed text of the user query along with the relevant video frames to the orchestration runtime.

The back brain then performs deeper reasoning asynchronously and can return intermediate summaries, plans, or final results.

The cerebellum integrates these outputs back into the ongoing interaction.

Evan: This sounds quite sophisticated.

Can you explain a bit more about the back brain?

Ashley: The back brain is a task execution agent responsible for complex reasoning and long-horizon workflows.

Think of it as the powerhouse behind the system that performs in-depth tasks like information retrieval, code and document processing, and long-term planning.

Interestingly, it's designed to be training-free and can incorporate stronger reasoning models without retraining the core interaction model.

Evan: And how do they handle the interaction between these three components?

Ashley: The interaction is managed through an intermediate layer known as the agent orchestration runtime.

This runtime supports two control modes: lean and coordinator.

In lean mode, the runtime directly executes the task actions classified by the cerebellum, while the coordinator mode introduces an independent control plane model that generates execution directives.

Evan: That’s quite detailed.

What about the perceptual and interaction capabilities of the cerebellum?

Ashley: Those capabilities are built on two modality encoders that run concurrently – one for the visual stream and another for the acoustic stream.

Both operate incrementally over fixed-duration windows.

This way, visual and auditory observations are continuously refreshed and exposed to the backbone at every step of the streaming exchange.

Evan: How do they manage to keep this process efficient in real time?

Ashley: For visual perception, each frame is first decomposed into slices, which are independently encoded by a vision transformer.

The resulting patch features are condensed into a smaller number of visual tokens per slice for efficiency.

For acoustic perception, the audio stream is downsampled to a manageable rate before entering the backbone, ensuring real-time processing.

Evan: And how does the system determine when to interact?

Ashley: The system uses a flattened interaction formulation, where all modalities and outputs are serialized into one causal token stream.

From here, it can decide whether to generate responses, remain silent, or interrupt an ongoing utterance based on incoming data.

Evan: Quite a dynamic process.

How do they ensure the spoken interaction matches the context effectively?

Ashley: Spoken interaction is managed through a speech token decoder that produces discrete speech units for both prosody and context.

These are then converted into continuous audio waveforms by a flow-matching decoder, which keeps the interaction responsive and contextually appropriate.

Evan: What unique data do they use for training this system?

Ashley: To train Gander, they construct a large-scale corpus that includes real-time speech interaction, audio-visual interaction, and agentic interaction data.

They also include robustness-oriented data to cover complex scenarios like background noise, multi-party interactions, and various interruption events.

Evan: What about the quality of these interactions?

How do they maintain reliability?

Ashley: They put a significant focus on quality control.

For instance, in the speech interaction data, dialogues are synthesized to replicate realistic scenarios, including interruptions and overlapping speech.

Additionally, these dialogues undergo multiple rounds of quality filtering to ensure that the interaction points are linguistically plausible and coherent.

Evan: Any special considerations for multi-modal data?

Ashley: Yes, for audio-visual data, they use systems like Qwen3-5-297B-A17B to refine alignments and ensure high-quality, time-aligned targets.

This refinement helps in better integrating the visual evidence with spoken interactions.

Evan: And the agentic interaction data, how are they assembled?

Ashley: Agentic interaction data are created using a seed-driven synthesis pipeline for audio-based tasks and visually grounded tasks.

These interactions are simulated to span the full task lifecycle, ensuring that the front cerebellum can manage ongoing interactions while coordinating with the back brain for asynchronous tasks.

Evan: Sounds like a comprehensive setup.

Does the system have real-world robustness?

Ashley: Yes, the robustness data includes challenging conditions like irrelevant visual context, no-command environments, and speaker tracking in multi-party scenarios.

These help the model stay reliable and ignore distractions, which is crucial for practical usage.

Evan: This must require quite a bit of computational resources.

How do they handle the scalability?

Ashley: Scalability is addressed by the architecture itself, particularly the separation of responsibilities between the cerebellum and the brain.

By splitting real-time interaction from more complex reasoning tasks, the system stays efficient without compromising on its multi-agent capabilities.

Evan: That's impressive.

So, essentially, Gander creates a unified, continuous interaction framework that can handle both real-time and complex tasks by leveraging a modular architecture.

Ashley: Exactly.

And with that, we’ve covered the Method section of the paper.

Evan: Ashley, let’s discuss the Experiments and Results section.

How did they validate Gander’s capabilities?

Ashley: They conducted comprehensive evaluations along three main axes: full-duplex interaction with tool use, spoken conversational ability, and omni understanding over audio-visual streams.

Let's start with the full-duplex interaction.

Evan: Sure.

How did they evaluate the full-duplex interaction?

Ashley: They used Full-Duplex-Bench v3, which evaluates a spoken assistant's performance in tool-augmented service scenarios containing natural disfluencies.

They tested Gander across 100 scenarios, focusing on metrics like tool-selection accuracy, argument correctness, response quality, and several interaction-specific metrics.

Evan: What were the results like?

Ashley: Gander performed exceptionally well in turn-taking behavior, achieving a 100% rate in taking turns at appropriate moments.

It also had the lowest premature interruption rate at 8%.

However, it did have a higher filler rate, around 51.6%, due to its strategy of holding the floor during back brain tasks.

Evan: How did it perform on task accuracy metrics?

Ashley: In terms of task accuracy, Gander had a Pass@1 rate of 40.0%, ToolSel accuracy of 75.9%, and Argument Correctness of 50.3%.

While these are good, they aren't at the top of the table.

The back brain, when evaluated in isolation, achieved higher metrics, indicating room for improvement in the interaction between the cerebellum and the back brain.

Evan: And how about the spoken conversational ability?

Ashley: For spoken conversation, they evaluated Gander on the SpokenQA and VoiceBench datasets.

Here, Gander led the full-duplex models on both knowledge-oriented subsets, achieving 75.60% on Llama Questions and 59.30% on Web Questions.

It placed second within the group on voice quality evaluations, showing that it maintained competitive conversational abilities despite its streaming requirements.

Evan: That’s impressive.

How did it do in the omni understanding benchmarks?

Ashley: They evaluated Gander on WorldSense and Daily-Omni benchmarks, focusing on its ability to integrate audio-visual data.

Gander scored 49.62% on WorldSense and 78.53% on Daily-Omni.

These results are competitive and indicate that Gander effectively utilizes both audio and visual inputs in its reasoning process.

Evan: Did the evaluations reveal anything about how well Gander integrates multiple modalities?

Ashley: Yes, actually.

An interesting finding is the fusion gain observed in their ablation studies.

When audio and video inputs were presented jointly, Gander performed notably better than with either input alone, especially in benchmarks that required temporal alignment of audio-visual streams.

Evan: What does that mean for its use in real-world applications?

Ashley: It means Gander is quite effective in environments where understanding context from both audio and visual inputs is crucial.

This fusion gain shows its strength in situations that need multi-modal integration for better understanding and interaction.

Evan: It sounds like they've built a robust system.

Did they mention any challenges or areas for improvement?

Ashley: Yes, although Gander showed strong performance, the paper notes some challenges.

Particularly, the need for improved integration between the front cerebellum and the back brain for task execution.

They also identified the importance of scaling data and model capacity for better robustness and generalization in complex scenarios.

Evan: And how does Gander compare to existing systems?

Ashley: The Full-Duplex-Bench results reveal that while Gander excels in interactive capabilities, some existing systems outperform it in specific task accuracy metrics.

This highlights the need for balanced optimization between real-time interaction and complex task handling.

Evan: So, to sum up the experiment section, Gander shows promise but also presents areas ripe for further development.

Ashley: Exactly, Evan.

And that concludes the Experiments and Results section.

Evan: Alright, Ashley, let's talk about the related work in the field.

How does Gander fit into the broader landscape of AI research?

Ashley: The paper organizes related work into three main areas: audio interaction models, omni interaction models, and voice agent systems for practical task execution.

Each of these areas contributes to the capabilities seen in Gander.

Evan: Let's start with audio interaction models.

What have been the recent advancements in this area?

Ashley: Recent advancements in audio interaction models focus on moving beyond turn-based dialogues to full-duplex, real-time interaction.

Early models used systems like Voice Activity Detection to define when a user begins and ends an utterance.

However, newer models like BayLing-Duplex incorporate interaction timing directly into the model, enabling decisions on when to listen, speak, or terminate a response during ongoing speech generation.

Evan: How does Gander expand on these developments?

Ashley: Gander builds on these advancements by using a streaming Thinker-Talker architecture.

This offers more granular control over real-time interaction by continuously observing and generating speech, blending listening and speaking actions in a fluid manner.

Evan: Interesting.

What about omni interaction models?

How do they contribute to Gander's capabilities?

Ashley: Omni interaction models integrate multiple modalities like audio and video into a unified representation.

Systems like Qwen Omni and MiniCPM-o 4.5 have established foundational work in this area by jointly interpreting linguistic, acoustic, visual, and temporal information.

This enables models to reason about not just speech, but also the surrounding visual context and its evolution over time.

Evan: How does Gander leverage these ideas?

Ashley: Gander leverages these ideas by adopting a comprehensive multimodal approach.

It uses concurrent modality encoders for visual and acoustic streams, ensuring that visual and auditory observations are continuously refreshed and integrated into the interaction context.

This allows Gander to effectively understand and respond to evolving multimodal inputs.

Evan: It sounds like a solid integration.

How do voice agent systems for practical task execution fit into this picture?

Ashley: Voice agent systems aim to execute practical tasks based on voice commands.

Early systems were confined to simple command-response interactions.

More advanced ones, like GPT-Live and Qwen Audio Agent, extend this by incorporating runtime environments where agents can execute complex workflows, generate code, manipulate files, and more.

Evan: And what’s Gander’s approach here?

Ashley: Gander takes this a step further by integrating both interactive conversational capabilities and task execution within a single unified framework.

The agent orchestration runtime ensures that tasks requiring complex reasoning and planning are efficiently handled by the back brain, while the front cerebellum maintains seamless interaction with the user.

Evan: Do they address the challenges these systems typically face?

Ashley: Yes, Gander addresses key challenges like interruption management, proactive interaction, multi-party conversation handling, and context retention.

This makes it capable of managing conversations that involve spontaneous user inputs and dynamic task requirements, much like a human assistant.

Evan: Quite comprehensive.

Are there any specific prior works the paper mentions as particularly impactful?

Ashley: The paper acknowledges a variety of prior works.

For audio interaction, it references models like BayLing-Duplex and Qwen-Audio-3.0-Realtime.

For multimodal interaction, systems like Qwen Omni and MiniCPM-o 4.5 are highlighted.

In practical task execution, works like GPT-Live and Qwen Audio Agent are notable mentions.

Evan: So, to wrap up the related work section, Gander builds on a rich landscape of research, incorporating advances in real-time audio interaction, multimodal understanding, and voice-based task execution.

Ashley: Exactly, Evan.

And that concludes the Related Work section.

Evan: Alright, Ashley, let’s wrap up this discussion by summarizing the key contributions and takeaways from the paper.

Ashley: Sure, Evan.

The paper, 'Omni Interaction Agent Technical Report,' presents 'Gander,' an end-to-end model designed to unify omni perception, real-time interaction, and agentic capabilities within one framework.

Evan: Right.

Gander utilizes a Cerebellum-Brain collaborative framework to separate real-time interaction and complex reasoning.

The front cerebellum handles streaming inputs and real-time conversational control, while the back brain manages long-horizon, agentic tasks.

Ashley: Exactly.

This architecture allows Gander to support continuous, low-latency interactions, and complex multi-step tasks, providing a seamless and fluid user experience.

Its Thinker-Talker model predicts interaction behaviors on the fly, enhancing responsiveness and interaction flow.

Evan: Moreover, the system was evaluated across several dimensions, including conversational ability, omni understanding, interactive capability, and agentic intelligence.

Gander demonstrated competitive performance across these different metrics, particularly excelling in turn-taking and interaction timing.

Ashley: The ability to integrate audio and visual inputs effectively is a standout feature, as evidenced by the fusion gains in the benchmarks.

This makes Gander highly suitable for complex, dynamic environments where multimodal understanding is crucial.

Evan: And the authors have made the models, code, and data publicly available, encouraging further research and development in the field.

Ashley: That’s right.

So, that’s a wrap on Gander, an impressive step forward in AI interaction models.

We hope you found our discussion insightful.

Evan: Stay tuned for our next episode, where we'll dive into another exciting paper from the forefront of AI research.

Thanks for listening to Daily Paper Cast.

Ashley: Until next time, keep exploring and stay curious.

Goodbye!

Evan: Goodbye, everyone!