VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction

🤗 Upvotes: 51 | cs.CV

Authors:
Qiutong Chen, Yuchan Guo, Zhenlong Yuan, Haobo Yang, Fangfang Lin, Xinyi Long, Yin Wang, Zijian Song, Rui Lan, Shi Qiu, Boyuan Pan, Yang Luo, Yuyin Zhou

Title:
VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction

Arxiv:
http://arxiv.org/abs/2610.06293v1

Abstract:
Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in video understanding, yet their reliance on retrospective summarization and text-centric priors often limits their ability to bridge unobserved causal transitions when applied to Video Event Prediction (VEP). To address this, we propose VepAgent, an agentic framework that integrates causal-transition reasoning with tool-augmented reinforcement learning (RL) for robust VEP. Unlike prior methods that passively project future trajectories from historical dependencies, our approach explicitly models the logical progression from terminal observed states to future events. Specifically, we first construct futurebench-4K, a high-quality chain-of-thought dataset for supervised fine-tuning (SFT) that effectively bridges the causal-logic gap by structuring the deduction of unobserved intermediate states. Subsequently, we develop a diagnostic tool library integrating state tracking, frame retrieval, and region magnification, enabling the agent to dynamically augment reasoning with external tools to recover missing spatio-temporal evidence and resolve visual ambiguities during inference. Moreover, we propose a composite reward mechanism that jointly optimizes prediction accuracy, causal coherence, and reliable prior, compelling the agent to rely on genuine visual grounding rather than superficial textual similarities. Extensive evaluations on FutureBench and NEPBench datasets demonstrate that our method achieves state-of-the-art performance, significantly outperforming larger MLLMs and validating the empirical effectiveness of our agentic, future-oriented reasoning paradigm.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Evan: Today's paper is from the Hugging Face daily paper list of October 8, 2026, and it has 51 upvotes.

Ashley: The title is 'VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction'.

Evan: The first two authors are Qiutong Chen from Nankai University and Yuchan Guo from Carnegie Mellon University.

The corresponding author is Yuyin Zhou from the University of California, Santa Cruz.

Evan: Alright, Ashley, let’s dive into the Introduction.

What’s the background and main goal of their research?

Ashley: Sure, Evan.

The paper addresses Video Event Prediction, or VEP, which involves anticipating plausible future events based on partially observed visual evidence.

This forward-looking capability is critical for next-generation applications like proactive robotics, autonomous driving, and intelligent surveillance.

Evan: Interesting.

But what’s the main challenge in achieving effective VEP?

Ashley: The main challenge is developing agents that can bridge observed reality with unobserved futures under incomplete visual evidence.

Despite advances in Multimodal Large Language Models, or MLLMs, these models predominantly reconstruct historical event chains retrospectively.

This often results in a reliance on language-derived priors rather than visual evidence, making it difficult to predict future events accurately.

Evan: So the authors aim to improve this by leveraging a new approach?

Ashley: Exactly.

They propose VepAgent, an agentic framework that integrates causal-transition reasoning with tool-augmented reinforcement learning to enhance VEP.

Unlike traditional methods, VepAgent explicitly models the logical progression from the terminal observed state to future events.

Evan: Sounds sophisticated.

How do they ensure VepAgent can navigate through incomplete visual evidence?

Ashley: Great question.

They address it in a few specific ways.

First, they construct a high-quality dataset called futurebench-4K to fine-tune their model in a supervised manner.

This helps bridge the causal-logic gap by structuring the deduction of unobserved intermediate states.

Evan: And this futurebench-4K dataset, what's special about it?

Ashley: The futurebench-4K dataset contains chain-of-thought examples that explicitly demonstrate how to move from observed states through ongoing actions to future events.

This structured data helps the model understand and predict the logical sequence of events rather than merely summarizing past events.

Evan: So they’re actively teaching the model how to reason through these transitions using this dataset?

Ashley: Yes, exactly.

And to further improve the model’s reasoning during the reinforcement learning stage, they’ve introduced a diagnostic tool library.

This library includes tools for state tracking, frame retrieval, region magnification, and visual detail extraction.

Evan: How do these tools fit into the reinforcement learning framework?

Ashley: These tools are dynamically used by the model during inference to augment its reasoning with external evidence.

For example, frame retrieval helps recover missing spatio-temporal details, and region magnification helps resolve visual ambiguities.

Evan: That sounds like it adds a lot of complexity.

How do they ensure all these components work together effectively?

Ashley: They also propose a composite reward mechanism that optimizes prediction accuracy, causal coherence, and reliable prior.

This discourages the model from relying on superficial textual similarities and encourages genuine visual grounding.

Evan: Got it.

So this composite reward mechanism ensures the model stays focused on visual evidence rather than taking shortcuts based on text patterns?

Ashley: Exactly.

By using these combined strategies, the VepAgent framework is designed to predict future events in a more grounded and accurate manner.

Evan: It sounds like they've thoroughly covered the challenges associated with video event prediction.

That ends the Introduction section.

Evan: Now that we've covered the introduction, let’s move on to the methodology behind VepAgent.

So, Ashley, can you break down how the authors structured the approach to enhance video event prediction?

Ashley: The methodology presented in this paper is multifaceted, involving several stages of training and tool integration to improve prediction accuracy and causal reasoning.

They start by defining the problem formulation wherein the goal is to predict the most plausible future event from a given partial video observation, a query, and a set of candidate future events.

Evan: So how do they build the actual workflow for VepAgent to achieve this goal?

Ashley: Great question.

The authors propose a two-stage workflow that combines observation with dynamic evidence acquisition.

Evan: Interesting.

Can you walk us through these stages and how they operate?

Ashley: Certainly.

Stage 1 is called 'Observation & Tool Selection'.

Here, the model first processes the partial video and the query to estimate the terminal observed state and identify any ongoing action.

If any critical transitions are ambiguous, the framework then decides which tools from their diagnostic library should be used to fill in the gaps.

Evan: So the model doesn’t always use all the tools?

Ashley: Exactly.

The model selectively uses tools based on the requirements of the specific situation.

Their tool library includes a State Transition Tracker for summarizing event sequences, a Focused Evidence Retriever for dense frame extraction, a Region Focus Magnifier for local detail inspection, and a Visual Detail Extractor for converting fine-grained visual facts into textual data.

Evan: That sounds comprehensive.

What happens next?

Ashley: This brings us to Stage 2: 'Causal Bridging & Tool Augmentation'.

The selected diagnostic tools are executed and their outputs are incorporated into the model’s reasoning process.

This enhanced reasoning context is then used to make the final future event prediction.

Evan: So it's really a dynamic and interactive process.

How exactly do they manage the training to ensure this setup works effectively?

Ashley: The training involves two key phases: Supervised Fine-Tuning, or SFT, and Group Relative Policy Optimization, or GRPO.

Initially, they leverage a new dataset called futurebench-4K for SFT.

This dataset includes structured chain-of-thought examples to teach the model about causal transitions from observed states to future events.

Evan: And then they move on to reinforcement learning, correct?

Ashley: Yes, they do.

During the GRPO phase, the policy network is optimized to dynamically interact with the diagnostic tools, retrieving additional evidence as needed.

The composite reward mechanism in place ensures that the model’s predictions are accurate, causally coherent, and grounded in visual evidence rather than language priors.

Evan: That's fascinating.

But how do they ensure the predictions are both accurate and reliable?

Ashley: They design a composite reward structure for reinforcement learning.

This includes an accuracy reward to ensure correct predictions, a causal gap reward to enforce logical event progression, and an anti-prior reward to penalize reliance on language priors.

The format reward serves as a hard gate to ensure predictions follow the structured reasoning format.

Evan: It seems like a lot of thought has gone into making the model robust.

How does the diagnostic tool library function in this context?

Ashley: The diagnostic tools are integrated as part of the agentic reinforcement learning framework.

They are dynamically invoked based on the causal gaps that need to be bridged.

For instance, the Focused Evidence Retriever addresses issues of missing temporal data by extracting dense frames within a specific window, ensuring no critical actions are missed.

The Region Focus Magnifier helps in enhancing minute visual details to clarify the observed state.

Evan: Can you give a bit more detail about how each tool functions?

Ashley: Sure.

The State Transition Tracker re-analyzes the video to clarify the sequence of events, transforming observed state trajectories into a clear textual narrative.

The Focused Evidence Retriever extracts detailed frames to fill temporal gaps.

The Region Focus Magnifier zooms in on critical areas to resolve local ambiguities.

Lastly, the Visual Detail Extractor converts visual nuances into textual descriptions that anchor the final reasoning process.

Evan: And these tools only operate on the observed portion of the video, right?

Ashley: Exactly.

They do not have access to future frames, ensuring that any prediction is based on visual evidence currently available, thus avoiding any future information leakage.

Evan: That’s an intelligent design to prevent biases.

How did they validate the effectiveness of VepAgent?

Ashley: For validation, the authors conducted extensive evaluations on FutureBench and NEPBench datasets.

They compare their model against existing state-of-the-art Multimodal Large Language Models and demonstrate significant improvements in prediction accuracy and reliability.

Specifically, VepAgent achieved higher average accuracy scores across multiple reasoning depths and diverse scenarios.

Evan: It certainly sounds like a robust system.

Summarizing the methodology, they’ve synergized multiple cutting-edge techniques in a dynamic, tool-augmented framework to significantly improve video event prediction.

Ashley: That's right, and these components—structured fine-tuning datasets, dynamic tool use during inference, and a well-balanced reinforcement learning reward system—are all critical to their approach.

That wraps up the Method section.

Evan: Now that we've gone over VepAgent’s methodology, let’s dive into the experiments and results.

Ashley, how did the authors validate the effectiveness of VepAgent?

Ashley: They conducted extensive evaluations using two public benchmarks: FutureBench and NEPBench.

These benchmarks cover various real-world scenarios requiring accurate future event prediction.

Evan: That sounds comprehensive.

What metrics did they use to measure success?

Ashley: They primarily focused on event prediction accuracy.

For FutureBench, they reported accuracy across different reasoning depths like 1-Hop, 2-Hop, and 3-Hop predictions, as well as interpolation accuracy and overall average accuracy.

For NEPBench, the results were broken down into specific subsets such as Charades, VidSitu, and YouCook2, along with an overall accuracy score.

Evan: Alright, let’s start with FutureBench.

How well did VepAgent perform compared to other models?

Ashley: In FutureBench, VepAgent achieved significant improvements.

The most notable result is its average accuracy of 81.44%, a substantial increase from the previous best model, Qwen3-VL-30B-A3B, which had a score of 66.86%.

VepAgent also outperformed other strong models like Qwen2.5-VL-32B and MiMo-VL-7B in all measured reasoning depths.

Evan: That's impressive!

It seems like VepAgent made quite a leap ahead of the competition.

How did it perform in NEPBench?

Ashley: On NEPBench, VepAgent also excelled.

For example, it achieved an overall accuracy of 70.20%, which is 22.70 percentage points higher than the best previous model, Qwen2.5-VL-72B.

Besides, VepAgent achieved high accuracy in individual subsets, notably 83.46% in Charades and 54.78% in VidSitu.

Evan: That’s a consistent performance.

Can you share any specific qualitative analyses or case studies the authors provided?

Ashley: Certainly.

The authors presented detailed case studies to illustrate where VepAgent made a difference.

For instance, in a scenario where the base model misinterpreted an ongoing cooking event, VepAgent used the State Transition Tracker to identify the correct ongoing action, leading to an accurate future prediction.

Evan: What about other tools like the Focused Evidence Retriever or Region Focus Magnifier?

Ashley: The Focused Evidence Retriever was particularly effective in scenarios where critical actions were missed due to uniform frame sampling.

It retrieved dense frames from relevant time windows, allowing the model to capture transient actions.

Similarly, the Region Focus Magnifier helped resolve ambiguities in local details, such as verifying whether an object was released or not.

Evan: That’s a smart approach to handle missing or ambiguous visual data.

Did they conduct any ablation studies to show the impact of each component?

Ashley: Yes, they did.

The ablation studies were quite revealing.

They showed that each tool in the diagnostic library contributed uniquely to the model’s accuracy.

For instance, removing the State Transition Tracker significantly reduced the accuracy, demonstrating its importance in resolving action sequences.

Evan: How about the reward mechanism adjustments during reinforcement learning?

Ashley: They also analyzed this aspect.

They compared different reward setups, finding that the full composite reward—combining accuracy, causal gap, and anti-prior rewards—produced the best results.

Omitting any component led to noticeable drops in performance.

Evan: So, these experiments comprehensively validated the robustness and effectiveness of VepAgent’s design.

Did they mention any plans for future work?

Ashley: While the paper was focused on validating VepAgent in the context of future-event prediction, they did suggest that the diagnostic tool library could be expanded and applied to broader video understanding tasks.

This would further test the generalizability of their approach.

Evan: It's good to hear that they are considering broader applications.

This concludes the Experiment section.

Evan: Let’s transition to the Related Work section, where the authors contextualize their research within the existing body of work.

Ashley, can you summarize the different areas they touch upon?

Ashley: Evan.

The Related Work section of the paper is quite comprehensive, spanning several domains closely related to their research.

They start with Video Event Prediction and Temporal Event Reasoning.

Evan: And what do they highlight there?

Ashley: They discuss how earlier efforts in this area have mainly focused on action anticipation and script event prediction based on structured datasets like EPIC-Kitchens and VidSitu.

These approaches often involve modeling dependencies between events to predict future occurrences.

Evan: Did they mention any specific models or frameworks?

Ashley: Yes, they do.

For instance, VLEP is noted for formulating next-event prediction from partially observed videos, and VidEvent from AA AI 2025 introduced structured video events for action-centric prediction.

EventFormer is another example that uses hierarchical dependencies to model temporal evolution.

Evan: And how does their approach with VepAgent differ from these existing methods?

Ashley: Unlike these prior works, VepAgent explicitly bridges the gap between observed and unobserved events by leveraging causal transitions and diagnostic tools to enhance visual evidence recovery.

They make a significant leap by focusing on dynamic tool augmentation and causal reasoning rather than just historical data summarization.

Evan: That makes sense.

What's the next area they explore in their Related Work section?

Ashley: Next, they cover Multimodal Large Language Model reasoning and reinforcement learning.

They discuss the advancements in large language models like OpenAI-o1 and DeepSeek-R1, which have shown the potential of reinforcement-learning-based post-training to enhance structured reasoning.

Evan: So they’re saying reinforcement learning has been pivotal in improving these models?

Ashley: Exactly.

Studies like LMM-R1 and Video-R1 apply reinforcement learning to mathematical visual question answering and video understanding, respectively.

This highlights the flexibility of RL in augmenting model reasoning beyond text-centric tasks.

Evan: And how does VepAgent leverage these principles?

Ashley: VepAgent integrates reinforcement learning with a specific focus on causal reasoning and visual grounding.

It uses structured fine-tuning followed by group relative policy optimization to train the model effectively.

Evan: That’s quite intricate.

What’s the third area they delve into?

Ashley: They discuss Tool-Augmented Agentic Systems.

Early works like FAST and MVoT introduced the concept of visual evidence integration into reasoning steps for image and multimodal tasks.

More recent studies, like VITAL, show how tool-augmented RL can improve task performance by dynamically acquiring visual evidence.

Evan: But how does VepAgent stand out in this context?

Ashley: VepAgent uniquely addresses the needs of video event prediction by integrating a specialized diagnostic tool library designed to resolve specific types of visual ambiguities, such as temporal gaps and local detail verification, which are crucial for predicting future events accurately.

Evan: So the tool augmentation in VepAgent is more targeted compared to other methods?

Ashley: Exactly.

VepAgent’s tool use is specifically designed to handle the unique challenges of VEP, dynamically invoking tools to bridge causal gaps and verify visual states.

Evan: What other topics do they cover?

Ashley: They also explore Vision-Language-Action models and Embodied Reasoning.

This area includes models integrating visual perception, language reasoning, and action generation, exemplified by work like Pelican-Unify and AutoDrive-R2, which emphasize the importance of structured perception and decision-making in dynamic environments.

Evan: And how is this relevant to VepAgent’s approach?

Ashley: It’s relevant because VepAgent similarly requires structured visual perception and reasoning to predict future events from partially observed evidence.

The methodological principles overlap, emphasizing the importance of grounding predictions in reliable visual data rather than text-derived shortcuts.

Evan: Do they mention any other areas?

Ashley: Yes, they dive into Spatio-Temporal Perception and Geometric Evidence.

Studies like SD-MVS and TSAR-MVS have looked into reconstructing fine-grained visual evidence from partial observations, an idea that aligns with VepAgent’s use of tools to recover missing temporal and spatial details in video frames.

Evan: Interesting.

What about human-centric interaction understanding?

Ashley: They touch upon this by referencing works like InstructHOI and open-vocabulary HOI comprehension, which emphasize understanding how human-object interactions evolve.

This is crucial for predicting future events in scenarios where human actions play a significant role.

Evan: So, they essentially align VepAgent within a broader context of various cutting-edge research areas?

Ashley: Yes, by weaving together insights from these diverse fields, the authors position VepAgent as a robust solution that addresses the limitations of current models while benefiting from advances in reinforcement learning, tool-augmented systems, and human-centric interaction understanding.

Evan: It sounds like a well-rounded approach.

That concludes the Related Work section.

Evan: Alright, we’ve covered a lot of ground today with VepAgent.

To wrap up, let’s summarize the key contributions and takeaways from this paper.

Ashley, what stands out the most to you?

Ashley: Certainly, Evan.

VepAgent makes several significant contributions to the field of Video Event Prediction.

First, it introduces an agentic framework combining causal-transition reasoning and tool-augmented reinforcement learning, which enhances the prediction accuracy and reliability in VEP tasks.

Evan: The use of the futurebench-4K dataset for supervised fine-tuning is another highlight.

This dataset provides structured chain-of-thought examples, helping the model bridge the causal-logic gap effectively.

Ashley: Right.

And the diagnostic tool library is a critical innovation.

It includes tools for state tracking, frame retrieval, region magnification, and visual detail extraction, which dynamically augment the model’s reasoning during inference.

These tools enable VepAgent to recover missing spatio-temporal evidence and resolve visual ambiguities to make more grounded predictions.

Evan: Indeed.

The composite reward mechanism enhances the reinforcement learning process by optimizing prediction accuracy, causal coherence, and anti-prior robustness.

This ensures the model focuses on genuine visual grounding rather than relying on superficial textual similarities.

Ashley: Exactly.

And through extensive experiments on FutureBench and NEPBench, VepAgent demonstrated significant improvements over existing state-of-the-art models, validating the empirical effectiveness of this approach.

Evan: These contributions show that VepAgent not only advances the capabilities of video event prediction but also sets a new standard for integrating causal reasoning with dynamic evidence acquisition.

It's a noteworthy step forward in the field.

Ashley: It’ll be exciting to see how future research builds upon these findings and explores broader applications of the diagnostic tool library.

Evan: Thank you all for tuning into today’s episode of Daily Paper Cast.

We hope you found this discussion on VepAgent insightful.

Ashley: Be sure to join us again for our next episode, where we’ll dive into another fascinating paper from the world of AI and machine learning.

Evan: Until then, keep exploring, stay curious, and see you next time!

Ashley: Goodbye!