DriveZero: End-to-End Driving Beyond Human Demonstrations

🤗 Upvotes: 43 | cs.CV

Authors:
Hao He, Chengcheng Hu, Zirun Su, Heng Zhang, Haisong Liu, Jinke Li, Haochen Tian, Zhenwei Shen, Hongyang Li, Zhichao Li, Yunchen Yang, Bochao Huang, Siyu Zhang, Kuangye Chen, Xiongjie Zhang, Wentao Dai, Hengchen Dai, Siyuan Liu, Zehao Huang, Naiyan Wang

Title:
DriveZero: End-to-End Driving Beyond Human Demonstrations

Arxiv:
http://arxiv.org/abs/2609.06055v1

Abstract:
Most end-to-end autonomous-driving systems learn by imitating human driving logs, leaving their learned behavior constrained by the quality and behavioral coverage of the recorded trajectories. This report presents DriveZero, an end-to-end system that learns driving behavior beyond human demonstrations. It decomposes driving into a perception model and an action model, pretrains each in the regime best suited to it, and combines them into one end-to-end planner. The two models call for different learning recipes: perception must understand the world, and benefits from massive and diverse visual data; action must interact with it, and requires closed-loop feedback. On the action side, we introduce DriveRL, a mixed-agent closed-loop reinforcement-learning framework. It converts real driving logs into interactive worlds, where a privileged teacher policy is trained with PPO through closed-loop rollouts. For the perception model, DriveVFM consolidates multiple frozen vision foundation models, including DINOv3, SigLIP2, SAM and Depth Anything V2, into a single backbone from raw images alone, requiring no task-specific annotations. DriveZero then unifies the two: a camera-only planner that distills the frozen DriveRL teacher through its rolled-out trajectories. The goal-conditioned teacher can moreover be queried under augmented driving intents, yielding diverse, goal-consistent supervision that logged data cannot provide. On nuPlan, DriveRL with value-guided test-time action search achieves a mean score of 93.57 across the Val14, Test14-hard, and Test14-random community splits in both non-reactive and reactive modes, exceeding the Log-Replay expert on all three splits. DriveZero achieves state-of-the-art performance on NAVSIMv1, NAVSIMv2 and the closed-loop HUGSIM benchmark without any human trajectory supervision.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast!

Ashley: Today we'll delve into a recent paper from the Hugging Face daily paper list of September 9, 2026, which has garnered 43 upvotes.

Evan: The paper is titled 'DriveZero: End-to-End Driving Beyond Human Demonstrations.'

Ashley: It's authored by Hao He and Chengcheng Hu as the first two authors, with Zirun Su as the corresponding author, all from Xiaomi EV's AD & Robotics, L3 Team.

Evan: Let's dive into the Introduction then.

What's this paper all about, Ashley?

Ashley: Evan.

This paper tackles the limitations of current end-to-end autonomous-driving systems, which often rely on imitating human driving logs.

While effective to an extent, these systems are constrained by the quality and coverage of the recorded human driving trajectories.

Evan: Right, human driving logs capture only the actions a human driver chose in a given scenario, but don't encompass all possible actions, especially in critical situations.

Ashley: Exactly.

The main concern is that safety-critical deviations and recovery maneuvers are rarely captured.

This means the learned policy can drift away from the recorded demonstrations, leading to compounded errors.

Evan: I see.

So, what's the alternative proposed by the authors?

Ashley: The authors propose DriveZero, an end-to-end system that goes beyond human demonstrations.

They decompose driving into two models: a perception model and an action model.

Each is pretrained in its optimal learning regime before being integrated into a unified planner.

Evan: Can you break down how these models are trained differently?

Ashley: Of course.

The perception model, known as DriveVFM, is built from multiple frozen vision foundation models.

These models are trained using massive and diverse visual data, rather than task-specific annotations.

So, it understands the visual world comprehensively.

Evan: And what about the action model?

Ashley: The action model is called DriveRL.

It's trained using a mixed-agent closed-loop reinforcement learning framework.

This method uses real driving logs to create interactive worlds.

A privileged teacher policy is then trained through closed-loop rollouts.

Evan: How exactly does this DriveRL framework provide a robust learning environment?

Ashley: So, DriveRL converts static driving logs into dynamic, interactive environments where it can optimize its behavior by observing the consequences of its own actions.

This allows the model to understand states induced by its policy and learn how to correct mistakes effectively.

Evan: Interesting.

How are these individual models unified into the DriveZero planner?

Ashley: DriveZero integrates these pretrained models through a process called distillation.

The camera-only planner distills the frozen DriveRL teacher's knowledge by encoding multi-view images via DriveVFM and learning trajectories based on the teacher's rollouts.

Evan: And what are some of the key results and validations mentioned in the paper?

Ashley: The results are quite impressive!

On the nuPlan benchmark, the DriveRL with value-guided test-time action search achieves a mean score of 93.57 across multiple community splits, outperforming the Log-Replay expert.

DriveZero itself sets new state-of-the-art results on NAVSIMv1, NAVSIMv2, and the closed-loop HUGSIM benchmarks, showing significant performance improvements without any human trajectory supervision.

Evan: Those are substantial improvements.

It sounds like DriveZero really pushes the boundaries of end-to-end autonomous driving.

Ashley: Definitely.

By leveraging the strengths of both reinforcement learning and vision foundation models, DriveZero demonstrates how we can advance driving systems beyond the constraints of human demonstrations.

Evan: That wraps up the Introduction.

Next, we'll delve into the methodology behind DriveZero, stay tuned.

Evan: Alright, Ashley, let's break down the methodology behind DriveZero.

How does the paper propose we go beyond human demonstrations in autonomous driving?

Ashley: Evan.

The methodology of DriveZero is structured into three key stages: learning driving behavior, pretraining visual representations, and transferring the learned behavior to a camera-only policy.

Evan: Let's start with the first stage.

How does DriveRL, the action model, learn driving behavior?

Ashley: DriveRL implements a closed-loop reinforcement learning system to train a privileged teacher policy from scratch.

This system uses a mixed-agent simulator built from real nuPlan logs, transforming them into interactive worlds.

Evan: What do these interactive worlds involve?

Ashley: These worlds simulate real-world interactions, allowing DriveRL to experience dynamic scenes.

Each traffic participant has its own behavior model, created through a shared physical state-action interface.

This can include replaying real driving behaviors, using rule-based models like IDM, or incorporating learned policies from self-play.

Evan: How does the actual training process work in these environments?

Ashley: The training involves Proximal Policy Optimization, or PPO.

DriveRL, with its 5.7 million parameters, receives structured scene states and navigation goals as inputs, and it outputs actions in the form of longitudinal jerk and tire steering-angle rate commands.

During this training, the model not only generates actions but evaluates multiple action outcomes to select the best one based on reward metrics.

Evan: And how does DriveRL optimize for these actions?

Ashley: DriveRL optimizes its actions by considering a combination of hard safety events, goal arrival rewards, and driving-quality terms like comfort and compliance.

Importantly, it uses value-guided test-time action search at the inference phase to select actions that maximize expected returns, improving its policy beyond what's learned during training alone.

Evan: Got it.

Let's move on to the perception model, DriveVFM.

How is it different and what's its role?

Ashley: DriveVFM stands for Drive Vision Foundation Model.

Unlike traditional perception models that need task-specific labels, DriveVFM consolidates features from several powerful vision models, like DINOv3, SigLIP2, SAM, and Depth Anything V2, into a single backbone.

This means the model learns from raw images without requiring explicit annotations for tasks like detection or segmentation.

Evan: That sounds efficient.

How does the distillation process work for DriveVFM?

Ashley: DriveVFM employs a process called agglomerative distillation to merge these various foundation models.

The visual transformer uses a uniform set of spatial patch tokens and summary tokens per model.

Each frozen teacher model supervises different strengths, like spatial structure from DINOv3 or geometric cues from Depth Anything.

They train DriveVFM to become proficient across all necessary visual tasks.

Evan: And how does the DriveZero system bring these two models together?

Ashley: The fusion happens through a distillation process where the pretrained perception and action models are combined into a camera-only planner.

DriveZero uses multi-view images encoded by DriveVFM and decodes these into multiple trajectory proposals.

Then, it employs a winner-takes-all approach to select the most appropriate trajectory mimicking the behavior of the DriveRL teacher.

Evan: So DriveZero, by leveraging both models, can train without any human trajectory data.

What ensures the trajectories it generates are safe and reliable?

Ashley: DriveZero ensures this by continuously querying the DriveRL teacher with augmented driving intents during training.

This augments the supervision data significantly, providing a diverse range of goal-consistent trajectories that the static driving logs cannot offer.

During training, each candidate trajectory is scored against a learned scoring head, ensuring safety and compliance are embedded in the learnt behavior.

Evan: It's fascinating how the model continually learns and corrects itself.

What about the datasets used in training these models?

Ashley: DriveZero uses a comprehensive dataset known as navtrain, derived from nuPlan, which contains 100,000 real-world scenarios rich in interaction.

Additionally, it scales its training data using 237,000 out-of-distribution simulation scenes from SimScale.

Evan: Adding simulation data helps in covering a broader array of scenarios.

How does it affect the evaluation?

Ashley: It significantly improves DriveZero's performance across several benchmarks.

For instance, on NAVSIMv1, DriveZero-Scale which includes simulated data, performs exceptionally well, surpassing human drivers on some metrics and setting new state-of-the-art records on NAVSIMv2 and HUGSIM — that's a clear testament to its robustness and adaptability without human-trajectory supervision.

Evan: Indeed, benchmarking highlights the effectiveness of their approach.

How do they evaluate DriveZero on these benchmarks?

Ashley: For evaluation, DriveZero is tested on three key benchmarks: NAVSIMv1, NAVSIMv2, and HUGSIM.

NAVSIMv1 provides pseudo closed-loop evaluation, NAVSIMv2 evaluates with extended robustness via perturbations, and HUGSIM offers real-time, photo-realistic simulations.

Metrics like Predictive Driver Model Score (PDMS), Extended Predictive Driver Model Score (EPDMS), and HD-Score measure aspects like safety, compliance, collision avoidance, and progress.

Evan: So, the methodology of DriveZero combines sophisticated training techniques and extensive benchmarking to create a robust autonomous driving system.

That's quite detailed and impressive.

Ashley: Exactly, Evan.

And that's a wrap on the Method section of DriveZero.

Evan: Alright, Ashley, let’s move on to how DriveZero has been validated through various experiments and the results achieved.

Ashley: Sure thing, Evan.

The experiments are segmented into a few key areas, including the performance of DriveRL and DriveZero across different benchmarks, as well as ablation studies to validate the contributions of various components.

Evan: Let's start with DriveRL.

How did it perform on the nuPlan benchmark?

Ashley: DriveRL was tested on three nuPlan community splits: Val14, Test14-hard, and Test14-random.

The evaluation considered both non-reactive and reactive modes.

The results were impressive — DriveRL achieved an unweighted mean score of 93.01 across these six evaluations, outperforming the Log-Replay expert in all categories.

Evan: What about the value-guided test-time action search component?

How did that impact the results?

Ashley: Applying value-guided test-time action search, known as DriveRL-TTS, further improved the mean score to 93.57.

This approach enhanced the scores on five out of the six settings, emphasizing its role in refining the action selection process at inference time.

Evan: Sounds substantial.

What were the specific improvements in non-reactive and reactive modes?

Ashley: In non-reactive mode, on the Val14 split, DriveRL-TTS achieved a score of 95.54, and in reactive mode, it reached 94.53.

Particularly notable was an increase in the Test14-hard non-reactive split score to 91.13 from 89.97 with plain DriveRL.

Evan: Impressive results.

Now, let's delve into DriveZero's evaluation on NAVSIMv1, NAVSIMv2, and HUGSIM.

How did the system perform on these benchmarks?

Ashley: DriveZero achieved remarkable performance on these benchmarks as well.

On the NAVSIMv1 navtest, which is built from nuPlan, DriveZero matched the human driver’s performance, reaching 94.8 in Predictive Driver Model Score (PDMS).

When scaled with data from SimScale, DriveZero-Scale achieved 95.3 PDMS, surpassing the human driver's score.

Evan: That's quite a leap.

How did DriveZero fare on more challenging tests like NAVSIMv2 navhard?

Ashley: On NAVSIMv2 navhard, DriveZero maintained its edge by achieving an Extended Predictive Driver Model Score (EPDMS) of 51.5, topping other methods trained exclusively on the navtrain split.

Scaling with additional simulation data raised the score to 57.1, setting a new state-of-the-art.

Evan: It seems like DriveZero isn't just effective in pseudo closed-loop environments but also maintains its robustness in highly challenging scenarios.

What about the HUGSIM benchmark?

Ashley: For HUGSIM, which provides true closed-loop evaluation, DriveZero-Scale achieved the highest HD-Score of 46.6, setting a new benchmark across Easy, Medium, and Hard tiers.

This beats the previous best by a significant margin, showcasing DriveZero's generalization capabilities across diverse and realistic driving conditions.

Evan: What mechanisms contributed to these high scores on HUGSIM?

Ashley: A key factor was the diversity and quality of supervision from DriveRL, enhanced by goal augmentation.

This allows DriveZero to generate effective driving strategies even under varying and unpredictable scenarios, without relying on any manual trajectory supervision.

Evan: DriveZero’s success seems multifaceted.

How about the role of visual representation through DriveVFM?

Ashley: DriveVFM’s contributions are validated through various controlled studies.

For instance, replacing DriveVFM with other vision models showed that DriveVFM consistently provided better visual features essential for effective planning.

Ablation studies highlighted that each added foundation model teacher in DriveVFM cumulatively improved its performance.

Evan: It sounds like each component, DriveRL, DriveVFM, and their integration through DriveZero, plays a critical role in achieving these results.

Were there any specific ablation studies that stood out?

Ashley: Yes, an impactful study was comparing goal augmentation in training.

Without goal augmentation, DriveZero scored lower compared to when this method was included, emphasizing its importance in expanding supervision targets and enhancing policy learning.

Evan: Putting it all together, the experiments clearly validate that DriveZero's methodology of combining pre-trained perception with reinforcement learning in a distilled, unified planner, significantly advances the capabilities of end-to-end driving systems.

Ashley: Exactly.

And that's the end of the Experiment and Results section.

Evan: Alright, Ashley, let's now discuss the prior research that frames the context for DriveZero.

What's been covered in the related work section?

Ashley: Sure, Evan.

The related work section highlights advancements in three primary areas: closed-loop reinforcement learning for driving, visual foundation models for driving, and end-to-end autonomous driving.

Evan: Let’s start with closed-loop reinforcement learning for driving.

What are the key developments in this area?

Ashley: Closed-loop reinforcement learning, or RL, has been pivotal in autonomous driving.

Unlike imitation learning that relies heavily on fixed expert trajectories, closed-loop RL allows policies to learn directly from the consequences of their actions within an interactive environment.

Evan: Can you give us some examples of previous work in this field?

Ashley: Of course.

Urban Driver is one notable study where policies are trained using policy gradients in log-initialized urban scenes.

Similarly, BC-SAC combines imitation learning with RL to enhance performance in critical scenarios.

Evan: So, these methods create a dynamic learning environment that's more flexible than static imitation.

How do they compare in scalability?

Ashley: Indeed.

Later work has pushed the scalability of these methods considerably.

For instance, CarPlanner uses consistent auto-regressive rollouts for large-scale RL, while CaRL demonstrates that even simple rewards can drive effective policy optimization when scalable simulation is involved.

Evan: And what role does multi-agent training play here?

Ashley: Multi-agent training is crucial as it replaces static background traffic with learning agents, enhancing the interaction richness.

For example, SPICED and other systems use self-play strategies to develop robust, naturalistic driving behaviors without human data.

Evan: Moving on to visual foundation models for driving, how has this area evolved?

Ashley: This area leverages extensive pretraining on both general-purpose and driving-specific data.

Systems originally used ImageNet for supervision or self-supervised DINO objectives.

But there has been a shift towards more specialized models tailored to driving tasks.

Evan: Any prominent models or methods worth highlighting?

Ashley: Definitely.

RADIO is a significant development, consolidating multiple frozen visual foundation models into a unified backbone.

This approach, combined with PHI Standardization, enables robust feature standardization across diverse inputs.

Similarly, DriveVFM, used in DriveZero, distills multiple foundation models into one driving-specific backbone, learning directly from raw images.

Evan: How do these visual foundation models integrate with end-to-end autonomous driving systems?

Ashley: They form the perception core of many end-to-end driving systems.

These systems aim to map visual observations and navigation intents directly to driving actions, often through behavioral cloning.

Evan: And how do end-to-end systems typically operate?

Ashley: Most camera-based end-to-end systems use conditional control policies, unified perception–planning architectures, or multimodal trajectory generation.

They rely on human driving logs for training, which limits their capabilities to the quality of the logs.

Evan: What about more advanced methods that go beyond simple imitation?

Ashley: Recent approaches indeed look beyond imitation.

For example, ROACH distills a CARLA-trained RL agent into a visual policy.

TerraTransfer aligns visual encoders with self-play policies, completely bypassing human driving logs.

Pictura taps into perspective-view self-play at scale for end-to-end learning.

Evan: So DriveZero and similar methods build on these advancements by combining strengths from RL and visual models to achieve superior performance.

Ashley: Precisely.

DriveZero's methodology integrates these components effectively, offering a robust alternative by leveraging both independent pretraining and closed-loop interaction.

Evan: That wraps up the related work section.

It’s clear that DriveZero stands on the shoulders of giants in the realm of autonomous driving research.

Evan: Alright, Ashley, let's now summarize the key contributions and takeaways of the DriveZero paper.

Ashley: DriveZero represents a substantial leap in end-to-end autonomous driving by addressing limitations of existing systems that rely heavily on human driving logs.

Its primary contributions lie in its novel decomposition of driving tasks into separately pretrained perception and action models.

Evan: That's right.

By using DriveVFM, the perception model, distilled from multiple vision foundation models, DriveZero ensures a comprehensive understanding of the visual environment.

Ashley: Meanwhile, DriveRL, the action model, leverages closed-loop reinforcement learning to train in interactive, nuPlan-derived worlds, optimizing driving behaviors beyond static demonstrations.

Evan: And this combination is unified through a careful distillation process, allowing DriveZero to operate as a robust, camera-only planner, capable of producing state-of-the-art performance without human trajectory supervision.

Ashley: The experiments validate DriveZero's efficacy on multiple benchmarks, including setting new records on NAVSIMv1, NAVSIMv2, and HUGSIM, proving its scalability and robustness in diverse and challenging driving scenarios.

Evan: Together, these contributions demonstrate the potential of combining independently optimized models into a cohesive system, pushing the boundaries of what's possible in autonomous driving.

Ashley: Precisely.

DriveZero not only advances the technical frontiers but also presents a replicable approach for future research and development in autonomous driving technologies.

Evan: That wraps up our discussion on DriveZero: End-to-End Driving Beyond Human Demonstrations.

We hope you found this paper as fascinating as we did.

Ashley: Thank you for tuning in to Daily Paper Cast.

Join us again for more insights and deep dives into the latest AI research.

Evan: We look forward to bringing you more exciting papers and discussions.

Until next time, stay curious and keep exploring the frontiers of AI.