Robotics: Endgame – NVIDIA's Roadmap to Physical AGI

A comprehensive talk outlining NVIDIA's roadmap to Physical AGI, proposing video world models as a second pretraining stage, introducing World Action Models (WAM) for joint video-action prediction, and describing systems like EgoScale and DreamDojo for robotics scaling.

What is Jiggly Jar?

Testing

Robotics: Endgame – NVIDIA’s Roadmap to Physical AGI.

In a recent Sequoia AI Ascent keynote dubbed “Robotics: Endgame”, NVIDIA Robotics director Jim Fan laid out a sweeping vision for Physical AGI – that is, truly general-purpose robotic intelligence. Fan’s thesis is that the path to embodied intelligence largely parallels the trajectory of large language models: first massive pretraining, then alignment to tasks, and finally iterative reinforcement learning to polish capabilities. To achieve this, NVIDIA proposes new paradigms and tools: video world models as a second pretraining stage, World Action Models (WAMs) that jointly predict future states and actions, and expansive data-generation pipelines (from human wearables to egocentric video) to feed these models. Key systems include the EgoScale framework (pretraining on 20,000+ hours of human egocentric video, yielding a discovered “dexterity scaling law”) and DreamDojo, a neural physics engine for simulating dexterous manipulation at scale. In this article we analyze these concepts and how they fit together into NVIDIA’s roadmap toward robots that can truly think and act in the physical world.

The Great Parallel: Copying the LLM Playbook.

Fan opens with what he calls “the Great Parallel”: the insight that robotics can copy the success story of large language models by following an analogous multi-stage process. In brief, the LLM journey went through four stages over the last half-decade: (1) Unsupervised pretraining (e.g. GPT-3) to learn the shape of language, (2) Supervised fine-tuning (e.g. instruction/few-shot models) to align the model to useful outcomes, (3) Reinforcement learning via self-play, pushing beyond imitation (as in InstructGPT and RLHF), and (4) Automated research loops (systems that continuously generate richer data and models beyond human scalability). Fan suggests robotics can mirror these steps, but with the twist that we “simulate the next physical world state” instead of the next word. In practice, the plan is to:.

Pretrain a world model on massive data so it “understands” physics and object dynamics.

Align by fine-tuning the world model on the relevant slice of tasks or teleoperation data.

Apply reinforcement learning (in the sim or on slow real robots) to refine policies to peak performance.

Automate the loop with end-to-end learning and better simulators, much as LLM labs automated hyperparameter tuning and data generation.

In short, robotics will leverage foundation world models analogous to GPT, then fine-tune and apply RL to bridge the final gap. Fan’s mantra is essentially “be a good scientist, copy homework,” meaning the roboticist’s homework is written by the successes of LLMs.

Why Vision-Language-Action (VLA) Models Fall Short.

For the past few years, many robotics efforts have been built on Vision-Language-Action (VLA) models: typically large vision-language networks (like CLIP or Flamingo) grafted onto a robotic action head. VLA models excel at “semantic generalization” – an LLM-trained VLA can interpret commands or goals described in language and point to objects in its camera feed. But Foreshadowing the need for something new, Fan notes a key weakness: these models are really “LVAs” – i.e. language-vision-action models, where the bulk of capacity is devoted to language understanding. In practice this means they encode nouns and concepts well, but strike out on fluid physics and motor actions.

Put bluntly: you can ask a VLA model to locate “the Coke can on the table” or generically sort blocks by color, but it doesn’t really know how objects move or interact. For example, one famous demonstration used a VLA to move a robot arm to a picture of Taylor Swift, which proved semantic generalization in language (Swift was never seen before), but it didn’t show the model learned anything about the dynamics of picking up or shifting a can – tasks we care about in robotics.

NVIDIA’s argument is that while VLA models have pushed the field forward, they invest most of their parameters on language and knowledge rather than on world dynamics. As one of Fan’s quips puts it: VLAs have a large language “head” on a slim vision stem on an even slimmer action tail. The result is great at nouns (objects) and static scenes, but weak on verbs (motions) and physics. This mismatch shows up in practice: VLAs trained purely on imitation or minimal exploration struggle when asked to perform novel physical interactions or adapt to new environments they haven’t seen in demonstration.

Enter World Models. To go beyond the limits of VLA, Fan proposes moving to a world-prediction paradigm. Instead of only predicting actions from observations, models should predict how the scene itself changes in the next few moments – that is, learning an internal simulator or “world model.” By learning to predict video frames forward in time, these models will necessarily learn aspects of gravity, collisions, object permanence and other physical laws. In the LLM analogy, after learning the “grammar” of language, you now teach the model to “speak” and “move” inside its own world model of pixels and states.

In essence: use video prediction as the second pretraining stage. Large video models (especially diffusion-based or transformer sequence models) are now powerful enough that, when trained on lots of footage, they begin to implicitly understand physics. Their latent features encode quantities like object velocity, contact events, and even “gravity” relationships, although not in a clean factorized way. Indeed, recent work suggests that transformers trained to predict video frames learn representations of speed and motion direction at intermediate layers (the so-called “Physics Emergence Zone”). Simply put, an unchecked video generator “learns” how the world evolves because it has to. NVIDIA’s take-home: train a massive video model and physics will emerge automatically.

Thus, after image-language pretraining, the next pretraining pass should be on videos: whether large-scale internet videos or rich robot teleop data, run through a video-prediction objective. NVIDIA’s whitepaper for this stage is essentially: train on egocentric human video for as long as possible, to get latent physics. This unlocks “common sense physics priors” that can carry into downstream robot tasks. Fan’s mantra: “Physics emerges by predicting the next blob of pixels at scale.”.

World Action Models: Jointly Predicting Video and Actions.

Building on video world models, Fan introduces the notion of World Action Models (WAMs). A WAM is an embodied foundation model that simultaneously predicts future observations and the robot’s actions that will happen (or should happen). In other words, instead of a policy that maps vision-to-action or a planner that maps states forward, a WAM targets the joint distribution over future states and actions. It unifies world modeling (predicting what you’d see next) with policy generation (deciding what to do next).

This is a natural extension: if your model can dream up what the robot scene will look like in 1 second, it can condition on that dream to choose an action that leads toward a goal. NVIDIA’s example system for this idea is DreamZero. DreamZero is essentially a video diffusion model fine-tuned to robotics: given the current camera image and a language or goal prompt, it autoregressively decodes both a short video rollout and the accompanying continuous robot joint actions. The insight, as Fan puts it, is that motor actions are just another signal to predict per timestep – much like pixels. So DreamZero jointly decodes video frames and action vectors. If the predicted video prediction “makes sense” (physical), the predicted action is likely good. If the video hallucinates or violates physics, the action is likely wrong.

This WAM approach pays off: preliminary results show DreamZero yields strong zero-shot generalization to completely novel tasks and verbs. NVIDIA reports over 2× improved generalization on new tasks vs. vision-language models, even with no new demonstrations. In tests, a 14B-parameter DreamZero could run at 7 Hz closed-loop control after engineering optimization, giving it near-real-time responsiveness. Most strikingly, DreamZero can leverage any video: demonstrations from other robots or even humans yield large boosts in performance without requiring new robot data. In one result, adding just 10–20 minutes of video from one source led to a 42% relative improvement on an unseen task. Moreover, adapting DreamZero to a new robot embodiment with only 30 minutes of additional data yields it still maintaining its zero-shot abilities on old tasks.

In sum, World Action Models like DreamZero prove that by predicting rich video futures, policies emerge that handle diverse dynamics. We are not merely grafting a control head on a vision-language backbone anymore; we are building full simulators inside the model. As one NVIDIA paper puts it: WAMs learn physics by predicting future states and actions, using the video as a dense, self-supervising representation of how the world evolves. This shares the load of understanding dynamics and generating control, resulting in far more robust generalization. Fan’s bottom line: “Let’s all take a moment of silence for our dear friend VLAs… Rest in peace. Long live world action models.”.

Gathering Data at Scale – From Wearables to Egocentric Video.

Of course, even the best models will need vast data. Here Fan lays out what he calls the robotics data revolution. Historically, collecting robotic data meant teleoperation: humans in VR or special rigs controlling robots. This approach has been the norm for years, but it scales poorly. As Fan points out, teleoperation is fundamentally limited by human availability: even if a person never sleeps, a single robot yields only 24 hours of data per day – in practice it’s much lower, on the order of a few hours per robot per day. That’s far too slow if you want millions of hours of experience.

To get around this bottleneck, NVIDIA describes three tiers of data collection, each progressively more scalable:.

Tier 1 – Wearables (the Universal Manipulation Interface): Instead of connecting a human’s brain to the robot via VR, this approach collects human demonstrations with wearable hardware. A prime example is the Universal Manipulation Interface (UMI), which uses hand-held grippers as a data-gathering tool. People simply pick up objects while wearing these grippers and move them about. The key is that policies are trained on this data but are hardware agnostic, so the same recorded behavior can run on any robot hand. A UMI-captured demo of handing you a screwdriver can train robots on two-finger grippers or dexterous hands alike, because the interface ensures only the essential manipulations are captured. Work on UMI shows it can gather rich bimanual, dynamic manipulations in the wild and then transfer them zero-shot to real robots – enabling long-horizon or precise tasks that would be impossible to program manually. UMI hardware (handheld grippers) plus a carefully designed policy interface yields portable data. NVIDIA further extended this concept to fully-fledged hands with DexUMI: a wearable hand exoskeleton plus vision tracking that turns a human’s own finger motions directly into robot finger motions. DexUMI’s exoskeleton provides full haptic feedback and solves the kinematic gap between human and robot hands, while a neural inpainting module “replaces” the human’s visible hand in video frames with a high-fidelity rendered robot hand. The end result is that people can just use their hands (wearing the exoskeleton) to demonstrate tasks; DexUMI bridges to a 5-finger robot hand with minimal extra overhead. In experiments, DexUMI achieves an average success rate of 86% on a variety of dexterous tasks, with zero true teleoperation needed at all.

Tier 2 – Egocentric Video (EgoScale): Wearables help, but they still involve cumbersome hardware on the person. Fan’s next tier is hands-free data collection via egocentric video. The idea is analogous to Tesla’s Full-Self-Driving (FSD) program: every human-camera wearer becomes a data collection node, and each minute of footage contributes to training. NVIDIA’s EgoScale is their implementation of this idea. In EgoScale, people simply wear a head-mounted camera (like a VR/AR headset or action cam) along with high-precision hand tracking. They go about daily activities manipulating objects (“in the wild”), while the system records ego-centric video plus corresponding estimated hand motions. Crucially, no robot is involved during this data collection. It literally scales as fast as people can record themselves – fractionally limited only by battery and file storage.

In a recent effort, NVIDIA pre-trained a vision-language-action model on over 20,000 hours of such egocentric human video, more than 20× bigger than prior work. They found a log-linear scaling law: as you feed more human data to the model, the validation loss decreases predictably. (That means if you double the data, performance improves by a constant factor on the log scale, akin to the famous scaling laws of language models.) Critically, they showed this pretraining loss strongly correlates with robotic success. After this massive pretraining, only a tiny mid-training on aligned human-robot pairs (on the order of 50 hours) plus a few hours (4h) of actual robot teleoperation was enough to yield a full manipulation policy. The outcome is compelling: an end-to-end policy mapping raw camera images to 22-DoF dexterous robot hand actions, all learned largely from human video. On downstream tasks – like sorting cards, using tools, or folding clothes – this policy significantly outperformed one trained from scratch. For example, across five dexterous tasks the pretrained model showed 54% higher average success rate than a no-pretraining baseline. It even generalizes across embodiments: a model pretrained on human video worked well when deployed on robots with fewer fingers, suggesting the large human dataset imparts an “embodiment-agnostic motor prior”. Perhaps most saliently, this work confirmed a Neural Scaling Law for Dexterity: performance continues to climb smoothly as more human data is added. In Fan’s words, six years after scaling laws revolutionized language models, “robotics now has its own.” The projected data pyramid is clear: teleop will shrink to a rounding error, data wearables can yield hundreds of thousands of hours, but egocentric video could eventually produce tens of millions of hours per year if we fully emulate the FSD flywheel.

This second-tier approach is embodied in NVIDIA’s EgoScale infrastructure: pretrain on millions of hours of unlabeled video, then action-fine-tune on a sliver of robot-aligned data. The result is one-shot and few-shot learning for new tasks, with minimal actual robot trials. As Fan summarizes, “we now have data collection that fades into the background” – an ordinary person recording their life feeds the robot intelligence.

Tier 3 – Neural Simulation and Reinforcement Learning: The third tier concerns simulation and learning. Just as the Youtube content driver “creates” data for vision models, Fan argues we need a cheap, scalable way to create robot experience for RL. The idea is to treat our compute as the environment: many robots acting in parallel in a learned simulator. NVIDIA’s proposal has two parts:.

Real-to-sim world scanning: take real scenes and turn them into simulatable worlds automatically. For example, an iPhone camera can capture an environment; NVIDIA’s pipeline runs 3D reconstruction to extract objects and their poses. These objects can then be augmented with “digital cousins” – procedurally varied copies – to create an endless zoo of simulated scenes. In effect, your phone becomes a “pocket world scanner.” While details are not fully published, this fits the trend: we’re seeing more use of NeRFs, SLAM, and USD pipelines to digitize real-world environments for robotics.

DreamDojo – Neural Physics for RL: Perhaps the most central piece is DreamDojo, what Fan calls “an end-to-end neural physics engine” for scaling RL. DreamDojo is a foundation model of the world (trained on even more video than EgoScale), which can simulate arbitrarily long sequences of robot interaction. Concretely, DreamDojo is a video world model (actually an autoregressive transformer) trained on 44,000 hours of egocentric human video (roughly 15 years of footage) – far larger than any prior robotics dataset. It treats the video as though it comes from a first-person robotic camera, learning how objects move in response to “latent actions.” Because these videos lack true action labels, the authors introduce continuous latent actions as a proxy, letting the model figure out the relevant action signals.

The result is a giant robot-world simulator. After pretraining on human video, DreamDojo is post-trained on small sets of real robot data so it learns actual joint limits and embodiment. The happily surprising result: even after this up-zapping to robotic controls, DreamDojo can simulate complex contact-rich tasks and new scenes it has never seen, achieving broad generalization. And due to a clever distillation process, DreamDojo can run fast enough for practical use: the distilled model emits rollouts at ~10 FPS for minutes at a time. In demo, DreamDojo can live-teleoperate a robot in real-time, or evaluate and plan policies entirely in the neural sim—no traditional physics engine involved. The project highlights even show DreamDojo generating realistic object manipulations across four distinct robot platforms using just its learned world dynamics.

In summary, DreamDojo provides the “millions of environments” needed for RL. With it, NVIDIA can spin up thousands of parallel simulations using GPU clusters, solving RL tasks orders of magnitude faster than waiting for real robot trials. As Fan pithily concludes: “Compute now = environment now = data.” We can buy data by buying compute, in effect.

Across these tiers, a theme emerges: scale everything. Collect data wherever you can (from gloves to head-mounted cameras), train ever-larger models on it, and use the models themselves to generate more data (via simulation). This circular “self-reinforcing loop” is analogous to Tesla’s or OpenAI’s FSD loop: each increment in data or compute accelerates the next. NVIDIA’s roadmap posits that by plugging into these high-bandwidth data sources (video, simulations), robotics can hit the same breakneck ascendancy that language models did.

EgoScale and the Dexterity Scaling Law.

A centerpiece of the data strategy is EgoScale, so we’ll unpack it in more detail. EgoScale’s premise is simple but radical: human hands are the best robot arms we have. Instead of trying to coax more from expensive robot fleets, just gather as much human manipulation as possible. The key is converting human video into a form a robot can learn from. EgoScale does this by: (a) recording long egocentric video clips of people manipulating objects, (b) extracting or retargeting their hand and arm motions into a standard representation (e.g. the equivalent robot joint commands for some anthropomorphic hand), (c) training a vision-language-action (VLA) policy on these inputs as if they were robot data, and then (d) fine-tuning on minimal paired human-robot data to align the embeddings.

In practice, NVIDIA collected over 20,854 hours of such human video – everything from cooking in a kitchen to assembling mechanical parts. They then trained a flow-based VLA model (a kind of diffusion policy) to imitate this massive human corpus. The payoff was immediate: validation loss steadily decreased with more data, following a log-linear trend. This “dexterity scaling law” means that performance (as measured by loss or downstream task success) predictably improves as more pretraining video is used. Essentially, more human hours = a stronger motor prior for the robot.

Once the VLA was pre-trained on human video, only a small mid-training stage was needed: about 50 hours of aligned human-robot play and a few hours of teleoperation. Then the final policy could be trained on the actual robot tasks of interest. The outcomes were dramatic. With a 22-DoF Shadow hand, the EgoScale-trained policy achieved a 54% higher success rate on various complex tasks than an identical model trained from scratch. It could sort objects, manipulate cups, unstack objects, and even learn new subroutines (like a novel shirt-folding method) from a single demonstration. Moreover, the policy was embodiment-agnostic: the same underlying model worked on simpler grippers without retraining, showing that the human data produced a reusable motor prior for any hand.

Most importantly, scaling arguments came true. Fan’s talk notes that with EgoScale we observed physics-based scalability akin to language models – more data gave proportional gains. Graphs from the paper show a clean log-log improvement in loss; extrapolating suggests that if one could harness, say, 100 million hours of human video (on par with blanketing countless lifespans), robot dexterity could become astonishing. In other words, dexterity follows a law just like perplexity does for text models. This perspective reframes long-standing robotics effort: instead of hand-crafting or teleoperating every skill, we are now quantitatively mapping “hours of video” to “ability of robots.”.

This scaling insight also motivates Nvidia’s data predictions. If UMI can harvest tens or hundreds of thousands of hours, and if egocentric wearables can generate millions more (think of AR glasses on millions of people), then we’re in regime of LLM-scale data. Indeed, Fan claims a 10M-hour/year data source is within reach. If the trends continue, we may reach a point where adding more human video offers diminishing returns, analogous to how NLP moved beyond pure text once the scaling curve began to flatten. The end goal is a “data flywheel” where every human action in daily life became incremental robot training data – and where robots learn to imitate the entire spectrum of human dexterity with negligible direct programming.

Neural Physics and Simulation: DreamDojo.

With data collected and world models trained, how do we actually train final controllers? Here NVIDIA leans on reinforcement learning (RL), but not on hand-designed simulators. Instead, they treat the learned video models as the simulator. DreamDojo, discussed earlier, is the centerpiece of this approach.

DreamDojo’s ambition is to be the RL playground for generalist robots. It can take a robot’s state (current image and sensor readings) and any proposed action, and immediately roll out the predicted future (both visual and state sensors). Because it’s based on learned video, it bypasses traditional physics engines entirely – Newtonian dynamics are implicit. Training on DreamDojo means essentially running “self-play” of policies inside the neural world model. This is a form of model-based RL: the policy is trained by unrolling DreamDojo forward.

The technical novelties are worth highlighting. DreamDojo pretrains on 44k hours of human video, then is post-trained on robot interactions to ground it. Importantly, it uses a latent-action scheme so it can ingest continuous commands. After training, the model is distilled down (compressed) to inference-friendly scales, achieving real-time rollout. That allows two killer-apps: teleoperation in the loop, and policy evaluation/planning. In demos, DreamDojo can generate videos while a human teleoperator controls a real robot; the human sees the simulated outcome and can decide on-the-fly. Conversely, a policy can “try” actions in DreamDojo before applying them to the real robot, enabling risk-free planning.

Finally, DreamDojo is made available on GitHub and has spurred community interest – already there are model checkpoints and derivative work. It represents a monumental step: a first generalist robot world model. It validates the core premise: neural networks can learn to simulate kitchens, workshops, and object interactions from video and use that to train actual robots.

Bringing it All Together: The Road to Physical AGI.

NVIDIA’s “Robotics: Endgame” talk stitches these pieces into a roadmap. The idea is that by copying the LLM playbook, and by scaling each component massively, we reach a point where robots can solve almost any task through learned intelligence. The components are:.

Vision-Language Pretraining: initial models that understand basic concepts.

Video World Model Pretraining: models that predict physics, prime the network with dynamics.

World Action Model Fine-tuning: joint video-and-action models (like DreamZero) that learn controllers from diverse data.

Data at Scale: a self-reinforcing funnel of data (wearables, video, simulation) that fuels training.

Reinforcement Learning in Learned Simulators: DreamDojo-driven RL to polish final policies.

Iterate and Automate: continual retraining and expansion in a loop.

Fan’s keynote was peppered with optimism: robotics is “entering its endgame,” where the method is defined even if the implementation remains to be completed. The “physical AGI” they envision is a system that can handle arbitrary physical goals in the real world – a true successor to the language-based GPTs, but embodied.

Of course, there are huge challenges ahead. Scaling data and models is nontrivial, even with today’s hardware. Ensuring safety and reliability of learned behaviors is crucial before deploying “generalist” robots. And there are hard questions on how to evaluate such agents (the Physical Turing Test from last year’s talk is one notion). However, by laying out this blueprint and showing concrete progress (EgoScale, DreamZero, DreamDojo, etc.), NVIDIA’s roadmap provides the clearest path we’ve seen toward bridging the gap from today’s narrow robot skills to tomorrow’s intelligence.

For researchers and engineers, these ideas suggest many starting points. Can you train your own WAM on in-house data? How does EgoScale transfer to your robotic platform? What if we apply DreamDojo to new robot embodiments or tasks? The components are just emerging, and the toolbox is being built as we speak. In Fan’s words: “it’s closer than you think”. We just might be the generation to solve robotics – after all, as the talk closed, “we were born too late to explore the Earth, too early to explore the stars, but just in time to solve robotics.”.

Sources: This summary is based on Jim Fan’s Sequoia AI Ascent talk “Robotics: Endgame” (May 2026) and related publications. The concepts of World Action Models and DreamZero are described in NVIDIA’s technical reports. EgoScale’s findings (20k+ hr dataset, scaling laws, performance gains) are detailed in the Nvidia EgoScale project page and its arXiv paper. The DexUMI (exoskeleton interface) is presented in the DexUMI paper and website. The DreamDojo world model comes from NVIDIA’s Feb 2026 report (44k hr video pretrained, real-time distilled world model). Quoted remarks and structure are drawn from the talk transcript and a published overview.