Causal Video Models Are Data-Efficient Robot Policy Learners

Causal Video Models Are Data-Efficient Robot Policy Learners

March 2026·Rhoda AI Research

At Rhoda AI, we are building towards generalist robotics. Our Direct Video-Action Model (DVA) reformulates robot policies as video generation, unlocking data-efficient task learning, scaling, long-context memory, and one-shot learning.

The Challenge of Generalist Robotics

For decades, we have excelled at creating specialized robots — machines that perform a single, repetitive task with superhuman speed and accuracy in controlled factory settings. However, the transition to generalist robotics represents a generational leap: moving from fixed-function hardware to general-purpose agents capable of navigating the messy, unpredictable nature of the real world. Solving this challenge is critical because the next era of automation hinges not just on a robot's ability to follow a precise script, but on its capacity to generalize across diverse environments and tasks using a single, unified model.

The missing ingredient is one that today's most powerful AI systems all share — whether specialized for language, images, or video, the most capable models are trained on web-scale data. At massive data scale, AI models equip themselves with a broad base of knowledge and the ability to generalize across countless situations. We build our generalist robots on the same principle. The challenge, however, is data. Emerging approaches, such as vision-language-action models, collect tens or hundreds of thousands of hours of robot data, yet are still far from true generalist behavior. And no matter how much data we collect, it will always be a tiny fraction of the web-scale data available.

We believe web video is the most scalable data source capturing the dynamic physical world, and video generation is the most effective objective for a model to learn the deep physical knowledge robots need for decision-making. Our strategy directly formulates robot control as real-time video prediction through a new paradigm: Direct Video-Action Models (DVA). Trained on web-scale data, these models offer significant advantages over existing approaches:

Data-efficient task learning. Our models perform complex, long-horizon tasks reliably with as little as ~10 hours of total robot data.

Long-context visual memory. Unlike most vision-language-action models, which often have a context of only a few frames, our models natively have hundreds of frames of visual context, enabling them to orchestrate sophisticated, multi-step tasks end-to-end.

One-shot learning. Long-context visual memory also unlocks new model capabilities, such as learning to imitate human behavior from a single demonstration, in-context, at test time.

Interpretability through video generation. Because robot actions are generated as videos first, the robot's behavior can be directly visualized through autoregressive rollouts, enabling inspection of model decisions, comparison of configurations, and verification of safe behavior.

Most importantly, our approach offers a clear path for scaling, since video data exists at an orders-of-magnitude larger scale than robot interaction datasets.

Direct Video-Action Models

💡

Direct Video-Action Model: A robot policy that translates predictions from a pre-trained causal video model into actions in a real-time closed loop, with the video model directly responsible for decision-making.

Figure 1. Simplified diagram of a Direct Video-Action Model. Conditioned on a video history, we predict future video frames. An inverse dynamics model translates the video prediction into actions, which are executed on the robot. These steps are repeated in a streaming closed-loop, running multiple times per second.

We leverage large-scale pre-training by formulating robot control as video prediction. Conditioned on a long context of captured video from the robot, proprioception, and other conditioning signals (e.g. language), we predict a short period into the future, visually. This prediction captures how the robot should behave and how the environment will evolve. A separate inverse dynamics model then serves as a translator, converting the predicted future into robot actions. This cycle of video prediction and robot action translation repeats in a closed loop, multiple times per second. We call this approach a Direct Video-Action Model (DVA) (Figure 1), because the video model directly specifies the desired future behavior as the policy, providing the target signal for action translation. This new paradigm reduces robot control to video generation, enabling the benefits of web-scale pre-training.

Much of the prior work has explored using video models for robot control, including synthesizing training data with video models1-3[1] GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arxiv.org/abs/2503.14734 [2] DreamGen: Unlocking Generalization in Robot Learning through Video World Models. arxiv.org/abs/2505.12705 [3] UniSim: Learning Interactive Real-World Simulators. arxiv.org/abs/2310.06114, fine-tuning video models to predict actions4-12[4] Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation. arxiv.org/abs/2312.13139 [5] GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation. arxiv.org/abs/2410.06158 [6] Unified Video Action Model. arxiv.org/abs/2503.00200 [7] Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets. arxiv.org/abs/2504.02792 [8] Prediction with Action: Visual Policy Learning via Joint Denoising Process. arxiv.org/abs/2411.18179 [9] Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training. arxiv.org/abs/2402.14407 [10] Video Generators are Robot Policies. arxiv.org/abs/2508.00795 [11] Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning. arxiv.org/abs/2601.16163 [12] DreamZero: World Action Models are Zero-shot Policies. dreamzero0.github.io, performing open-loop control with video models13-18[13] Learning Universal Policies via Text-Guided Video Generation. arxiv.org/abs/2302.00111 [14] PLEX: Making the Most of the Available Data for Robotic Manipulation Pretraining. arxiv.org/abs/2303.08789 [15] VideoWorld: Exploring Knowledge Learning from Unlabeled Videos. arxiv.org/abs/2501.09781 [16] This&That: Language-Gesture Controlled Video Generation for Robot Planning. arxiv.org/abs/2407.05530 [17] TesserAct: Learning 4D Embodied World Models. arxiv.org/abs/2504.20995 [18] 1X World Model: From Video to Action. 1x.tech/discover/world-model-self-learning, and performing closed-loop control with a video model distilled from a non-causal video model19-21[19] Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations. arxiv.org/abs/2412.14803 [20] mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs. arxiv.org/abs/2512.15692 [21] LingBot VA: Causal video-action world model for generalist robot control. technology.robbyant.com/lingbot-va. To the best of our knowledge, our model is the first to pre-train a causal video model from scratch and also the first to perform full video denoising during real-time closed-loop robot control.

Native Causal Video Models

We pre-train on general web videos with a causal video generation objective: conditioned on a video history, predict the future. We pre-train from scratch as a causal video model, rather than distilling from a pre-trained bi-directional model. Training on large-scale, diverse video data natively imbues our model with a strong prior on 3D structure, physics, behavior, and conventions.

Existing methods for causal video generation encode an entire input sequence, yet supervise on only a few predicted frames. This means the full input video context must be processed for every small prediction step, which is computationally expensive. Diffusion Forcing22[22] Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. arxiv.org/abs/2407.01392, another recent approach, trains on sequences where each video frame is independently noised to a random level. But in this case, predicting future frames from a noise-free captured input video occurs with vanishingly small probability during training, and performance degrades with a long video context. Hence, neither approach satisfies our desire to both match our training noise mask with inference, and also to ensure efficient training.

0

1

2

3

4

5

6

7

context

prediction

Figure 2. During training, our causal video architecture amortizes the cost of encoding a long context of video frames by simultaneously predicting future frames at every position in the sequence. When training with diffusion, each prediction can be noised independently, with different noise levels.

💡

Context Amortization: A training strategy that predicts future video at every point along a long history of noise-free context, in order to efficiently train causal video generation.

We introduce the Context Amortization training strategy (Figure 2), which accomplishes both goals, in that we always train with the mask we use at inference (conditioned on a long history of noise-free captured context, predict the future), but we also increase the number of frames on which we calculate the loss, which makes training more efficient. Inspired by language models, which predict next tokens at every position in a text sequence, we encode noise-free context and predict future frames at every position in a video sequence. This strategy allows us to efficiently train with hundreds of frames of context during pre-training.

At inference time, KV-caching ensures the encoded context is reused across steps, minimizing redundant computation. When inferencing for robot control, we always use real observations as context, grounding the generation in the continuous observations of the physical environment. However, for visualizations, our models can also generate long videos auto-regressively.

Translating Video to Action with Inverse Dynamics Models

The second main component of our system is an inverse dynamics model, which performs video-to-action translation: given a predicted video, it produces the precise end-effector motion needed to re-enact the depicted actions. Causal action prediction — as in a typical robot policy — predicts future actions conditioned on the past and thus requires modeling behavior and decision-making. Behavior may be arbitrarily complex, and for generalist behavior, may require a data scale infeasible to collect on robots.

In contrast, non-causal video-to-action translation is a much more constrained problem. The complex decision making has already been handled at the video generation stage by a model that can leverage large-scale pre-training. What remains — inferring end-effector motion from a demonstration video — is comparatively straightforward.

Video 2. Non-causal action prediction allows precise video-to-action translation with only a handful of hours of data per robot embodiment type (1x speed).

Consequently, we can solve the inverse dynamics task with a small model trained on as little as ~10 hours of data collected from the embodiment type (Video 2). As an added bonus, the data need not involve high-quality task demonstrations: even random motions can be used for training inverse dynamics. Once trained, the inverse dynamics model can be used across different robots of the same type and across many tasks.

Leapfrog Inference

💡

Leapfrog Inference: A strategy for continuous robot control that predicts long enough into the future to cover the next prediction's inference latency. The predictive model is conditioned on the action currently being executed (predicted at the previous time-step), in order to ensure trajectory continuity.

Model inference takes time, but the physical world does not wait for the model to decide. Therefore, we o

[truncated for AI cost control]