Vision-language-action models bring perception, instruction following, and physical control into one learning system. Their promise is easy to state: tell a robot what should happen, let it understand what it sees, and have it produce the actions that make the instruction real. The hard part is everything between those three verbs.
A generalist robot is not merely a model that can generate actions. It must decide what matters, plan across time, act continuously, learn from outcomes, and retain the right experience.
This article follows the presentation’s original path. We begin with the robot as a complete sensing–thinking–acting system, move upward through the embodied-data pyramid, and then examine how VLA research is expanding from action prediction into planning, real-time execution, reinforcement learning, human-to-robot transfer, memory, and world models.
01 · What Is a Robot?
A body with a perception–action loop
A robot combines sensors, computation, intelligence, a physical body, actuators, and power. Sensors observe the world through images, video, depth, audio, language, robot state, force, and touch. A computing stack turns these signals into estimates and decisions. Actuators then express those decisions as joint motion, end-effector poses, trajectories, or other low-level commands.
The word intelligence hides several different capabilities. A useful robot needs representations of the current observation, memory of what happened before, a world model of what may happen next, a way to plan or reason over sequences, a policy that chooses the next action, and a learning process that improves the whole system.
Embodiment also changes the output space. A fixed arm may expose six or seven joints. A humanoid may have dozens. A quadruped coordinates legs and balance, while a mobile manipulator combines wheeled motion with one or more arms. The same high-level instruction therefore has to be translated into very different trajectories and control rates.
VLA models sit inside this loop. They convert visual observations and language instructions into actions, but a deployed robot still requires state estimation, controllers, timing, safety constraints, and hardware-aware execution. The model is an increasingly central policy component—not the entire robot.
02 · Which Data Do We Need?
The embodied-data pyramid
Robot intelligence is fundamentally a data problem, but not all data has the same cost or usefulness. At the bottom of the pyramid, general images, text, and video are abundant and inexpensive. Simulation adds geometry, dynamics, and controllable variation. Egocentric human video brings actions closer to an embodied viewpoint. Human–machine interaction data connects human intent to a physical interface. Real robot demonstrations are the most directly useful—and usually the most expensive to collect.
Each layer contributes something different. General data supplies semantics and broad visual concepts. Simulation offers labels, counterfactuals, and failures at low physical cost. Egocentric recordings preserve the human viewpoint and long-horizon task structure. Interaction devices—VR rigs, 3D mice, teleoperation systems, and multi-arm interfaces—capture intent in a form closer to robot control. Real robot data finally exposes friction, delay, calibration error, contact, and hardware-specific constraints.
The best dataset is rarely a single dataset. Generalist robotics needs a mixture: broad data for concepts, synthetic data for coverage, human data for behavior, and robot data for physical grounding.
The central trade-off is not simply “more data.” It is how to allocate collection effort. A million visually diverse clips may teach recognition but not torque-sensitive manipulation. A small collection of expert demonstrations may teach precise actions but fail to cover new objects and environments. The value of a dataset depends on which missing capability it is meant to supply.
03 · How Do Robots Think and Act?
From action generators to generalist intelligence
The early trajectory of generalist robot learning established a powerful recipe. RT-1 showed how transformer policies could scale across many real-world tasks. Action Chunking with Transformers predicted coherent blocks of actions for precise manipulation. Diffusion Policy modeled continuous, multimodal action distributions. RT-2 then transferred knowledge from vision-language pretraining into robot control.
VLA · 01
Action generation: autoregressive or continuous?
A VLA model must represent action. One option discretizes motion into tokens and predicts them autoregressively, much like language. This creates a clean interface with pretrained language models and makes robot actions part of the same sequence-modeling problem. OpenVLA is a prominent open implementation of this direction.
Continuous approaches instead model trajectories directly. Flow-matching and diffusion-based policies can represent smooth, multimodal action distributions without forcing every motor command into a fixed vocabulary. π0, for example, places a flow-matching action expert on top of a pretrained vision-language backbone.
The choice is not merely stylistic. Autoregressive tokens inherit mature language-model infrastructure, while continuous generators naturally match the geometry and timing of control. Hybrid designs increasingly separate a semantic backbone from an action expert so that each component can operate in the representation best suited to its role.
VLA · 02
Planning: make the intermediate structure explicit
Direct action prediction works well when the mapping from observation and instruction to motion is short. Long-horizon tasks are different: “clean the table” hides object selection, ordering, navigation, grasping, failure recovery, and completion checks. Planning methods introduce intermediate structure before or alongside the action sequence.
Language plans decompose a task into readable subgoals. World-state plans predict goal images, visual tokens, or future video before decoding actions. Action-space plans generate a coarse trajectory that guides fine control. Latent plans internalize the intermediate computation rather than exposing it as text. Hybrid methods combine language and vision so that reasoning can stay grounded in what the robot actually sees.
These choices form a spectrum between interpretability and compression. An explicit subtask list is easy to inspect but may omit geometry. A latent plan can be compact and differentiable but difficult to diagnose. The right interface depends on whether the dominant uncertainty is semantic, spatial, dynamical, or procedural.
VLA · 03
Real-time control: remove the stop between chunks
Action chunking improves temporal consistency by predicting several future actions at once, but a naïve implementation creates a stop-and-go rhythm: execute a chunk, pause for inference, then execute the next one. Physical interaction exposes this delay immediately.
Three families of ideas address this. Chunk continuation inpaints the overlap between an executing plan and a newly generated one. Adaptive horizons use longer chunks when behavior is predictable and shorter ones when uncertainty rises. Reactive execution checks progress during the chunk and revises the suffix when the world diverges from the plan.
This changes the evaluation question. A strong model must not only predict accurate actions offline; it must also preserve continuity under inference latency, disturbances, and imperfect observations.
VLA · 04
Reinforcement learning: adapt from experience
Imitation learning teaches a VLA to reproduce demonstrations. Reinforcement learning adds a different signal: what happened after the robot acted. It can reward successful rollouts, use a critic to identify valuable corrections, optimize a small control interface while freezing most of the foundation model, or learn preferences between better and worse trajectories.
Full-policy optimization can change the action generator directly, but it is computationally expensive and can destabilize useful pretrained behavior. Critic-guided methods separate evaluation from generation. Lightweight methods keep most of the VLA fixed and train a compact actor–critic or action-refinement layer. Preference-based methods avoid dense reward design by learning from chosen, rejected, corrected, or failed trajectories.
The deeper shift is from static imitation to continual improvement. A robot that learns from deployment can turn failures, interventions, and corrections into new training signal—but only if the learning loop is safe, sample-efficient, and resistant to catastrophic forgetting.
VLA · 05
Learning from human video: bridge the embodiment gap
Human video contains enormous behavioral diversity, including tasks and environments rarely captured in robot datasets. The challenge is that a human hand is not a robot gripper, and a human body does not share a robot’s action space.
One route learns a shared representation in which human and robot motion become comparable. Another converts human demonstrations into robot data through action retargeting, visual embodiment transfer, or executable reconstruction. Both approaches try to retain the task structure of the human demonstration while replacing the body-specific details.
If this transfer becomes reliable, the data pyramid changes dramatically: large-scale human video would no longer be only a semantic pretraining source, but a direct reservoir of embodied skill.
VLA · 06
Memory: decide what the present observation cannot tell you
A single image cannot reveal whether an object was already moved, which instruction was given several minutes ago, or how a similar attempt failed yesterday. Long-horizon robot behavior therefore needs a mechanism for preserving and selecting history.
Explicit multi-scale memory separates fast perceptual traces from slower semantic summaries. Recurrent memory carries compressed state from frame to frame or chunk to chunk. Retrieval-based memory searches previous experience for the few episodes relevant to the present decision. Latent memory converts selected history into the model’s native internal tokens.
The important question is not how much history to keep. It is which memory is useful for which decision. Manipulation may need a precise recent view of contact; task planning may need a long-term symbolic record; recovery may need an analogous failure from a prior episode.
VLA · 07
World models and new modalities: predict, verify, and feel
A world model gives the robot a way to ask, “What is likely to happen if I do this?” That prediction can support policy learning, generate interactive simulation, evaluate candidate behavior, or monitor execution for risk and inconsistency.
World models also expose the limits of vision alone. Transparent, reflective, deformable, occluded, and contact-rich objects may be difficult to understand from RGB images. Geometry can be strengthened with depth or explicit 3D information; contact can be grounded through tactile sensing and force. A genuinely embodied model should use the modality that resolves the uncertainty rather than treating every physical problem as a visual one.
Taken together, planning, feedback, memory, world models, and multimodal sensing turn a VLA from a one-step action generator into a system that can maintain intent across time and revise its behavior as the world changes.
04 · Discussion and Outlook
Four questions on the road to generalist robots
The presentation closes with four questions that cut across model architecture, data collection, and deployment.
- Which data is worth collecting? The field must balance real robot demonstrations, simulation, human video, synthetic data, failures, corrections, and preferences. The answer depends on the capability gap—not on scale alone.
- How should human–computer interaction data be used? Demonstrations, interventions, preferences, and everyday video each reveal different aspects of intent. The challenge is converting them into learning signals without erasing uncertainty or embodiment.
- Can a VLA act when vision is unreliable? Transparent, reflective, deformable, occluded, and contact-rich objects demand 3D, tactile, force, memory, and active perception rather than passive RGB input alone.
- When are vision, language, and action each necessary? A unified end-to-end model offers shared representations and simplicity. A hierarchy can assign semantics, planning, and control to specialized components. The best architecture may be neither fully unified nor permanently modular.
VLA research is moving outward from the action head. The next generation of systems will be judged not only by whether they can imitate a trajectory, but by whether they can understand a task, gather the evidence they need, plan at the right level, act without hesitation, recover from error, learn from experience, and transfer those abilities to a new body or environment.
That is the path from action generation to generalist robot intelligence: not one monolithic leap, but the deliberate integration of perception, control, learning, memory, prediction, and embodiment.
From biology to digital intelligence
“It seemed to me some time ago that you could sort of think of humanity as a biological boot loader for digital super intelligence.”
— Elon Musk, World Artificial Intelligence Conference, 2019