How Do Robots Learn? Teleoperation, Simulation and Vision-Language-Action Models

A chatbot can learn from the whole public internet. A robot cannot, because the internet is full of words and pictures but almost empty of the thing a robot needs most: a record of which motor moved, by how much, at what moment, to fold a towel or open a jar. Almost everything happening in robotics right now is an attempt to get around that shortage.

There are three main routes. People can steer the robot and record what they did. The robot can practise in a simulated world, where mistakes are free. Or a large model that already understands language and images can be taught to output movements as well. Most serious systems now combine all three.

Why it matters now

This week’s robotics news was almost entirely about this problem. Physical Intelligence opened a hosted API that lets outside developers upload their own demonstrations and fine-tune the company’s robot models on its compute. A research group published VioLA, a humanoid policy trained on a pool of 140.6 million frames, 93.2% of them videos of people rather than robots. And Google pitched Gemini 4 Argon as a robot brain, reporting that 4,500 simulated practice episodes lifted a real task’s success rate from 56% to 75%.

Three stories, three routes to the same goal: more useful training data for machines that have to act in the physical world.

How it works, step by step

Step 1: Teleoperation, or learning by being steered

Teleoperation means a person controls the robot remotely, often by moving a matching pair of “leader” arms or wearing a motion-capture rig, while the robot’s “follower” arms copy them. Every camera frame and every joint position is logged.

Think of a driving instructor with dual controls. The learner (the robot) feels exactly what the instructor did, at exactly the moment they did it.

The recordings are then used for imitation learning: a model is trained to predict the next action from what the robot currently sees. A widely cited 2023 Stanford project called ALOHA showed how far this can go cheaply. Its authors reported that the whole two-arm system cost under $20,000 using off-the-shelf robots and 3D-printed parts, and that their method learned six fiddly real-world tasks, such as slotting a battery, with 80 to 90% success from only 10 minutes of demonstrations.

The catch is cost at scale. Every demonstration needs a person, a robot and time. That is why teleoperation data is scarce, and why the next two steps exist.

Step 2: Simulation, or practice in a video game

In simulation, the robot’s body and its surroundings are recreated in software, with physics. The robot can attempt a task millions of times, fail safely and learn from a reward signal (a score for doing well), which is called reinforcement learning.

The problem is the sim-to-real gap: the simulated world is never quite like the real one. Friction, lighting, cable stiffness and camera noise all differ. A policy that is perfect in the game can be clumsy in a kitchen.

One well-known fix is domain randomisation, described in a 2017 paper by researchers at OpenAI and UC Berkeley. Instead of trying to make the simulation perfectly realistic, you make it wildly varied: random colours, textures, lighting and camera positions. A model trained across all that variety tends to treat the real world as just one more variation. It is like a pilot who has trained in every kind of weather, so a real storm is nothing new.

Step 3: Vision-language-action models, or borrowing common sense

A vision-language-action model (VLA) takes in camera images and an instruction in plain language, and outputs robot actions. It is usually built by starting from a model already trained on huge amounts of web text and images, then teaching it to “speak” in motor commands as well as words.

Google DeepMind’s RT-2, announced in July 2023, is the best-known early example. It learned from both web data and robot data, and Google reported that it roughly doubled its predecessor’s success on scenarios it had never seen, from 32% to 62%, across more than 6,000 robot trials. The point was not that RT-2 had more robot practice. It was that knowledge from the web (what a “toy dinosaur” or “something you could use as a hammer” is) transferred into action.

The analogy is a translator. The model already understands the “language” of pictures and words; robot training teaches it one more language, movement.

Step 4: Pooling data across robots

Because no single lab can collect enough demonstrations, labs have started pooling them. The Open X-Embodiment project, published in 2023, assembled data from 22 different robot types through a collaboration of 21 institutions, covering 527 skills. The bet is that a model trained on many bodies learns something general about manipulation that carries over to a new one.

Step 5: Learning from people, not just robots

The newest trick is to treat ordinary video of people as training data. People film themselves doing things constantly, so this source is enormous. The difficulty is that a human hand is not a robot gripper. VioLA’s approach, covered this week, is to translate both human and robot movement into a shared intermediate description of body motion, so a clip of a person becomes a usable example for the robot.

A worked example: teaching a robot to load a dishwasher

Imagine a small company wants a robot arm to load mugs into a dishwasher.

1. Collect. An operator teleoperates the arm through 200 loads, with mugs in different positions. That produces a few hours of camera footage paired with joint movements.
2. Start from a base model. Rather than train from scratch, the team fine-tunes a VLA that already knows what a mug, a handle and a rack look like.
3. Practise in simulation. They build a simulated kitchen, randomise mug colours, lighting and rack positions, and let the policy rehearse thousands of loads.
4. Test for real. On the real robot, the arm succeeds most of the time but fumbles mugs lying on their side.
5. Close the loop. The failures tell the team what to record next: 30 more teleoperated demonstrations of sideways mugs. Retrain, retest.

That collect, train, test, repeat loop is essentially what Physical Intelligence’s new API packages as a service.

What people get wrong

“The robot understands what it is doing.” A VLA links words and images to movements very effectively, but the reliability you would expect from something that truly understood a task is not there yet. Small changes in lighting or object position can still cause failure.

“A 100% success rate means the problem is solved.” Robotics results are often reported on a small number of tasks, trials or robots. Our VioLA brief put it plainly: the numbers need their denominator. Always ask how many tasks, how many attempts and whether anyone else has reproduced it.

“Simulation will replace real data.” Simulation is cheap and safe, but the sim-to-real gap means nearly every serious system still needs real-world testing and real demonstrations.

“More robot data is all that matters.” RT-2’s lesson was the opposite: a lot of what helps a robot generalise comes from web knowledge that has nothing to do with robots.

Where it is heading

Three directions are visible in this week’s news and the research behind it.

• Robot learning as a service. Physical Intelligence’s API moves fine-tuning onto the company’s own cluster, so a team with a robot and demonstrations does not need its own training infrastructure.
• World models that predict the future. Models such as SmolWAM and LimX’s HWAM predict what the scene will look like after an action, not just which action to take, which gives the robot a way to “imagine” outcomes.
• Big general models as the reasoning layer. Google’s Gemini 4 Argon post argues that a frontier model can plan and reason for a robot, with smaller specialised models handling the fast motor control. Its benchmark figures are Google’s own and unaudited.

Key terms

• Teleoperation
• Sim-to-real
• Vision-language-action model (VLA)
• Embodied AI
• World model
• Fine-tuning
• Transformer

Sources

• Zhao et al., “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware” (ALOHA), arXiv:2304.13705, 2023.
• Tobin et al., “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”, arXiv:1703.06907, 2017.
• Google DeepMind, “RT-2: New model translates vision and language into action”, 28 July 2023.
• Open X-Embodiment Collaboration, “Open X-Embodiment: Robotic Learning Datasets and RT-X Models”, arXiv:2310.08864.
• AIooze briefs this week: Physical Intelligence Pi API; VioLA; Gemini 4 Argon robotics; SmolWAM; LimX HWAM (linked above, each citing its primary source).

Last reviewed: October 2026