Vision-language-action model (VLA)

A model that takes camera images and a written instruction and outputs robot actions.

Also known as VLA, VLA model

A vision-language-action model, or VLA, combines what a vision-language model knows about pictures and words with the ability to output movement commands. Typically it starts from a model trained on web text and images, then is trained further on robot data.

The idea is that web knowledge, such as what a “toy dinosaur” is, carries over into action. Google DeepMind’s RT-2, announced in July 2023, is a well-known early VLA; Physical Intelligence’s π0 is another.