Black & White – the premiere of Brody Neuenschwander’s book
For decades, we’ve associated industrial robots with machines that are extremely precise but not very flexible. They could flawlessly repeat a programmed sequence as long as every subject was in its designated place and the environment did not deviate from the designer’s specifications. A change in the product, workstation layout, or task execution usually required reprogramming the system, reconfiguring it, and retesting.
Vision-Language-Action models, or VLA for short, propose a different approach. They combine image analysis, language understanding, and motion control, allowing the robot to receive a description of the goal rather than a complete list of instructions. For example, this could be the command “arrange the items from smallest to largest,” which, while conveying the intention, does not specify coordinates, the order of grasps, or the arm’s trajectory. The VLA system is designed to recognize objects on its own, determine the relationships between them, and translate the result of this reasoning into physical action.
Of course, this does not yet signify the birth of a fully universal machine that can be placed in any warehouse and immediately entrusted with any task. However, VLA represents a significant step forward from automation based on rigid procedures toward robotics that responds to the meaning of a command and the situation observed by cameras.
What exactly is the Vision-Language-Action model?
Simply put, the VLA model can be broken down into three components that make up its name and perform interrelated functions:
- Vision is responsible for interpreting the scene—that is, detecting objects, their locations, characteristics, and relationships to one another.
- Language is used to understand commands and connect words with elements visible in the surroundings.
- Action converts the recognized intention into a series of control commands, such as changing the gripper's position, rotating a joint, or moving a mobile platform.
In a traditional system, these stages are typically handled by separate modules, between which engineers design rules for information exchange. The VLA model, on the other hand, aims to learn a unified mapping—from image and command to motion. In the paper describing RT-2, the robot’s actions were represented as tokens—symbols processed similarly to text fragments. The model was then jointly fine-tuned using web data linking language to images and robot trajectories.
It is precisely this combination of visual-linguistic knowledge and motor experience that defines the potential of VLA. The robot does not need to be familiar only with objects that appear in a single, precisely described process. It can draw on broader concepts—such as size, color, purpose, or similarity—and then apply them to the current scene.
Voice commands, however, require further explanation. A typical VLA model accepts text-based instructions, so an employee’s spoken words are first converted to text by a speech recognition system. Solutions that process the audio signal more directly are already emerging, but this is not yet a feature of every model classified as a VLA.
From Recommendation to Action
Let’s imagine a workstation with containers holding various parts. The operator asks the robot to set aside the defective parts for inspection and sort the rest by size. For a human, this is an understandable task. For a traditional automated system, each concept would have to be described in advance using rules, object classes, and possible process states.
The VLA system analyzes camera footage, associates words with visible objects, and generates subsequent movement sequences. It does not need to plan the entire operation all at once. It can operate in a loop—that is, execute part of the movement, reassess the scene, correct the trajectory, and proceed to the next step. This closed-loop cycle of perception and action is necessary because the object may move, the grasp may be imprecise, or a person may enter the workspace.
Training on multiple datasets and robot types plays a significant role. OpenVLA, one of the most important open-source projects in this field, has 7 billion parameters and was trained on 970,000 robotic episodes from the Open X-Embodiment dataset. This model can support various configurations, and it can be adapted to a new setup using methods that fine-tune only a subset of its parameters.
Open-source code, ready-to-use model scales, and simulators lower the barrier to entry for laboratories and research and development teams. However, they do not eliminate the need to collect demonstration data from a specific machine. The same goal must be achieved through different movements performed by a small tabletop arm, a mobile robot, and a humanoid robot.
Why is logistics a natural field of application for VLA?
A warehouse combines a large number of repetitive operations with constant variability. Goods vary in shape, packaging, and location; orders create different sequences; and the layout of zones may change with the seasons. Traditional automation works best where this variability can be limited. VLA is designed to help machines cope even when it is impossible to anticipate every situation in advance.
Potential applications therefore include:
- picking orders that contain products of various shapes,
- sorting packages based on label information and visual characteristics,
- placing merchandise in the designated zone without defining a separate program for each category,
- transporting containers between workstations based on a natural-language command,
- checking that the product is properly arranged, packaged, or labeled,
- Supporting the employee with tasks whose order changes during the shift.
The scale of the need is significant. According to the International Federation of Robotics, sales of professional service robots designed to transport goods and cargo increased by 44% between 2021 and 2022. The organization also points to a labor shortage in logistics and the growing use of vision systems and autonomous transport platforms.
field of study This trend is also evident in the solutions being developed by the largest operators. Amazon has announced a new version of its Proteus mobile robot, which can be given tasks using everyday language. The system is ultimately designed to transport containers and heavy carts between different zones within a facility. Although the manufacturer does not explicitly refer to it as a VLA model, this can be seen as evidence of a broader trend toward language-based robot interfaces.
Automation for Medium-Sized Companies, Too?
A traditional automated production line can be costly for reasons other than just the equipment. A significant portion of the investment goes toward integration, programming process variants, and subsequent modifications. If the VLA system can be fine-tuned through demonstrations and commands, the time needed to implement the next task can be reduced. In a company handling a variable product mix, this is more important than the record-breaking speed of a single, unchanging operation.
Three phenomena are driving democratization. The first is the development of open models, such as the aforementioned OpenVLA. The second is simulation environments, where data can be generated and robot behavior tested without halting operations on the factory floor. The third is fine-tuning techniques that require modifying only a small part of the model, which reduces the demand for computational resources compared to training from scratch.
A good example of a complete ecosystem is the NVIDIA Isaac GR00T platform. It combines base models for robots, data generation tools, simulation, and solutions for running models on an onboard computer. The manufacturer notes, however, that some of the available versions are for research and prototyping purposes, and adapting them to new hardware still requires demonstrating a specific task.
For a medium-sized company, therefore, the most realistic approach is not to purchase a “jack-of-all-trades robot.” It is more likely that the company will implement a ready-made hardware platform with a base model and then fine-tune it for a few processes with high labor costs or high variability. Cost-effectiveness depends on the stability of the model, the frequency of product line changes, integration costs, data availability, and the ability to securely isolate the work area.
The flexibility of the VLA does not mean it is infallible
A language model can generate an incorrect response without any physical consequences. A robot’s error, on the other hand, can damage goods or machinery, or endanger people. Therefore, VLA should not independently replace deterministic safety systems. Speed and force limits, emergency stops, zone control, collision detection, and verification of permitted movements must operate independently of the model’s decisions.
Resilience to changes in conditions also remains a challenge. Ina study involving seven models and four manipulation tasks, slight modifications to the camera’s position reduced the average number of successful trials to 34% of the baseline result. Changes in lighting also reduced performance, and unfamiliar objects primarily hindered the correct localization of the target.
Of course, these aren't all the challenges, because we also need to take into account aspects such as:
- latency, because a large model must respond quickly enough to ensure that the controls remain smooth,
- data quality, as the demonstrations should include various camera settings, lighting conditions, packaging, and exceptional situations,
- predictability, which is needed to explain why the robot chose a specific movement,
- cybersecurity, including the protection of images, commands, models, and communication channels,
- validation that cannot be limited to average performance in a laboratory benchmark.
However, the visual input requires special attention. Researchers have shown that a properly prepared pattern placed within the camera’s field of view can disrupt the VLA model’s trajectory, even during tests on a physical robot. These results do not prove that every system can be easily attacked in the same way, but they do confirm the need for independent motion monitoring and resilience testing prior to deployment.
What might a secure VLA implementation look like?
A sensible pilot project should start with a narrow, measurable process. Instead of the instruction “manage the reception area,” it is better to choose a task involving sorting several categories of packaging or moving containers between defined points. area The scope of the activity remains limited, but the method of execution can be more flexible than in a traditional program.
The next steps include collecting demonstrations, training or fine-tuning the model, simulation tests, trials on a dedicated test bench, and gradually expanding the scope. It is important to measure more than just the percentage of completed tasks. Other important factors include the number of human interventions, cycle time, grasping errors, instances of safety zone violations, as well as the robot’s ability to adapt to changes in lighting and its behavior when faced with ambiguous commands.
The system should also be able to refuse to act. If the camera cannot see the entire scene, the command is contradictory, or the object is outside the safe range, guessing is not the correct response. The robot should stop, report the problem, and ask the operator for clarification.
VLA as a New Human-Machine Interface
Until now, humans have primarily described the procedure—that is, the coordinates, the sequence of movements, and the responses to specific signals. With the VLA approach, they can focus more on describing the result and leave part of the planning to the machine.
This shifts the skill sets required in the workplace. Specialists in automation, mechanics, and safety will still be essential, but the importance of data engineering, simulation, model testing, and human-robot interaction design will grow. Those who wish to deepen their knowledge of artificial intelligence, computer vision, robot programming, and autonomous systems may consider pursuing a postgraduate program in Autonomous Robotics at our university.
VLA does not eliminate robot programming, but rather shifts part of it to the level of learning from data and defining constraints. The machine is given more freedom in choosing its actions, so the system surrounding the model must define the boundaries of that freedom even more precisely. If flexibility can be reconciled with repeatability and safety, robots that understand the purpose of a task could become one of the most important tools in the next phase of automation in warehouses, logistics, and manufacturing.
See other news
Event to disseminate the results of the InfoTester project
Students from the " PJAIT " jointly designed an acoustic panel in Building C
