How AI models connect to robot sensors and controllers

Separate the model from the surrounding software

OpenVLA is a research example of a vision-language-action model. Its project describes a model that uses visual observations and language to predict robot actions. LeRobot documents a broader workflow around supported robots and imitation learning. Neither name alone establishes compatibility with every arm, camera or gripper. [1] [2]

A motion planner has another role. MoveIt accepts requests and constraints and constructs trajectories for a configured robot model. A learned policy can be part of a larger system that also uses conventional planning and control. [3]

Scroll sideways for all columns.

An action vector needs units and a reference frame

In an illustrative interface, an action contains three translation values, three rotation values and a gripper command. A value of 0.01 means very different movements when interpreted as meters, normalized joint position or an incremental command. A program can accept the vector without interpreting it as its author intended.

Before a hardware trial, document whether a movement is absolute or relative, which coordinate frame it uses, its units and how long it remains valid. Specify the behavior when observations stop arriving. These are interface requirements for the example, not instructions to bypass the robot's protective systems.

A useful integration test first compares predicted actions with recorded observations and the intended command definition. Physical testing then belongs in a controlled setup using the manufacturer's operating procedures.

Match the recording setup to the deployment setup

LeRobot's supported-robot documentation organizes hardware setup, calibration, teleoperation and recording as explicit steps. Keep the camera view, robot configuration and calibration with the dataset. A dataset name alone is not enough to reconstruct those conditions. [2]

Consider a cup-picking policy trained from one camera position. Moving the camera changes the visual input even when the robot and cup remain in the same places. An evaluation should distinguish this change from a new object, a different gripper or a lighting change. Altering all four at once makes a failed trial difficult to explain.

Separate task coverage from completion rate

Here is an illustrative evaluation. A team selects 12 household objects, runs five trials on each and gets 42 unassisted completions from 60 attempts. That is 70% on the selected set. It gives no measured rate for objects that were never included.

Report results per object as well as the total. One object completed in all five trials and another completed in none require different follow-up work. Record grasp retries and interventions instead of silently resetting the scene.

Keep simulation and hardware results in separate columns. A physical success should include the intended final object state, not merely a model producing an action or a robot reaching a pose.

What to keep with a reproducible result

A usable project record includes the model checkpoint, code revision, software environment, robot variant, camera arrangement, calibration and action definition. Add the task description, trial log and conditions that caused a stop.

The linked projects provide starting points for this record. Their documentation does not certify a custom assembly or supply an independent safety case for a workplace. A downloaded checkpoint should not be described as a ready-to-run general robot operator.