What a High Robot Manipulation Benchmark Score Can Hide

The researchers examine shortcut solvability, statistical evidence, repeated adaptation to the test distribution and dependence on the source of training data. They apply these diagnostics to LIBERO, CALVIN, SimplerEnv, RoboCasa and RoboTwin 2.0.

On LIBERO, they report that a 0.09B probe without a language encoder reaches results near leading reported scores. This challenges how much a high aggregate score alone establishes about language-conditioned behavior. It does not show that language is unnecessary for every manipulation task.

The project's CALVIN analysis changes the block poses while remaining within the training range. The reported average number of completed tasks out of five falls from 4.17 to 3.14. That result motivates looking at the distribution of starting states, rather than only the named task suite.

A narrow recurring arrangement can reward a policy that depends on familiar placement. For a reader interested in deployment, the next questions concern the size and direction of position changes, whether objects stay visible and whether the same object identities appear during training. Each condition defines a different generalization claim.

Two papers can report the same percentage after using different numbers of attempts, different task mixtures or different reset rules. The percentages alone leave the reader unable to reconstruct the comparison. Publish successes and attempts per task, the tested seeds and the success detector.

For example, 90 successes from 100 attempts and 900 from 1,000 both produce 90 percent. The second sample contains more information about uncertainty under the same sampling assumptions. Repeated runs from one fixed scene also differ from independent runs across varied scenes. More repeats of the same situation do not automatically establish wider coverage.

A comparison between two policies should explain whether they used paired starting states. A failed reset, simulator crash or timed-out episode needs a stated treatment. Excluding difficult episodes after observing the result can alter the reported success rate.

Record the dataset release, task-specific fine-tuning data, object identities and checkpoint selection procedure. The audit shows why closeness between training examples and test conditions can make a smaller model look competitive with a larger model trained on broader data.

This also affects claims about open models. Releasing weights lets another team run the policy, but a missing training-data description can still prevent that team from interpreting a benchmark gain. Model size, compute and data coverage belong beside the score.

A practical result card would identify the model checkpoint, simulator version, task list, environment seeds, training sources, action frequency and time limit. For each task it would provide successes, attempts, intervention counts and a description of the success detector.

A second evaluation can vary one condition at a time. Move an object, change its identity, alter the camera position or rewrite the instruction while retaining its meaning. Report the size of each change. This proposed reporting format makes failure patterns visible without collapsing every variation into one average.

Simulation remains useful for controlled comparisons. A humanoid working with physical objects also needs tests for contact, sensor delay, actuator limits and recovery. The audit supplies diagnostics for interpreting a score; it does not certify a physical robot for deployment.