Gemini Robotics ER 2: The High-Level Brain Coordinating Robots as a Team

Google describes ER 2 as a high-level brain. Developers expose low-level interfaces such as VLA policies, navigation functions or manipulation APIs as tools. ER 2 receives multimodal streams, chooses which tool to call and uses the next observation to decide whether the action worked.

This separation matters because task reasoning and motor control run on different timescales. A planner may decide to fetch a container once every few seconds; a leg controller may need updates hundreds of times per second. Asking one model to do both can create unnecessary latency and make safety harder to verify.

A multi-step robot task needs a state variable humans rarely notice: where am I in the procedure? If a drawer was already opened but becomes occluded, a reactive policy may repeat the step or move on incorrectly. ER 2 watches continuous video to identify task boundaries and key events.

Google reports 57.4% accuracy on progress classification and 91.3% accuracy for moment finding, with a 0.96-second mean absolute temporal distance. These numbers are not perfect, but they turn progress recognition into a measurable subsystem rather than leaving it implicit inside a demo.

ER 2 integrates with the bidirectional Gemini Live API. The model can process the stream while the robot is executing, preparing the next decision rather than waiting for an action to finish before beginning all reasoning. This matters when high-level pauses make physical workflows slow or unnatural.

Parallel reasoning does not remove the need for synchronization. The high-level state can become stale if the low-level controller fails, if a human moves an object or if a tool call returns late. Robust systems need timestamps, explicit completion signals and a way to invalidate a plan when the world changes.

Google demonstrates ER 2 orchestrating an Apptronik Apollo 2 and a Franka F3 Duo. A heterogeneous team can assign mobile tasks to the humanoid and precision manipulation to a fixed arm rather than forcing one robot to be optimal at everything.

That advantage introduces coordination costs: shared workspace collisions, handoff timing, inconsistent object state, communication latency and deciding which robot owns the next subtask. Multi-robot AI only creates value when those costs are lower than the parallelism or specialization gained.

ER 2 can call user-defined functions and external tools. Google also demonstrates it with Spot APIs. This turns an existing robot stack into a capability registry: navigate, grasp, inspect, search, ask a human or stop safely.

The design is attractive because a verified low-level skill can be reused by different planners. The failure mode moves upward, however. A planner can select the wrong tool, pass bad arguments or assume a skill has guarantees it does not have. Interfaces therefore need explicit preconditions, postconditions and safety limits.

Long tasks fail because the physical world does not obey the plan. Objects slip, doors resist, humans intervene and cameras lose sight of the target. A useful embodied agent must treat the observation after an action as evidence, not as a cosmetic update.

ER 2's architecture points toward a broader trend in agentic robotics: plan one bounded step, execute through a constrained interface, observe again and recover. That loop is slower to describe than end-to-end autonomy, but it is easier to inspect and potentially easier to govern.