RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

The Problem: Robot Benchmarks Are Fragmented and Incomplete

Current benchmarks for evaluating robot manipulation policies are scattered and shallow. Some test only in simulation (fast but unrealistic), others only in the real world (costly and hard to reproduce), and most cover narrow skill sets with short-horizon tasks. There's no unified way to systematically measure whether a policy can generalize, remember, execute precise movements, handle long sequences of actions, or follow natural language instructions—let alone do all of this in both simulated and physical environments.

RoboDojo: A Unified Evaluation Framework

The authors introduce RoboDojo, a benchmark that bridges simulation and reality. It includes 42 simulation tasks and 18 real-world tasks designed to test five complementary dimensions of robot capability: generalization (how well policies adapt to new conditions), memory (multi-step task dependencies), precision (fine-grained control accuracy), long-horizon execution (sustained action sequences), and open-vocabulary instruction following (understanding natural language commands).

The simulation component runs on Isaac Sim with heterogeneous parallel processing to scale evaluation cheaply. The real-world component, called RoboDojo-RealEval, provides a reproducible cloud-accessible system with standardized hardware, automatic scene resets, and consistent evaluation protocols—solving the reproducibility nightmare that plagues physical robot research.

The key insight is that integration matters as much as evaluation. The authors built XPolicyLab, a unified interface that lets researchers integrate a policy once and have it evaluated across both simulation and real settings with minimal code changes. This removes friction that usually forces researchers to rewrite policies for different environments.

Results and Scale

The authors integrated 30 existing policies into XPolicyLab and ran them through RoboDojo, creating a public leaderboard and performance analysis. This gives the community immediate concrete data on how current generalist policies perform and where they fall short. By publishing task definitions, evaluation metrics, and baseline results, RoboDojo establishes reproducible ground truth.

Why This Matters

Robot manipulation research has been fragmented: a policy might excel in one paper's simulation benchmark but fail on a different group's real robot. RoboDojo solves this by providing a common language and shared infrastructure. The benchmark's five dimensions move beyond toy tasks to real deployment concerns—a policy that works in short, simple scenarios may crumble under long-horizon pressure or when asked to generalize to new objects.

The real-world component is particularly novel. Most benchmarks avoid real robots due to cost and logistical burden. RoboDojo's cloud-accessible standardized hardware and automated resets make real-world evaluation tractable at scale, reducing the sim-to-real gap that has plagued the field.

For practitioners, this means faster iteration and clearer signals about what works. For researchers, it's a shared yardstick that accelerates progress by making trade-offs visible across methods.