If a policy wins in SIMPLER but loses on the robot, first check whether the two evaluations show the policy the same kind of image and execute its output in the same way. A plausible-looking scene is insufficient. SIMPLER was built to make simulated evaluation informative about real manipulation policies by reducing visual and control disparities; its reported sim-to-real agreement comes from the authors’ paired tests, not a guarantee for a new robot cell. The SIMPLER paper frames those gaps, and the official repository exposes the evaluation and policy-integration paths.
Consider a hypothetical comparison of two checkpoints for a drawer task. Checkpoint A leads in simulation; B leads in ten paired physical trials. Before attributing the reversal to “the sim gap,” retain the images and actions for the same starting conditions. The disagreement might begin before the policy emits an action, in the conversion of that action, or in the simulated response. Each case calls for a different repair.
Find where the two rollouts first diverge
Start with the observation reaching the model, not a beauty comparison of the scene. Compare camera position, crop, resolution, image normalization, object pose and the robot’s visible geometry. Preserve a real frame and the corresponding simulated frame as model inputs. A simulator can look convincing to an operator while presenting a different crop or object scale to the policy. If the policy reacts differently to these inputs, changing controller gains will not repair the visual mismatch.
Next, feed recorded real evaluation frames through the same policy inference code used for simulation and inspect the resulting actions. SIMPLER’s Adding New Policies instructions recommend this diagnostic when policy behavior deviates from real rollouts. If real-frame inference still produces a sensible command but the simulated robot moves incorrectly, check action ordering, units, coordinate frame, gripper convention and action-space conversion. This is a diagnostic clue, not proof that one conversion caused the ranking reversal.
| Earliest disagreement | Compare on a paired episode | A useful next test |
|---|---|---|
| Policy input | Real and simulated frames after the same crop and preprocessing | Hold the policy and controller fixed; change one visual condition at a time. |
| Policy output | Commands produced from saved real frames in both inference paths | Log dimensions, order, frame, units and gripper sign before the simulator adapter. |
| Executed motion | Policy command versus robot motion at the same control interval | Check controller mode, update rate, saturation and termination behavior. |
| Outcome only | Paired success and failure modes, not a single pooled score | Repeat matched starts and report uncertainty before choosing a checkpoint. |
These rows are an investigation order, not a claim that every mismatch has one cause. A task may expose both perception and execution differences. Change one interface at a time and rerun the same policy pair; otherwise a ranking improvement is hard to attribute.
Compare the ranking under a stated evaluation setup
The repository documents two real-to-sim setups. Visual Matching aims to align a named physical scene with simulation. Variant Aggregation averages across changes such as backgrounds, lighting and distractors. Use the first when deciding which checkpoint suits one known cell; use the second when the question is robustness across visual conditions. Keep their results separate. Averaging variants can hide a failure in the specific cell where the robot will run, while tuning every texture to one camera can hide sensitivity elsewhere.
For either setup, record task and object versions, initial states, image pipeline, policy checkpoint, action adapter, control frequency and rollout rule. The repository’s comparison section provides Mean Maximum Rank Violation and Pearson correlation as ways to compare simulated and real policy outcomes. A useful local report shows per-policy results and the ranking, then states how many policies and tasks produced it. An aggregate correlation from the paper cannot be transplanted to a new cell. Even agreement on success rates may miss a changed failure mode that matters during operation.
If the evaluation is really about which transfer claim a suite tests, the LIBERO suite guide addresses that separate experimental choice. SIMPLER’s narrower job here is checking whether a simulation preserves the relative behavior of your candidate policies.
Stop at a calibrated comparison
The endpoint is a paired report: same checkpoint identities and comparable starts; real and simulated images; raw policy commands and executed actions; per-task success and failure descriptions; and the selected SIMPLER setup. If the ranking still conflicts after the input and action paths are checked, retain that conflict. It is evidence that this simulated test is not currently a reliable selector for those policies. No simulator or robot was run for this article; the procedure is a proposed diagnostic grounded in the authors’ paper and repository.