When two bimanual policies finish the same task, compare how they reached the outcome: progress through task stages, motion length and duration, coordination between arms, and unwanted contact or slip. RoboEval organizes these measurements so a success label can be supplemented with evidence about execution quality.
The distinction is practical. Two policies may both lift a tray, while one repeatedly tilts it and corrects the error. A single completion bit keeps the shared outcome but discards the behavior that explains why the second policy is harder to use.
Start with the task’s coordination requirement
The RoboEval project pairs manipulation tasks with variations and fine-grained metrics. Its task set contains different coordination patterns: carrying a tray requires a different relationship between hands from handing over an object or pushing separate box flaps.
That means a metric’s preferred direction is conditional on the task. Small gripper-height difference is meaningful when the hands should carry a level object together. During a handover, unequal heights may be intentional. A value should be interpreted against the required relationship, not minimized simply because it appears on a dashboard.
The bimanual control guide explains the physical relationship between object motion, relative pose and internal forces. Evaluation adds another layer: which recorded signals expose a failure of that relationship during this particular task?
Keep outcome, effort and interaction separate
RoboEval’s paper, Section III-D, distinguishes task outcome and progression from behavioral measurements. It describes path lengths, completion time, jerk, collision and slip counts, and coordination measures such as gripper-height discrepancy and velocity divergence.
Use these as separate diagnostic views before constructing any combined score:
| Observation | A question it helps answer | Context needed |
|---|---|---|
| Stage progression | Where did the episode stop advancing? | Stage definitions and transition conditions |
| Cartesian or joint path length | How much motion was used? | Units, participating joints and completed stages |
| Completion time | How long did the episode take? | Timeout rule and whether failures are included |
| Jerk | How abruptly did acceleration change? | Sampling interval and signal processing |
| Slip or contact events | Which unwanted interactions occurred? | Event definition and relevant task phase |
| Arm coordination | Did the required relationship hold? | Symmetric, alternating or role-specific behavior |
A shorter path is not automatically better if an episode stopped before reaching the difficult stage. Report success and progression beside the path measurement. Likewise, averaging only successful runs answers a different question from averaging every attempt. State that denominator.
Inspect the measurement definition before comparing numbers
The current project page describes trajectory length as integrated path length, while the paper also uses the phrase for control-step count in its temporal-efficiency discussion. Use an explicit field such as “Cartesian path in meters” or “number of control steps” in an evaluation record. The label alone is ambiguous.
The paper’s jerk definition uses a third finite difference divided by the cube of the control interval. This makes sampling part of the quantity. Comparing values computed from differently sampled trajectories without inspecting that definition can turn a preprocessing difference into an apparent policy difference.
For a proposed diagnostic, retain a time-aligned trace around a slip: object pose, both gripper poses, contact state and the current stage. That can show whether the error began during grasp establishment, lifting or placement. The trace localizes an event; identifying its physical cause may require additional sensing or a controlled intervention.
Preserve failures and task variations
Evaluate the same variations, initial-state distribution and termination rules when comparing policies. Keep per-episode records so a low average does not conceal a small set of severe events. Separate task stages when their coordination demands differ.
RoboEval supplies a simulation evaluation framework, and this article examines its published definitions rather than running the benchmark. Sources were checked October 16, 2026. The practical output is a small task-specific metric panel whose units, denominator and event definitions remain visible beside success.