Read a Colosseum V2 result as a matrix of task success under named changes. Keep the unperturbed score, the perturbed score, and their difference together. A single robustness average can conceal a policy that tolerates visual changes but fails under rephrased instructions or altered object geometry.
The Colosseum V2 project describes separate single-arm and bimanual suites. Its initial ACT and π0.5 comparisons train separate models for each suite using 100 perturbation-free demonstrations per task, then evaluate each perturbation individually over 200 episodes. The authors report different weaknesses across visual, language, and action changes; this article interprets that protocol without presenting a new benchmark run.
Keep absolute performance beside the drop
Consider this invented example, with success measured over an equal number of episodes:
| Policy | Base success | Changed lighting | Change from base |
|---|---|---|---|
| A | 90% | 60% | −30 percentage points |
| B | 60% | 50% | −10 percentage points |
B loses less under the change, but A still completes more trials in that condition. The question of sensitivity and the question of absolute task performance lead to different comparisons. A percentage-point change is also different from a relative percentage loss; state which you use.
For a real report, include episode counts and uncertainty alongside these columns. Preserve task-level rows before aggregating. A mean across tasks can hide a severe degradation on the one manipulation your application needs.
Separate the factor you changed from the factors you trained on
Train-on-clean/test-on-perturbed evaluation asks how a policy behaves under an introduced shift. Training on the same randomized conditions asks another question. Do not compare those entries as if both demonstrate the same kind of generalization.
Record object color, camera pose, instruction variation, or size changes by their exact factor and magnitude when available. Keep combined perturbations separate from individual-factor rows. A combined failure may arise from interacting changes, so it cannot be assigned to lighting merely because lighting was among them.
An illustrative review starts with the worst degradation in a relevant task, then watches matched clean and perturbed rollouts. If both already fail before grasping, the perturbation may not explain the dominant error. If only the perturbed run misidentifies the object, a visual boundary becomes a more plausible follow-up target. That is diagnostic reasoning, not a causal finding from an unperformed experiment.
Choose a hardware check from the failure pattern
The authors’ hardware validation uses a separate multi-task ACT policy trained from scratch with 60 demonstrations per task across three single-arm tasks. It supports the ordering of relative degradation within that tested setup. It does not establish absolute hardware success for every simulated task, every bimanual policy, or every robot.
Use the matrix to select a small physical validation question: for example, whether the largest relevant visual degradation recurs on your camera and object setup. Keep its result separate from the simulation score. The broader sim-to-real transfer guide explains why a paired physical check remains necessary.
For model selection, write the claim with the task suite, training conditions, and perturbation family attached. A policy can be less sensitive to one shift and still be the weaker choice for another. Project protocol checked October 15, 2026.