An execution score for a generated manipulation video measures both the predicted behavior and the process that converts it into robot actions. Retain that conversion with the reconstructed scene before attributing a failed rollout entirely to the world model. A visually plausible video and a usable embodied action sequence require different evidence.
The RoboWM-Bench paper, revised May 14, 2026, converts generated human-hand and robot videos into action sequences and evaluates them in physically grounded simulation. It uses real-to-sim scene reconstruction and reports that visual plausibility and executability can diverge. Its analysis discusses spatial reasoning, contact prediction and geometric distortion; the main model scores are simulation-based, rather than proof of deployment. The paper separately checks reconstruction and action extraction against real trajectories.
Retargeting and inverse dynamics can fail differently
The paper’s video-to-action methods follow two paths. Human videos use HaMeR to recover 3D hand poses and retarget them to end-effector position, orientation and gripper opening. Robot videos use an inverse dynamics model (IDM) that takes two consecutive frames and predicts a joint-space action chunk. A problem in one path is not automatically evidence about the other. Keep the input family and conversion method with each score.
Imagine an illustrative clip that shows a hand lifting an object while the converted action closes the gripper beside it. The failure may begin in the predicted geometry, the translation of that geometry, or the simulator's recreated setup. The final outcome alone cannot choose among those explanations.
Preserve the generated video identifier, translator revision/settings, output action sequence, embodiment configuration and reconstructed scene. Keep intermediate pose or contact estimates where the conversion pipeline exposes them. These are proposed provenance records, not asserted release fields.
Inspect whether the action sequence preserves object identity, intended contact and temporal order. An array with valid dimensions can still represent a different interaction from the source video.
Use comparison inputs that isolate the conversion
For a proposed diagnostic, pass a reference video with known task evidence through the same translator. A poor reference conversion limits what a generated-video failure can establish about the world model. Keep the translator fixed when comparing models unless the experiment explicitly tests the joint pipeline.
Use the following proposed diagnostic order when a generated clip looks convincing but its replay fails. These are review decisions, not results from a new RoboWM-Bench run.
| Comparison you can inspect | Next evidence to examine | Conclusion to avoid |
|---|---|---|
| A known successful reference video also fails after conversion | Recovered pose or action, embodiment mapping and the first missed contact | That the generated-video model alone explains the failure |
| The reference converts successfully; generated videos fail with the same translator | Generated geometry and contact order at the first failed stage | That visual realism guarantees a usable grasp |
| The original robot actions fail in the reconstructed scene | Scene calibration, interaction parameters and the success checker | That the video-to-action converter is necessarily responsible |
Document whether human-hand and robot-video conversion share the same assumptions. Differences in grasp geometry, reach and observed motion can change the conversion difficulty. Do not pool those input families without reporting their separate outcomes.
A stronger generated video can still produce a worse executable sequence if the translator handles its viewpoint or motion differently. Preserve this disagreement as a diagnostic result rather than discarding the unsuccessful conversion.
State the environment in which execution was measured
Record the simulated scene's object geometry, initial state, dynamics configuration and success rule. Scene reconstruction supports a grounded comparison but does not make every contact or deformation match hardware.
Use the site's sim-to-real guide for that separate evidence boundary. Before a physical claim, the translated behavior still needs a robot-specific interface and physical evaluation; the simulation score alone does not provide them.
No video model, translator or simulator was run here. Read executability results as evidence for the recorded video–translator–embodiment–scene chain, then inspect which link explains the observed failure. Paper checked October 13, 2026.