A robot policy can fail after learning from human video even when the video looks clear and its reconstruction looks plausible. To locate the cause, keep the task, robot, retargeter and controller fixed while changing one upstream input at a time: clean reconstruction, deliberately corrupted reconstruction, then the candidate reconstructed from video. Compare both the intermediate handoff and the resulting policy behavior. A single end-to-end score cannot tell you which stage broke.
NVIDIA’s Video to Data challenge makes reconstruction and robotic grounding separate evaluation problems. That provides a useful way to organize a local test, provided its input conditions and reference targets remain explicit.
What has to survive each handoff
The first handoff turns third-person RGB video into four-dimensional human-object interaction: body and object geometry, object pose, motion and contact over time. A visually neat mesh can still put an object a few centimeters off, move contact a few frames late, or lose it behind a hand. A downstream learner sees those errors as training instructions.
The next handoff maps the reconstruction into a robot embodiment and policy. The challenge supplies clean multi-view input, synthetic corruption and off-the-shelf reconstruction as different grounding tiers. A local investigation can borrow that separation without claiming a challenge score: build a known-input baseline, perturb one field, then substitute the candidate reconstruction.
That separation matters when choosing what to improve. If object pose is wrong before retargeting, increasing policy capacity may merely help memorize an incorrect target. If the reconstruction is stable but a finger mapping saturates at a joint limit, a better camera model will not repair the command. A pipeline needs records at the boundary between stages, not only an input video and a final success flag.
For a separate discussion of what human-video indexing contributes before physical grounding, see the site’s Figure Index analysis. For the fields a training episode should preserve after grounding, see the robot action-data guide. The present test concerns the error propagation between those jobs.
A controlled substitution exposes the first bad boundary
Start with one task and a held-out set of clips. Freeze the scene, calibration, target robot, retargeter, policy-training recipe and controller. Save three input conditions under the same artifact schema: a clean reconstruction, a synthetic corruption of it, and the reconstruction produced by the candidate model. The synthetic condition should perturb a named field, such as object pose, contact time or visibility, at measured levels. A vague “noisy video” label cannot identify what the policy had to tolerate.
Then compare the conditions in order. Check whether the candidate reconstruction differs from the clean input in the field the task actually uses. Run the same retargeter and note whether any joint limit, collision or contact constraint changes the intended motion. Only after those checks compare policy outputs or task measures under an identical evaluation setup. This is a proposed engineering audit, not a challenge run or reported result.
| Observation in the paired runs | First place to inspect | Why a final score alone misleads |
|---|---|---|
| Object path jumps behind an occluding hand | Pose continuity and object identity in reconstruction | A policy may appear to “choose badly” because its training target teleported. |
| Hand-object contact shifts in time | Contact labels and synchronization at the handoff | Accurate geometry can coexist with a wrong grasp phase. |
| Clean and candidate inputs retarget to the same clipped joints | Robot reach, joint limits and retargeting losses | Improving pose accuracy may leave the limiting command unchanged. |
| Grounded trajectories match, but learned actions diverge | Policy training and evaluation seed controls | The error may be downstream of reconstruction. |
Record the original clip identifier, reconstruction version, coordinate frames, timestamps, object trajectory, contact sequence, retargeted command and policy version together. Keep a few failures with their raw artifacts, not only aggregate metrics. This lets another reviewer trace the first point of divergence without guessing from rendered video.
Pin the evaluator before comparing tiers
The current challenge page contains a material ambiguity. Its robotic-grounding description says submissions in all three tiers are evaluated against the clean multi-view object trajectory. Its leaderboard section says each tier supplies a different reference trajectory and explicitly says scores across tiers are not comparable. Those statements should not be silently reconciled by a reader.
Before interpreting a score as “this reconstruction caused the gain,” record the evaluator version, the reference trajectory used for that run and the exact tier input. Until the organizers’ pinned evaluator or clarified protocol resolves the wording, compare methods within the same tier and reference. A cross-tier score difference could reflect a changed target as well as a changed reconstruction. The named object-tracking measures also need their definitions and direction of improvement copied from that evaluator before they become an acceptance rule.
A repository example is not the challenge protocol
NVIDIA’s Video to Data repository is useful because its stages write inspectable artifacts to disk. Its current main README also describes an end-to-end reference example using Sharpa embodiments, while the challenge’s robotic-grounding track specifies G1 and Dex3. Treat these as different configurations. Running an unchanged repository example would not establish a Track 2 result; embodiment, task data, evaluator and input tier would still need to match the challenge protocol.
No result in this article comes from downloading the dataset, running a simulator or testing hardware. The decision rule is simpler: if the first discrepancy appears in pose or contact, investigate reconstruction; if it appears during mapping, investigate retargeting; if grounded trajectories remain equivalent, inspect policy learning and evaluation. Use an end-to-end score only after those boundaries and the reference trajectory are explicit. Keep the first divergent artifact with the score so the next experiment has a specific target. Sources checked September 29, 2026.