A robomimic HDF5 file can provide recorded camera frames without providing everything needed to recreate the motion in a simulator. Choose the playback method first. Viewing stored images, restoring recorded simulator states and rerunning recorded actions answer different questions and require different data.
The official dataset structure and playback tutorial explain the available fields and modes. The source behavior below was checked against robomimic revision d309eae on October 15, 2026. No dataset or simulator was executed for this article.
Decide what the playback should establish
If you want to inspect what a camera recorded, start with stored observations. If you want to inspect a trajectory from another rendered view, you need simulator states and a compatible environment. If you want to check whether the recorded actions produce the recorded motion, you need action replay.
| Playback route | What it does | What must be available |
|---|---|---|
Stored observations, --use-obs |
Writes saved image observations to video | Requested image keys and the episode actions array used to determine length |
| State playback, the simulator default | Restores successive recorded states and renders them | Valid simulator states and a reconstructable environment |
Action playback, --use-actions |
Restores the initial state, then steps recorded actions | Recorded states, equally many actions, matching action interpretation and compatible simulation |
These routes can produce similar-looking videos while establishing different things. A stored image does not show whether today’s simulator can load the recorded state. A state-by-state reconstruction does not show that today’s dynamics would reach those states from the recorded actions.
The pinned playback script does not create the environment on its stored-observation path. That path requires video output and rejects combining --use-obs with --use-actions. It also uses the actions array to determine the episode length, so an image-only file is not enough for this script. Supply the actual observation key: a simulator camera name and an HDF5 image key are not interchangeable merely because they describe the same viewpoint.
For a real-robot dataset with usable recorded images but no reconstructable simulator, image playback may be the appropriate inspection route. It should be reported as inspection of saved observations, not simulated reproduction of the robot’s motion.
Preserve the environment recipe, not just the arrays
The HDF5 data group’s env_args attribute stores environment metadata as JSON: env_name, env_type and env_kwargs. For robosuite demonstrations, the per-episode model_file attribute holds the MJCF model XML. The recorded states and actions retain their time order.
Those fields describe what should be reconstructed; they do not package the installed simulator, environment wrapper or every dependency. A receiving machine still needs compatible software and assets. Preserve the versions used for collection and processing alongside the file, then investigate any missing environment or model asset before treating a replay failure as a bad demonstration.
There is also a format boundary: the dataset overview permits empty or dummy states for non-robosuite datasets. A field called states is therefore not sufficient evidence of a usable simulator checkpoint. Check the dataset’s environment type and the actual state representation before choosing a restoration route.
The pinned script reads the first stored state for simulator playback and adds the episode model for robosuite. In its action path, subsequent motion comes from stepping the recorded actions, but the script still uses the full states array for trajectory length and divergence checks and requires equally many actions. An initial-state-only file is not sufficient for this implementation. In the state path, recorded states are restored one by one. Changing a controller, action convention or environment version can therefore matter even when the HDF5 structure still loads.
For example, suppose saved images show a successful placement while action replay misses the target. That is an illustrative diagnostic situation, not a result reported here. First compare the reconstructed initial scene and action interpretation with collection. The two videos alone do not identify whether the cause was a version mismatch, altered dynamics, incorrect input or another difference.
Keep the episode list with the replay result
A filter key under mask names a subset of demonstration IDs. The same file can therefore supply different training, validation or inspection episodes. Record the key and its resolved member list, not just the filename and a statement that “the dataset was replayed.”
The dataset tutorial’s get_dataset_info.py --verbose route lists the exact demonstrations under each filter key. That provides a way to check which members you intend to inspect before playback. Verify that each listed ID exists and that your intended train and validation memberships do not overlap; the presence of two differently named masks does not itself prove independence.
Be careful when using a small playback sample as a regression set. In the pinned script, supplying --n shuffles the selected demonstration list and then takes that many entries. A request for five demonstrations is consequently not an instruction to always inspect the same first five IDs. Retain the actual selected IDs, or prepare an explicit filter group for the fixed set you want to compare.
This is a software-behavior finding from the source, not a replay trial. It matters because two attractive sample videos may have inspected different episodes, making a claimed before-and-after comparison difficult to interpret.
Report reconstruction and task outcome separately
A useful handoff to another machine includes the dataset hash, environment and software versions, filter key and episode IDs, playback mode, camera or observation keys, and any required model information. Add what was actually observed: saved images decoded, states restored, actions replayed, or a particular task criterion checked. Leave unperformed checks unclaimed.
The action path in the inspected script compares intermediate resulting states with the next recorded states and can print a divergence warning. Such a warning is evidence to investigate; it is not a diagnosis of the cause or a task-success score. The wider distinction between state agreement and task completion is illustrated in the Isaac Lab demonstration-replay guide, whose format and implementation are different from robomimic’s.
The portable result is more specific than “the file plays.” It names which episodes were inspected, which data drove the display or simulation, and which reconstruction conditions were available. That makes the next replay a comparison someone else can reproduce rather than another video with an unknown basis.