An Isaac Lab demonstration is not validated just because its file opens and its actions can be stepped. Keep five questions separate: did the episode decode, was the intended initial scene restored, do the stored actions retain their meaning under the current task configuration, did the selected state comparison run and agree, and was the task goal achieved? A pass at one layer does not stand in for the next.
This article examines the pinned Isaac Lab v2.3.2 workflow. It explains source behavior at that version; it does not claim that a current installation, a different recorder, or a later release behaves identically.
Keep five kinds of replay evidence separate
Start by naming the evidence you have. “The demo replayed” is too vague for debugging because it can describe anything from successful HDF5 decoding to a task-specific success signal at the end of an episode.
| Evidence layer | What a positive result establishes | What it still leaves open |
|---|---|---|
| File decoding | Expected episode fields can be reconstructed from the selected file | Whether the current task interprets those fields as the recorder did |
| Initial-scene restoration | Named scene fields were applied from the saved initial state | Whether that state is a complete physics checkpoint or guarantees deterministic evolution |
| Action interpretation | The stepped field and its layout, frame, scale and timing match the recording contract | Whether the resulting states agree or the task succeeds |
| State agreement | Selected stored and runtime state components meet the chosen comparison | Whether the task goal was achieved |
| Task success | The current task’s success term is true at the point where it is checked | Whether every intermediate state matched or the behavior transfers to a policy or robot |
The distinction matters most when the layers disagree. A correctly decoded file can drive the wrong action interface. A replay can end in a goal region even after a large intermediate deviation. Conversely, a strict state comparison can reject a replay that still satisfies the task’s goal condition.
Loading the episode does not validate its meaning
The pinned HDF5 dataset handler recursively reconstructs episode groups and can attach attributes such as the environment name, seed, and success label when present. That is useful decoding evidence. It is not a schema-level guarantee that the chosen task, asset set, controller configuration, or action dimensions match the recording.
Record the following before drawing conclusions from a replay:
- The task identifier and environment configuration used now.
- The stored episode fields that are actually present, including
initial_state, actions, and optional later states. - Whether the actions are raw commands, processed commands, or another field family.
- Which validation modes were enabled for this run.
- Which task success term was evaluated and at what point.
The stored seed deserves its own note. The examined loader can recover it, but the pinned replay call does not pass that per-episode seed into reset_to. Seeing a seed in the file therefore does not prove that this replay restored it as part of the episode reset.
Task selection also needs an explicit check. The pinned script can read an environment name from metadata for configuration parsing, yet its environment constructor uses the command-line task argument. The official examples provide that argument. Without executing the exact setup, do not promise that metadata alone is a working task fallback.
For broader checks on coverage, field consistency, and dataset provenance, use the robot-learning data quality audit. If the uncertainty is the meaning of the action field itself, the robot action data guide is the closer prerequisite.
The saved start is applied before the actions
In the inspected replay loop, the episode’s initial_state is sent to reset_to with relative positioning enabled, then stored actions are passed to env.step. Later stored states are not forced onto the environment at every step; they are consumed only by the optional comparison path.
This ordering rules out a common but misleading diagnosis: the normal path does not simply accept an arbitrary newly randomized start and then replay the commands. A deliberately altered start or changed task geometry is a counterfactual test. Label it as an input change rather than evidence of nondeterministic reset behavior.
Restoration is also narrower than “load a complete simulator checkpoint.” The examined reset_to implementation runs reset machinery before applying the supplied scene state, forwards the resulting kinematics, and recomputes observations. The scene state covers named poses, velocities, and articulation joint state. That is substantial, but it is not every hidden solver, contact, randomization, or physics-history variable.
The action sequence then enters the action manager configured by the current task. A stored vector has no self-contained promise that its slices, scales, frames, controller type, or timing still mean what they meant during recording. First prove the task and action configuration match; only then interpret a divergence as a replay-state problem.
State agreement is optional and narrower than success
The pinned comparison is optional and limited to one replay environment. It visits selected articulation and rigid-object state leaves, checks their lengths, and compares components with a threshold of 0.01. The number has field-specific units. It must not be presented as a universal 10 mm Cartesian tolerance, a joint-angle rule, or a hardware accuracy specification.
There is also a source-level shape caveat. In the examined normal recorder path, one stored vector sample returned by EpisodeData.get_state carries a leading singleton shape [1, D], while the runtime value selected for the one replay environment is [D]. The relevant recorder manager path supports that shape reading. For an ordinary vector width greater than one, the pinned compare_states length guard selects its error path before the numeric threshold, if this branch reaches a leaf following that examined shape path.
That is a conditional source finding, not a reproduced error. The branch may be disabled, skipped, lack stored states, encounter a different recorder shape, or never reach such a leaf. It would be inaccurate to say that validation always fails, that every run fails, or that the source already supplies a working validate_states recipe for every dataset.
Even a successful component check answers only state agreement under that comparator. It does not evaluate the task’s goal. The examined recording configuration exports successful episodes only, so it should not be assumed to preserve every failed attempt. It labels success after a configured consecutive-success window, as shown in the pinned recording loop. Replay-end validation instead keeps the current task’s success term and checks it when an episode has no next action. The stored success label, consecutive-window recording decision, and replay-end success term are related but separate evidence. The pinned guide also warns that cropping a recording at the exact success trigger can prevent that trigger from occurring again during replay. That is a documented caution, not a failure reproduced here.
A finite geometry example isolates the logical difference
We constructed and measured five authored, static three-state cases in Blender 5.2.2 LTS. Each case places a finite 160 × 120 × 80 mm tool box at three post-state centers and compares it with declared reference centers. The final box is also tested against a 176 × 136 × 96 mm goal box. The component rule is an illustrative XYZ-only 10 mm check chosen for this article; the endpoint rule requires every final tool corner to lie inside the goal half-spaces.
| Authored case | Declared change | Max XYZ component (mm) | Max XYZ norm (mm) | Component check | Minimum goal margin (mm) | Full-box goal |
|---|---|---|---|---|---|---|
| Baseline | No offset | 0.00 | 0.00 | Pass | +8.00 | Inside |
| 7 mm each | +7 mm XYZ at all three post states | 7.00 | 12.12 | Pass | +1.00 | Inside |
| Middle 16 mm | +16 mm X at the middle post state | 16.00 | 16.00 | Fail | +8.00 | Inside |
| End 9 mm | +9 mm XYZ at the final post state | 9.00 | 15.59 | Pass | -1.00 | Protrudes |
| Combined | Middle 16 mm X and final 9 mm XYZ | 16.00 | 16.00 | Fail | -1.00 | Protrudes |
The 7 mm XYZ case passes the component rule even though its largest Euclidean displacement is about 12.12 mm. A 16 mm intermediate X difference fails that rule while the final box remains inside the goal. In the final 9 mm XYZ case, every retained component stays below 10 mm, yet some final-tool corners protrude about 1 mm. The combined case fails both predicates for different reasons. Values are rounded to two decimal places; the full finite measurements were checked independently.
The reference post-state centers are (0, 0, 0.04), (0.4, 0, 0.04), and (0.8, 0, 0.04) metres. Every case has the same separately declared initial center at (−0.4, 0, 0.04) metres; the compared post-state offsets are authored counterfactuals, not the outcome of replaying an action sequence. All boxes share a 23° Z rotation and a (0.15, −0.08, 0.12) metre translation. The independent audit recovers the comparison frame from the measured world geometry and derives the goal planes from its finite faces. Component differences are evaluated in the declared local XYZ frame, not in rotated world axes. An 8 μm numerical comparison allowance applies only to Blender floating-point readback against the declared inputs and independent geometric quantities. It was not added to either acceptance predicate. It is not a task tolerance, simulator error estimate, robot precision claim, or safety margin.
These cases are explanatory geometry. They do not decode HDF5, execute Isaac Lab, apply actions, simulate contacts, interpolate motion, check collision along a path, or test hardware. Their purpose is to make one diagnostic point visible: close state components and task-goal containment are different predicates.
What to record before accepting a replay
Use a replay report that keeps the layers separate:
- File: path, episode ID, available fields, shapes, stored task name, seed, and success label.
- Configuration: exact task, environment count, action terms, controllers, frames, scales, and replay options.
- Start: which initial scene fields were applied and whether positions are relative to environment origins.
- Actions: which stored field was stepped and why its semantics match the current task.
- State check: whether it ran, which fields and units it covered, its threshold, and any shape exclusion or error.
- Success: which task term was checked, whether a consecutive window was involved, and when the value was sampled.
- Limits: no claim of policy generalization, deterministic physics, complete collision safety, or physical transfer.
The pinned imitation-learning guide warns that reset-based replay may not reproduce physics deterministically. Treat that as a reason to preserve configuration and validation evidence, not as proof that a particular divergence came from nondeterminism.
A useful replay conclusion is therefore specific: “the episode decoded, its declared initial scene was restored, actions were interpreted by this task configuration, these state fields met this comparison, and this task success term was true at replay end.” Anything shorter hides which claim was actually tested.