Changing a ManiSkill demonstration’s control mode can change the number of commands needed to reproduce its motion. The conversion tool therefore treats controller conversion differently from replaying the original actions: in its CPU path, it explicitly rejects conversion combined with forced per-step environment states.
The distinction matters when preparing training data. Restoring an initial state establishes a starting point; restoring every recorded state imposes an entire sequence. A controller conversion generates a new action sequence, whose steps may no longer align one-to-one with the original state sequence.
The trajectory replay documentation describes the options. The current replay implementation shows the precise CPU branch that separates them. This explanation uses those sources checked on October 16, 2026, without executing a replay or conversion.
A new controller changes the meaning of an action
A demonstration is more than an array of numbers. Its episode metadata identifies the original control mode, and the replay tool needs that mode to decide how to interpret the stored actions. The documentation requires the trajectory HDF5 file and its companion JSON file to remain in the same directory.
In replay_cpu_sim, each episode supplies its original control mode and reset arguments. When no target mode is requested, or the requested mode matches the original, the tool steps through the original action array. Changing the observation mode can record a different view of that replay without requiring a controller conversion.
When the control mode differs, the code dispatches to a conversion helper. The inspected CPU function has routes from pd_joint_pos and from pd_joint_delta_pos. Each helper receives the requested target mode, original actions, the original environment and the target environment. An original mode outside those two branches raises NotImplementedError in this function.
That dispatch is narrower than a universal action-format converter. The source mode determines which helper can interpret the demonstration, while the requested destination must be supported by that helper. The documentation identifies Panda robots as the best-supported case and says control-mode conversion is not supported in GPU-parallelized environments.
Why the original state index can stop matching
The implementation explains its incompatible-options assertion in terms of action count: converting controllers can change how many actions are taken to reach the same states. A recorded state index is consequently not a ready-made index into every converted action sequence.
Consider a hypothetical symbolic example. A source episode contains N action steps and the associated recorded states. A different controller produces M command steps to follow the motion. If M differs from N, there is no one-to-one correspondence between converted step k and original recorded step k simply because both use the integer k.
Forcing the old state after every new command would impose an alignment that the conversion has not established. It would also replace the target environment’s evolving state with the recorded sequence. The CPU code prevents this combination before dispatching to a conversion helper.
Its condition permits use_env_states when the target control mode is absent or unchanged. When a different target mode is requested, use_env_states must be false. This distinction is about the sequence being imposed, rather than a blanket ban on loading any saved state during conversion.
First-state restoration and per-step restoration take different routes
The CPU replay function resets the target environment for each attempt and also resets the original environment when one is present. If either state-restoration option is enabled, it loads the stored environment states and applies the first state before replay proceeds.
After that shared starting-point operation, the options diverge:
| Option and requested operation | What the inspected CPU path does |
|---|---|
use_first_env_state with a different control mode |
Restores the initial state, then allows conversion to generate subsequent commands |
use_env_states with the original control mode |
Steps each original action and then replaces the environment state with its corresponding stored state |
use_env_states with a different control mode |
Stops at the incompatible-options assertion before conversion |
The middle row is an important implementation detail. Although the option is described as replaying states, this CPU branch still calls env.step(a) and then calls set_state_dict with the saved state. It is not simply a viewer that ignores the action array.
The initial-state option also has a documented use when moving demonstrations between CPU and GPU simulation. Those backends can reset differently from the same seed, so restoring the recorded start removes that particular mismatch. That cross-backend workflow is separate from adding controller conversion to a GPU-parallelized run.
Saving follows the replay outcome and selected filter
After a CPU attempt, the implementation reads success from the returned info, defaulting to false when the field is absent. With the default allow_failure=False, a successful attempt reaches the branch that flushes a trajectory or video when the relevant save flag is enabled. With allow_failure=True, that branch also accepts an unsuccessful attempt.
The code increments its successful_replays counter inside that same acceptance branch. As a result, enabling allow_failure changes how that counter should be interpreted: accepted failures can contribute to it. For a converted dataset, the environment’s outcome label and the filtering options explain the retained episodes more precisely than the counter name alone.
The current source also has a specific interaction worth recognizing before combining conversion with discard_timeout. In this function, truncated is assigned by env.step in the unchanged-control-mode branch; the conversion branches only assign info. If conversion reports success, discard_timeout=True makes the later expression read truncated. Without an earlier assignment, that read can reference an unassigned local variable. This is a static reading of the checked source, not an observed runtime failure.
The default discard_timeout=False does not evaluate that extra timeout condition. This source-level distinction is why the conversion path and its output filter should be read together, rather than treating every replay option as freely interchangeable.
Choose the route from the data you need
If the task is to regenerate observations along the recorded states, the per-step state route keeps the original state sequence central. If the task is to create actions for a different controller, the conversion route makes the new controller’s command sequence central. First-state restoration can establish the start of that second route without forcing its later step count to match the original.
The ManiSkill visual-simulator throughput guide explains the cost of physics and rendered observations. Here, the prior decision is semantic: which controller interprets each saved action, which state sequence produced the observations, and which replay outcomes were retained. Those choices determine what the resulting demonstrations teach, before throughput becomes the limiting factor.