Use hard resets when reproducing the documented LeRobot LIBERO benchmark protocol. If you opt into soft resets to save time, report that choice with the score and test its effect separately. Matching a seed and initial-state identifier does not establish bit-identical observations after the environment settles.
The LeRobot LIBERO guide, checked October 13, 2026, preserves hard reset by default. With fixed initial states enabled, hard_reset=false avoids rebuilding the simulator model and renderer on each episode. The guide explicitly warns that soft-reset states are not bit-identical after settling and that camera observations and policy results can differ. This article reviews those documented conditions; no simulator or policy was run.
A speed change can become an input change
Consider an illustrative evaluation in which a team switches reset mode and then observes a different success rate for the same checkpoint. Faster episode turnover does not explain whether the policy improved. Reset mode changed an upstream part of the experiment before the policy received its first observation.
Preserve the selected initial-state identity, state after settling, first camera observations, checkpoint and action preprocessing for each trial. These are proposed diagnostic records, not a claim that every field is automatically exported by LeRobot. Compare the actual observations rather than inferring equivalence from the seed alone.
If the first difference appears before the first policy action, investigate initialization and settling before changing inference settings. If starts agree but actions diverge, the remaining explanation may involve policy stochasticity, preprocessing or later simulator behavior. Keep that distinction visible instead of attributing every result change to reset mechanics.
Pair trials before comparing completion rates
For a reset-mode investigation, run each checkpoint under both modes on the same named tasks and intended initial-state set. Hold the observation and action contracts fixed. The guide recommends the same seed, fixed initial states, and a single task batch containing its evaluation episodes when comparing policies on the same episodes.
As a proposed report, retain three outputs separately:
- Initialization differences before the first policy action.
- Per-episode completion and failure location under each mode.
- Reset duration and total evaluation duration.
A reduced wall-clock cost can be useful even when the resulting protocol is different. It should not be presented as a reproduced hard-reset score until the comparison supports that claim. Avoid selecting whichever reset mode makes each checkpoint look best.
Keep suite choice and reset choice distinct
The LIBERO suite-selection article explains which knowledge-transfer question a suite tests. Reset mode answers a separate reproducibility question: what initial observations and dynamics the chosen task actually supplies. Retain both in the run configuration, along with the LeRobot and environment revisions. A suite name alone leaves this experimental change invisible.