Compare a LeRobot run by its data-parallel workers, per-worker microbatch and gradient accumulation; the displayed step count alone hides those differences. Equal loop steps can represent different numbers of sample presentations and optimizer updates. A checkpoint resumed on a new topology can also continue with a different sample sequence.
The LeRobot multi-GPU guide describes current main behavior. Each data-parallel worker loads its own microbatch at each loop step; accumulation combines several such steps into an optimizer update. The documented effective batch is microbatch × data-parallel world size × accumulation steps. It covers a source install of main; an older installed release can behave differently.
Equal loop steps can hide half the data exposure
Consider two hypothetical configurations with full microbatches and 100 loop steps:
| Quantity over 100 loop steps | Configuration A | Configuration B |
|---|---|---|
| Microbatch per worker | 32 | 8 |
| Data-parallel workers | 1 | 2 |
| Accumulation steps | 1 | 4 |
| Sample presentations | 3,200 | 1,600 |
| Optimizer updates | 100 | 25 |
These are arithmetic examples. Configuration B has an effective batch of 64 for each update, yet it processes fewer sample presentations over the same 100 loop steps. The counts describe training draws; the same episode or frame can be drawn more than once.
For a comparison intended to isolate distributed execution, decide which quantity should stay fixed: data exposure, update count, effective batch, or elapsed compute budget. They answer different questions. Retain the learning-rate schedule and accumulation boundary as well; identical optimization needs all of these to match.
LeRobot documents that steps counts loop steps rather than optimizer updates, and that any rescaling of the learning rate or step count is left to you when the effective batch grows. Write those scaling choices into the experiment design so readers can see them.
Loading a checkpoint is weaker than sample-exact continuation
The guide allows model and optimizer state saved through PyTorch Distributed Checkpoint (DCP) to be resharded for a changed topology. It separately limits sample-exact sampler resumption to matching the original world size and batch size. A run can therefore resume cleanly on a new topology and still see a different training stream. The page also warns that the Fully Sharded Data Parallel (FSDP) checkpoint layout used by LeRobot 0.6.x and earlier cannot be resumed by the described implementation.
Before comparing a resumed run with an uninterrupted one, retain the original and resumed code revisions, topology, batch, accumulation, dataset revision and checkpoint format. Check actual sample order near the resume boundary if that equivalence matters.
The SmolVLA fine-tuning guide covers a broader training workflow. For this comparison, publish the sample/update arithmetic and the resume conditions beside the learning curve so readers can tell a distributed-run change from a model improvement. LeRobot multi-GPU guide checked October 15, 2026.