Before merging LeRobot RGB-depth datasets, check the video-stream fields and the physical meaning of the depth codes separately. Matching file geometry and codec can make concatenation possible while leaving different depth quantization settings or a wrong decode unit unresolved. A successful merge is not an encode/decode accuracy test.
The LeRobot video-encoding documentation, checked October 13, 2026, describes a depth pipeline that clips values to a selected range, quantizes them to 12-bit codes and stores those codes in a high-bit-depth video stream. Lossless compression of those codes does not recover information already lost through clipping or quantization. This is a documentation review of main; no recordings were encoded or merged.
Check the stream contract, then the depth contract
The documented merge concatenates videos without re-encoding. Codec, pixel format, height, width and FPS must match. Some encoder-tuning fields are combined more loosely and can become null when the inputs disagree. Those rules concern stream compatibility; they do not demonstrate that the depths decoded from both sources have the intended physical meaning.
Use the following as a proposed pre-merge worksheet, then inspect representative decoded frames from each source:
| Contract | Evidence to inspect | Question to resolve |
|---|---|---|
| Video stream | Codec, pixel format, dimensions and FPS | Can the actual streams be concatenated? |
| Depth feature | Depth-map flag and quantizer bounds, shift and log/linear setting | Do the stored codes retain the correct mapping to distance? |
| Loaded observation | Requested depth output unit and policy preprocessing | Does the policy receive the physical scale it expects? |
| Working range | Source values near both clipping bounds | Was task-relevant geometry lost before compression? |
The documentation does not establish that every semantic mismatch is automatically rejected by the merge tool. Treat the worksheet as an additional check, not an existing validator's guaranteed coverage.
Separate stored mapping from loaded units
LeRobot interprets integer depth inputs as millimetres and floating inputs as metres. The quantizer bounds and shift are specified in metres regardless of that input convention. Its separate read-time depth_output_unit selects millimetres or metres without re-encoding the stored stream.
Consider an illustrative combination of datasets from two collection sessions. Both may load successfully, yet one consumer can interpret the resulting values with the wrong scale. Trace input units, stored quantizer metadata, requested output units and downstream normalization before blaming the policy. Changing the read-time unit cannot restore depth detail clipped during collection.
First-episode metadata cannot certify every later recording
The documented metadata is populated from the first encoded episode and assumes consistent encoder settings. Changing settings midway through recording is unsupported. If provenance indicates such a change, inspect the affected episodes rather than treating the single metadata entry as proof of uniform encoding.
The LeRobot v3 format guide explains shard and episode relationships. For this merge decision, retain the per-source depth contract and decoded checks as well. If that contract cannot be reconciled, prepare a separately validated compatible revision before training on the combined data.