Robot evaluation leakage occurs when training or model selection gains information from the intended test distribution. The common failure is not a visible answer label in the input; it is shared collection context across episodes, sessions, objects, operators, sites or derived media.
The correct split unit follows the deployment claim. A model intended for new frames in the same session needs a different test from one intended for new objects, operators, sites or embodiments. Random frame splitting rarely supports the broader claims.
Use this guide with the robot evaluation guide and dataset lineage guide. Freeze test membership and access after the protocol is approved.
Write the generalization claim before the ratio
State whether the model must handle new time, episode, object instance, category, operator, site, robot or task. The holdout groups follow that statement; a universal 80/10/10 ratio does not create independence.
Create several tests when deployment asks several questions. One convenient test set cannot isolate every axis of generalization.
.jpg){kind=link}
Split at the lowest independent collection unit
Frames within an episode share background, object state, operator and temporal neighbors. Episodes within a session may share lighting, calibration, tool wear and scripted setup. Keep dependent groups together.
Use immutable group IDs and enforce disjointness in the pipeline. Avoid reconstructing groups later from filenames that may have been renamed.
| Claim | Minimum holdout group | Leakage example | Reported test |
|---|---|---|---|
| New moment | Temporal block | Adjacent frames | Future segment |
| New episode | Episode | Same rollout windows | Episode holdout |
| New object | Physical instance | Different views of one object | Instance holdout |
| New operator | Operator or collection team | Teleoperation habits | Operator holdout |
| New site | Facility and setup | Background and fixture | Site holdout |
Treat session context as learnable data
A model can exploit fixed camera pose, lighting, table marks, fixture wear, initialization routine and software timing. Collecting on a different day in the same unchanged setup may still share most shortcuts.
Audit metadata and images for session identity signals. Hold out entire site or procedure where the claim requires new environments.
Keep physical object instances together
Different images, orientations and trials of the same object are not independent evidence for new-instance generalization. Scratches, texture and geometry can identify the instance.
For pose or grasp evaluation, link crops and annotations to immutable object IDs. The 6D object-pose guide explains why instance and category objectives differ.
Hold out operator and instruction style
Operators create characteristic trajectories, recovery habits, camera framing and language instructions. A random episode split can place the same person in train and test, inflating apparent human-generalization performance.
Use operator-grouped tests and report experience level. Preserve privacy with controlled identifiers rather than deleting the grouping information needed for audit.
Trace augmentations and derived siblings
A crop, resize, re-encode or color augmentation has a different file hash but remains a sibling of the same source frame. Keep source lineage and split before generating derivatives.
Use perceptual similarity to find missing relationships, then confirm with manifests. Near-duplicate detectors complement rather than replace provenance.
Fit preprocessing on training data only
Normalization statistics, vocabulary, tokenizer adaptation, class weights, feature selection and imputation can leak test information when computed globally. Treat each learned preprocessing artifact as part of training.
Version the training membership used to produce it. Apply frozen transforms to validation and test without refitting.
Control validation and test consultation
Repeatedly choosing models and hyperparameters from validation performance adapts development to that set. Looking at test results and then revising the system converts the test into another validation source.
Maintain a final frozen test, limit access and log evaluation count. Use fresh prospective or external data for major decisions after repeated iteration.
Audit overlap with metadata and content
Check exact IDs, parent episode, session, object, operator, site, robot and task intersections. Then use visual, temporal and trajectory similarity to catch mislabeled or copied siblings.
The DROID dataset project demonstrates large in-the-wild collection across sites and operators. Dataset scale and diversity do not remove the need for claim-specific grouped splits.
Report group counts and uncertainty
Frame count can be huge while independent objects or sites are few. Report samples at the split unit, successes and failures by group, confidence intervals and any excluded trials.
A high average with one site or operator does not establish broad generalization. Show condition-level results and lower-tail behavior.
Freeze and version the test manifest
Store immutable membership, grouping rules, preprocessing identities, evaluation code and access history. New incoming data should not automatically rebalance or replace historical test samples.
When the deployment claim changes, create a new version and preserve comparability to the old one. Connect it to the data-quality audit for missing metadata and label review.
| Leakage audit | Method | Pass condition | Residual risk |
|---|---|---|---|
| Identity | Exact manifest join | No cross-split source IDs | Missing IDs |
| Grouping | Episode, session, object, operator, site | No forbidden group overlap | Wrong grouping rule |
| Derivatives | Parent lineage | Siblings remain together | Lost parent |
| Similarity | Visual and trajectory search | Reviewed near duplicates | False matches |
| Access | Evaluation log | Protocol-limited consultation | Unlogged copies |
Release with a split acceptance checklist
Document deployment claim, group hierarchy, membership manifest, derivative lineage, train-only preprocessing, overlap audit, sample counts, uncertainty, access and refresh policy. Keep the test unavailable to routine training jobs.
Close validation with the following checklist.
- Choose holdout groups from the deployment claim.
- Keep dependent episodes and sessions in one partition.
- Trace objects, operators, sites and derived siblings.
- Fit every learned preprocessing artifact on train only.
- Freeze membership, access and evaluation count.
Frequently asked questions
What train-validation-test ratio should robot data use?
Choose group independence and enough evaluation power first; no fixed ratio solves leakage.
Is a different collection day a new test environment?
Not necessarily if site, objects, calibration, operator and procedure remain shared.
Do unique file hashes prove there are no duplicates?
No. Crops, re-encodes and augmentations have new hashes while sharing the same source.
Why is tuning from test performance leakage?
The development process learns which choices perform well on the test, so it no longer estimates untouched generalization.
Can one test set measure every kind of generalization?
No. Use separate grouped tests for new episodes, objects, operators, sites, robots or tasks as required.
Deployment-Independent Test and Frozen-Access Boundary
Evaluation independence is defined by the deployment claim, not a random percentage. Freeze grouped membership, derived lineage and test access before model selection begins.