Robot learning data quality is the consistency of observation, action, state, task context and outcome. A sharp video is not enough. If the action belongs to another timestamp, frame or controller interpretation, the episode teaches a false physical relationship even though every individual field looks valid.
A useful audit combines automatic integrity checks, visual playback, statistical summaries and held-out policy evidence. It preserves failures and human interventions when their meaning is known, while rejecting corruption, leakage and undocumented transformations. Quantity should be reported after these gates, not before them.
Use this guide with the robot action-data guide and teleoperation data guide. The Open X-Embodiment repository demonstrates why unified storage still requires per-dataset semantics and citations.
Define quality as preserved physical meaning
Begin with the causal unit: an observation and robot state lead to an action, which changes the world and produces an outcome. Quality checks should test whether this relationship is correctly encoded. File readability and image resolution are necessary but much weaker conditions.
Write the intended learning problem, prediction horizon and action application rule. A dataset collected for Cartesian delta actions at 10 Hz cannot be treated as absolute joint targets at 30 Hz. Every conversion needs an explicit, testable semantic contract.
Freeze an episode schema and field dictionary
Specify required sensors, task instruction, robot state, action, timestamps, calibration references, controller mode, termination cause and intervention state. Define types, units, coordinate frames, valid ranges and missing-value behavior. Optional fields need the same definition when present.
Version the schema rather than silently adding or repurposing fields. A parser should reject unknown critical versions. Store source robot and software identifiers so a future curator can distinguish a meaningful distribution shift from corruption.
| Field group | Required meaning | Automatic check | Human check |
|---|---|---|---|
| Observation | Sensor and capture time | Shape and monotonic time | Visible alignment |
| Action | Command semantics and horizon | Range and dimension | Playback direction |
| State | Measured robot configuration | Limits and continuity | Pose plausibility |
| Outcome | Success or failure cause | Allowed label set | Task result |
| Context | Calibration and versions | Reference exists | Correct hardware |
Measure synchronization and action delay
Compare sensor capture, robot-state sampling, operator input, command issue and command application on one clock. Estimate fixed offset, jitter, dropped samples and buffering. Matching array indices is not proof that events occurred together.
Replay sequences with overlays of gripper pose, action vector and contact events. Sudden changes should appear in the expected order. Quantify the residual timing uncertainty and decide whether it is acceptable relative to robot speed and the policy horizon.
Make action frames, units and conventions explicit
Actions may be joint positions, velocities, torques, Cartesian poses, deltas, twists or gripper states. State whether values are absolute or relative, which frame is used, how orientation is represented, what interval a command covers and where limits are applied.
Normalize only after retaining the original units and transform. Test known motions: positive x, one-joint rotation and gripper close should produce predictable signs. A dataset merger that assumes matching seven-dimensional vectors can combine incompatible physical commands.
Track calibration and hardware configuration
Camera intrinsics, hand-eye transforms, joint zero, tool center point, gripper geometry and payload can change during collection. Store calibration IDs and validity intervals with episodes. A successful recalibration should not rewrite old data without recorded provenance.
Detect sessions collected after impact, remounting or tool change. Compare projected geometry and task landmarks across sessions. If a transform is uncertain, quarantine the affected range instead of distributing a clean-looking but geometrically inconsistent sample.
| Change | Data at risk | Evidence to retain | Audit response |
|---|---|---|---|
| Camera remount | Images and robot pose | Extrinsic before and after | Split validity range |
| Tool change | Actions and contacts | TCP and payload ID | Separate configuration |
| Joint service | State and kinematics | Zero calibration | Revalidate motion |
| Controller update | Applied action | Firmware and mode | Version behavior |
| Clock source change | All alignment | Offset and drift test | Re-estimate timing |
Label failures by stage, not only final success
A binary success label hides whether perception failed, the grasp slipped, planning timed out, collision recovery activated or placement was wrong. Use a compact taxonomy with a primary failure stage, termination reason and whether the episode remained physically safe.
Keep difficult and failed episodes when their labels and signals are reliable. Removing every failure teaches only ideal state distributions and hides recovery behavior. Remove or quarantine episodes whose meaning is unknown, not episodes merely because the task failed.
Treat human intervention as supervision
Teleoperation takeover, corrective nudge, emergency stop and manual reset reveal states where autonomy lost competence. Record who or what initiated the intervention, its start and end, the preceding policy context and the action source after takeover.
Do not blend autonomous and human actions without a source flag. A learner may otherwise imitate an abrupt command that only made sense as recovery. Intervention rate and time-to-intervention are also deployment metrics that can guide the next collection cycle.
Control mixture weights and domain coverage
Dataset size should be broken down by robot, task, object, scene, operator, success state and hardware version. Millions of near-identical frames from one slow episode do not equal broad experience. Report episodes, duration and effective diversity together.
Set mixture weights from the deployment objective and inspect which domains dominate batches. Preserve rare safety-relevant and recovery cases even when they reduce headline success rate. Weighting decisions should be versioned with the training run.
Detect duplicates and evaluation leakage
Near-duplicate frames can cross random train and test splits when a continuous session is divided at sample level. Split by scene, object instance, trajectory, time block or collection campaign according to the generalization claim. Hashes catch exact copies; embeddings and metadata help find near duplicates.
Keep validation and test decisions outside iterative curation where possible. If the test set influences filtering, hyperparameters or prompt design, it is no longer an untouched estimate. Create a new held-out slice and document the change.
Combine automated gates with visual review
Automated checks should cover schema, ranges, monotonic timestamps, sample gaps, calibration references, action continuity, image decode, duplicate rate and label validity. Stratify failures by session so a systemic capture bug is not mistaken for isolated noise.
Human reviewers should inspect synchronized multi-view playback, action overlays, contact and termination. Use a sampling plan that includes random episodes and every automated outlier class. Record review decisions and reviewer agreement rather than making invisible deletions.
Version lineage from raw episode to training batch
Retain immutable raw data and create reproducible transforms for resampling, calibration application, filtering, relabeling, chunking and normalization. Each derived shard should identify input versions, code revision, parameters and output checksums.
The Open X-Embodiment paper and OpenVLA paper are useful primary technical references, but their results do not validate a new private mixture. Close the loop with the robot data-factory guide and held-out hardware tests.
- Freeze schema, units and coordinate frames.
- Measure capture-to-action timing and jitter.
- Retain calibration, hardware and controller versions.
- Preserve labeled failures and interventions.
- Prevent leakage and version every transformation.
Frequently asked questions
Is clear video enough to call robot data high quality?
No. The observation must align with robot state, action, timing, calibration and outcome. Visual quality alone cannot preserve physical meaning.
Should failed episodes be deleted?
Not when their signals and failure labels are trustworthy. Failures and recoveries can be valuable supervision; corrupted or ambiguous episodes should be quarantined.
Can data from different robots be merged directly?
Only after embodiment, action, state, units, frames, timing and task semantics are mapped explicitly. A shared file format does not make the meanings identical.
How much timestamp error is acceptable?
It depends on motion speed, control rate, sensor exposure and policy horizon. Estimate the resulting spatial or action error and set a task-specific limit.
Can quality review be fully automated?
Automated gates scale well, but synchronized playback and task judgment still catch semantic errors that ranges and hashes miss. Use both with recorded decisions.
Robot Dataset Evidence Boundary
Robot dataset quality is specific to the learning objective, embodiment, action semantics and deployment domain. Dataset scale or a unified format does not substitute for lineage and held-out hardware validation.