A robot data factory is the operational system that collects demonstrations, autonomous rollouts, interventions and failures, converts them into traceable episodes, and feeds verified evidence back into model training and deployment decisions. Its product is not raw video; it is usable, governed learning data.
The difficult work sits at the interfaces. Cameras and actions must share time, robot embodiments need explicit schemas, task outcomes need labels, and every transformation must be versioned. Otherwise a larger dataset can quietly combine incompatible units, frames, policies and success definitions.
This guide complements the synthetic robot data guide and Physical AI PoC framework. Collection should follow current privacy, consent, security and licensing requirements for the deployment context.
Design the episode before collecting more data
Define one episode from task request through termination, including reset and outcome. Record observations, robot state, actions, timing, task text, environment, operator input and success evidence. A dataset cannot repair an undefined task boundary.
Choose units, coordinate frames, image conventions and action semantics in a schema registry. Mark required and optional fields. Schema changes should produce a new version and a migration record rather than silent reinterpretation.
Clock alignment determines whether behavior is learnable
Camera frames, proprioception, commands, force signals and controller states arrive at different rates and delays. If timestamps are wrong, training pairs an observation with the wrong action and evaluates policies against misleading ground truth.
Use a common clock or measured offsets, preserve source timestamps and document resampling. Inject known motions to validate alignment. Monitor dropped frames, queue delay and jitter during real operation rather than assuming a nominal frequency.
Embodiment metadata keeps robots from becoming false equivalents
The Open X-Embodiment project represents many robot datasets in a common episode format while retaining dataset and action-space details. Its public materials show why a shared container does not erase hardware differences.
Record robot model, joints, limits, gripper, sensors, calibration, controller, firmware and action interpretation. A seven-dimensional action vector can mean absolute pose, delta pose or velocity; these are not interchangeable without a defined adapter.
| Episode field | Why it matters | Failure if missing | Quality check |
|---|---|---|---|
| Timestamps | Align cause and action | Wrong supervision | Known-motion test |
| Embodiment | Interpret state and actions | Unsafe transfer | Schema and adapter review |
| Task and outcome | Define intent and success | Ambiguous labels | Independent verification |
| Calibration | Map sensors to geometry | Systematic spatial error | Reference target test |
| Policy provenance | Trace behavior source | Biased evaluation | Version and hash |
Demonstrations contain operator choices and hidden corrections
Teleoperation and kinesthetic teaching capture successful behavior quickly, but operators differ in style, speed and recovery. A demonstration can include pauses, camera searches or corrections that a learner interprets as intended behavior.
Record operator identity or protocol, interface latency and whether assistance occurred. Keep failed and recovered episodes. Compare multiple experts and ask whether the collected trajectories cover the states the autonomous policy will actually visit.
Curation should preserve useful difficulty
Remove corrupt files, broken timing and impossible labels, but do not curate away every hard case. Near-failures, recoveries and unusual objects often contain the most valuable information for deployment robustness.
Use automated checks for range, continuity, duplication and missing fields, followed by sampled human review. Assign rejection reasons. Curation decisions should be reproducible against a pinned raw-data snapshot.
Labels need definitions and measured agreement
Success, grasp quality, contact, intervention and failure cause require written definitions. For subjective labels, use multiple annotators and measure agreement. Escalate ambiguous cases instead of forcing certainty.
Model-assisted labeling can accelerate work but introduces systematic errors. Sample by task, site and predicted confidence, then audit against human or instrumented evidence. Preserve the labeler and tool version.
Dataset versions must protect evaluation splits
Create immutable dataset releases with source episode IDs, schema, transforms, filters and licenses. Content hashes make it possible to reproduce training and investigate regressions. Keep deletion and access-control procedures for governed data.
Split by episode families, objects, environments, time or site so near-duplicate sequences do not leak into evaluation. A random frame split can make a model appear to generalize while seeing almost the same trajectory during training.
| Factory metric | What it reveals | Misleading shortcut | Operational use |
|---|---|---|---|
| Valid episode yield | Collection reliability | Raw hours only | Fix capture pipeline |
| Label agreement | Definition quality | Label count | Refine taxonomy |
| Held-out success | Generalization | Training loss | Release decision |
| Failure coverage | Deployment relevance | Success-only data | Collection priorities |
| Time to trace episode | Governance maturity | Storage volume | Incident response |
Training should retain a path back to every episode
Record dataset version, sampling weights, preprocessing code, model configuration, random seeds and checkpoints. If a policy fails, engineers should trace which examples and transformations shaped the behavior.
Balance tasks without erasing their real operating distribution. Oversampling rare failures can improve learning but distort probability. Evaluate both balanced diagnostic sets and representative deployment sets.
Deployment feedback closes the data loop
Log task requests, policy versions, confidence, interventions, failures and verified outcomes from live robots. Prioritize collection where failure severity, frequency or business impact is high rather than where data is easiest to gather.
Use shadow evaluation and canary deployment for new policies. Compare regressions by task and condition, and retain rollback. A data flywheel is healthy only when each cycle improves held-out and operational evidence.
Governance is part of the production architecture
Define data ownership, consent, retention, access, encryption, export and deletion. Workplace video can capture people, labels and proprietary processes. Open dataset licenses may differ across code, images and contributed sources.
Audit permissions before training and publication, and isolate restricted data. Incident procedures should identify affected episodes and derived models. A scalable factory makes compliance and traceability routine rather than a manual emergency.
- Define episode and schema before capture.
- Measure timestamp and calibration quality.
- Preserve embodiment and policy provenance.
- Version data, labels, transforms and splits.
- Feed verified field failures into bounded collection plans.
Frequently asked questions
What is a robot data factory?
It is the collection, validation, labeling, versioning, training and feedback system that turns robot operation into governed learning episodes.
Is more robot data always better?
No. Incorrect timing, ambiguous actions, leakage and unrepresentative collection can make a larger dataset worse than a smaller traceable one.
What metadata is essential for cross-robot data?
Record embodiment, sensors, calibration, action semantics, units, frames, controller and task outcomes so adapters can interpret each episode correctly.
Should failed robot episodes be kept?
Yes when they are valid and governed. Failures and recoveries reveal states that successful demonstrations may never visit.
How do you prevent evaluation leakage?
Split by episode, object, environment, site or time as appropriate, deduplicate related sequences and keep immutable held-out sets.
Data Governance Note
Robot data may contain personal, proprietary or safety-sensitive information. Apply current consent, licensing, access, retention and security requirements, and preserve a traceable path from model outputs back to governed source episodes.