Teleoperation data is recorded while a person controls a robot through a leader arm, joystick, motion capture, headset, keyboard or another interface. The useful artifact is not the video alone. It is a synchronized episode that connects what the robot observed and knew to the action the operator commanded and the result that followed.
The interface shapes the data. A leader arm may provide natural joint or end-effector trajectories but inherit its own geometry. A VR controller may express relative poses and require retargeting. Keyboard control produces discrete commands. Latency, scaling, filtering and safety limits alter the relationship between human intent and robot motion.
Teleoperation can be an operating mode and a data-production method. The distinction described in teleoperation versus autonomy must be preserved in the dataset. Demonstration episodes can train a policy, but autonomous capability is established only through separate policy rollouts and documented interventions.
A demonstration is a synchronized multimodal record
A complete episode aligns camera frames, depth or force data, joint position and velocity, gripper state, task instruction and operator action. The robot action data guide explains these fields in detail. Calibration identifiers and software versions make the record reproducible.
The action label should specify its meaning: joint target, velocity, end-effector pose, delta command or another control space. Without units, frames and timing, the same numbers cannot be safely replayed or combined across robots. Store success, failure and intervention labels alongside the trajectory.
Clock alignment determines which action caused which image
Cameras, robot controllers and operator devices can run at different rates and clocks. If timestamps drift, an image may be paired with an action that occurred earlier or later. The learned policy then receives a blurred causal relationship, which becomes especially harmful during fast contact or recovery.
Use a defined master clock, hardware timestamps where available and logged synchronization diagnostics. Record dropped frames, queue delay and command application time. Resampling should preserve the original timestamps so a later pipeline can distinguish interpolation from measured data.
| Stream | Required metadata | Common failure | Check |
|---|---|---|---|
| Camera | Timestamp, exposure, calibration | Dropped or delayed frame | Frame interval and sync |
| Robot state | Units, joints, frame and rate | Stale observation | Age at action time |
| Operator action | Device and mapping | Hidden scaling or filtering | Raw plus applied command |
| Outcome | Success rule and intervention | Unlabeled partial failure | Episode review |
The teleoperator changes the action distribution
Leader-follower arms encourage continuous trajectories similar to the robot’s configuration. Motion capture or VR can support whole-body or end-effector commands but needs retargeting and workspace mapping. A joystick favors lower-dimensional velocity control. Each interface makes some behaviors easier and others less represented.
Document the operator, device, control mode, scaling, clutching and assistance. Mixing data across interfaces can broaden coverage, but it can also create inconsistent labels. Normalize only when the physical meaning is preserved, and retain source identity for later analysis.
Good demonstrations are consistent without being identical
Operators should complete the task efficiently and avoid unnecessary oscillation, yet the dataset needs variation in object position, approach and ordinary disturbance. Repeating one exact path can produce a policy that memorizes the setup. Random uncontrolled variation can make a small dataset incoherent.
Define a collection protocol with allowed strategies, task success and reset rules. Train operators, review early episodes and adjust the protocol before scaling. Record operator identity because skill and style can affect action smoothness and coverage.
Success-only data misses the states where policies fail
Behavior cloning learns from demonstrated states. During deployment, small errors can push the robot into states absent from successful demonstrations, and the error compounds. Recovery demonstrations and human interventions show how to return from weak grasps, misalignment and partial progress.
LeRobot’s human-in-the-loop documentation describes alternating autonomous segments with human takeover and correction, recording recovery without resetting the episode. Preserve control labels so training and evaluation can distinguish policy and human actions.
Curation protects training from silent data faults
Before training, check missing streams, timestamp gaps, calibration mismatches, impossible joint values, saturation and corrupted video. Review task labels and success criteria. Duplicate or near-duplicate episodes can overweight one setup, while canceled trials may contain valuable failure context if labeled correctly.
Curation should be reproducible. Version the raw data, transformation code, exclusion rules and output dataset. Do not overwrite intervention or failure labels to make the data look clean. Those segments often explain deployment errors that average training loss cannot reveal.
Training converts demonstrations into a policy, not proof
The LeRobot real-robot guide separates teleoperation, dataset recording, policy training and evaluation. That separation is conceptually important. A policy checkpoint reflects an optimization on the demonstration distribution; it does not establish autonomous success on new trials.
A vision-language-action model may combine task text with observations and actions. The dataset still needs consistent instruction semantics and embodiment metadata. Language cannot repair missing synchronization or ambiguous control labels.
The learning flow should close through evaluation
After training, run independent episodes with held-out positions, objects or environments. Measure success, intervention, recovery, time and safety events. Save failures with the same synchronized schema, then decide whether to collect new demonstrations, interventions or targeted environment variation.
The card shows a closed loop rather than a one-time recording campaign. Demonstrate, synchronize, curate, train and evaluate. Each evaluation failure should produce a data hypothesis, not an instruction to collect more random hours.
Dataset scale must be tied to coverage
There is no universal number of demonstrations. A narrow deterministic task may learn from dozens of consistent episodes; a diverse bimanual task can require far more. Count task-relevant variation, operators, objects, start states and recovery cases rather than quoting hours alone.
Learning curves help. Train on increasing subsets and evaluate the same held-out suite. If performance plateaus, more similar data may not help. Diagnose missing states, label noise, model capacity, action representation and hardware variability before expanding collection.
| Coverage axis | Record | Evaluation split |
|---|---|---|
| Object and pose | Identity, geometry and start state | Held-out combinations |
| Operator | Identity and interface | Unseen operator style where relevant |
| Environment | Camera, lighting and layout | Held-out conditions |
| Failure and recovery | Error, takeover and correction | Recovery-focused trials |
| Robot configuration | Calibration and software version | Cross-session stability |
A practical collection gate prevents expensive rework
Pilot a small dataset and verify replay, timestamps, calibration, labels and visualization before collecting at scale. Confirm that a person can perform the task from the recorded observations alone. Validate privacy and retention when cameras capture workers or homes.
Define who owns raw demonstrations, derived datasets and policy updates. Record consent and access controls. Operational data can become a valuable flywheel, but only when technical quality, safety and governance remain connected.
- Define action semantics and clocks.
- Pilot synchronization before scaling.
- Record success, failure and intervention.
- Version curation and calibration.
- Evaluate policies on held-out rollouts.
Frequently asked questions
Are teleoperation and remote control the same?
Teleoperation is a form of remote control, often emphasizing continuous human interaction with robot sensing and motion. The exact interface and autonomy assistance should be stated.
Does teleoperation data make a robot autonomous?
No. It supplies demonstrations for training. Autonomous capability must be measured by running the trained policy without undisclosed continuous human commands.
How many demonstrations are needed?
It depends on task complexity, variation, consistency, model and evaluation target. Use learning curves and coverage rather than one universal count.
Should failed demonstrations be deleted?
Corrupted data should be excluded, but genuine failures and recovery can be valuable when accurately labeled and used with an intentional training method.
Can simulated teleoperation data be mixed with real data?
Yes, if observation and action spaces, timing and embodiment differences are documented. Validate the mixture through held-out real-robot tests.
Dataset and Control Note
Teleoperation tools, dataset formats and policy workflows change. Verify the current documentation, repository version, robot configuration and data rights before collection or model training.