Robot action data is a time-aligned record of what a robot observed, what state it was in, what task it was given, which action was commanded and what happened next. A video can show the scene, but it usually cannot reconstruct the control decision because joint state, calibration, timestamps, gripper commands and intervention are missing.
Each episode should preserve the causal sequence needed to train or audit a policy. Camera frames and proprioception describe the current state; language or a task identifier defines the goal; actions change the robot; and success, failure or recovery labels describe the outcome. If those signals use different clocks or undocumented coordinate frames, the dataset can be large and still be unusable.
Quality depends on coverage and meaning, not only hours or episode count. A useful collection includes representative variation, failed attempts, operator interventions, schema versions and enough metadata to reproduce preprocessing. Cross-robot datasets add scale but also make action normalization and embodiment descriptions essential.
One training episode contains a synchronized loop
An episode starts from an initial state and records observations and actions until success, failure, timeout or interruption. The boundaries matter because a training example must connect the instruction and starting condition to the resulting sequence. Cutting an episode incorrectly can pair an action with the wrong goal or hide a recovery.
The card lists five data families: observations, robot state, task context, actions and outcomes. A dataset can add audio, force, tactile sensing, planner state or safety events, but these core families provide a practical minimum for understanding behavior.
Observations and robot state describe different things
External observations describe the environment through RGB, depth, point clouds, force, audio or other sensors. Robot state describes the body itself: joint positions, velocities, torque estimates, gripper state, mobile-base pose and controller mode. Policies may need both to distinguish a visible goal from what the body can currently reach.
Calibration connects the two. Camera intrinsics, camera-to-base transforms, sensor offsets and coordinate-frame names should be versioned with the episode. Without them, a pixel or 3D point cannot be interpreted consistently after hardware or software changes.
Action format defines the policy's target
Robot actions can be recorded as joint position, velocity or torque commands; end-effector poses; gripper commands; mobile-base velocities; or discrete tokens. They can be absolute targets, deltas from the current state or multi-step chunks. These formats are not interchangeable without transformation rules.
The distinction also matters for VLA models. A model that predicts a Cartesian waypoint needs a controller and inverse kinematics downstream. A model that predicts robot-specific joints carries more embodiment detail but may transfer less easily to a different machine.
| Action field | Example | Required metadata |
|---|---|---|
| Joint target | Arm joint positions | Joint order, units and limits |
| End-effector target | 6D pose or delta pose | Coordinate frame and convention |
| Gripper command | Position, force or open/close | Command meaning and hardware range |
| Action chunk | Several future commands | Rate, horizon and interruption rule |
Time alignment is part of the label
A camera frame captured before an action cannot be paired casually with a command delivered later. Sensor exposure, encoding, transport, inference and controller queues introduce delay. If timestamps are inaccurate, the model learns a shifted relationship and may respond late on hardware.
Use a shared clock or a documented synchronization method, preserve original timestamps and record sampling rates. When data is resampled, store the interpolation and alignment rules. Measure latency distributions so training can reproduce realistic delay rather than assuming every modality arrives together.
Teleoperation provides intent and executable motion
Teleoperation lets a person demonstrate a task through the robot’s own action interface. It captures motions the hardware can execute and can include corrections when objects move or grasps slip. The operator interface, camera view and control mapping influence the resulting data and should be recorded.
Demonstrations are not automatically optimal. Operators may move slowly, use shortcuts or adapt to interface lag. Quality review should identify idle segments, collisions, accidental commands and hidden assistance. Corrective demonstrations and intervention data can be more valuable than another easy success.
Cross-robot data needs an explicit common schema
Robots differ in joint count, link geometry, grippers, sensors and control rates. A field called action may represent degrees on one platform, radians on another and a tool-frame delta on a third. Combining them without a semantic schema creates silent label errors.
A cross-embodiment dataset should retain the original action, a normalized representation when used, the transformation code and enough robot metadata to reverse the mapping. Missing dimensions, unavailable sensors and different success definitions should be explicit rather than filled with unexplained zeros.
Open X-Embodiment shows both scale and heterogeneity
The Open X-Embodiment project brings together datasets from many robots, institutions and tasks. Its visual diversity shows why robot data can support broader learning, but the accompanying repository license and disclaimer also remind users to inspect dataset-specific terms and representations.
Diversity is useful when the model can distinguish shared task structure from embodiment-specific commands. Keep source dataset identity, robot description and preprocessing lineage so a surprising result can be traced to the contributing episodes rather than treated as an opaque mixture.
Dataset quality is measured by recoverable evidence
Useful data can be checked, replayed and traced. Every episode should have a stable identifier, schema version, robot and calibration metadata, task definition and quality status. Train, validation and test splits should prevent near-duplicate trajectories or scenes from leaking across evaluation.
Coverage should be reported across task conditions rather than only total size. Count objects, initial poses, environments, operators, failures and interventions. The table turns common quality claims into evidence that a reader or model developer can verify.
| Quality claim | Evidence to store | Failure if omitted |
|---|---|---|
| Synchronized | Original timestamps and alignment method | Action paired with the wrong observation |
| Diverse | Counts by task, object, scene and robot | Many episodes repeat the same easy case |
| Reproducible | Schema, calibration and transform versions | Preprocessing cannot be reconstructed |
| Valid evaluation | Group-aware split and duplicate audit | Test performance is inflated by leakage |
Training and evaluation should preserve failures
Robot datasets often emphasize successful demonstrations because imitation learning needs examples of desired behavior. A robot foundation model also benefits from knowing when actions fail, when a person intervenes and how recovery returns to a valid state. Removing all failures hides the decision boundary.
Failure data should be labeled by observable cause without inventing certainty. A dropped object may involve perception, grasp, control or unexpected contact. Preserve the complete trace and human annotation separately so later analysis can revise the cause while retaining the event.
A collection checklist before the first episode
Write the schema before collecting at scale. Define clocks, coordinate frames, units, action semantics, episode boundaries, success rules, intervention events and privacy handling. Run a small capture, reconstruct the task offline and verify that a second person can interpret every field without undocumented knowledge.
Then connect collection to the deployment question. Simulation and real-robot data can complement each other, but they must retain their domain and generation metadata. The objective is not a large folder; it is a trustworthy record that can train, evaluate and debug a physical policy.
- Define episode start, success, failure, timeout and intervention.
- Version sensor calibration, coordinate frames and action semantics.
- Synchronize observations, state and commands on a documented clock.
- Retain failures, recoveries and operator identity or protocol metadata.
- Audit duplicates and leakage before creating evaluation splits.
Frequently asked questions
Is robot action data the same as robot video?
No. Video is one observation modality. Action data also needs robot state, commands, timing, task context, calibration and outcomes aligned to the episode.
Why are failed robot episodes useful?
They reveal where the policy, controller or task boundary breaks and can teach recovery or safe abstention. The trace should be preserved even when the exact cause is uncertain.
Can action data from different robots be combined?
Yes, but only with explicit embodiment metadata, units, frames, action semantics and transformation rules. A shared field name does not make commands equivalent.
How much robot data is enough?
There is no universal duration or episode count. Coverage of tasks, objects, conditions, failures and evaluation independence matters more than one headline size.
Where can I find public robot action datasets?
Open X-Embodiment and Hugging Face LeRobot are useful starting points. Check each dataset’s schema, robot, license and task coverage before use.
Dataset Scale Does Not Guarantee Capability
Dataset size, robot count or hours of video do not by themselves establish quality, generalization or deployment readiness. Inspect schema, coverage, lineage and evaluation independence.