Robot failure mining finds rollout segments that deserve review, relabeling or new data collection. The target is not one final failure frame; it is a contextual window that preserves precursors, policy decisions, contact events, consequence and any recovery attempt.
Active learning then allocates limited labeling or hardware-collection budget across uncertainty, novelty, impact, diversity and safety. A high score creates a candidate, not a verified failure or permission to reproduce a dangerous state.
Use this guide with the robot evaluation guide and human intervention guide. Keep a holdout failure set outside the mining and retraining loop.
Mine episode windows instead of terminal frames
The final collision or dropped object may be visually obvious but too late to teach prevention. Capture a window long enough to include state drift, uncertain perception, action selection and the first recoverable deviation.
Retain commands, observations, contact, mode changes and timestamps. Use task semantics to set the horizon; a fixed number of frames means different physical time across control rates.
Build a candidate pool beyond confirmed failures
Include human interventions, protective stops, timeouts, low progress, high uncertainty, novel scenes and suspicious successes. Some successful completions contain unsafe near misses or shortcuts that need correction.
Preserve ordinary successful hard negatives so reviewers and detectors learn the boundary rather than only rare-looking visuals.
| Signal | Useful for | Main ambiguity | Calibration |
|---|---|---|---|
| Uncertainty | Model doubt | Can be miscalibrated | Held-out outcomes |
| Novelty | Distribution change | Novel may be safe | Cluster review |
| Progress | Task stagnation | Slow can be valid | Task-specific timing |
| Impact | Consequence severity | May be hard to predict | Risk taxonomy |
| Intervention | Human concern | Operator threshold varies | Reason and outcome |
Calibrate signals before combining scores
Uncertainty, novelty, predicted impact and collection cost have different units and error rates. Normalize them against labeled validation events and retain separate components before creating a combined priority score.
A weighted sum can hide a severe-impact candidate behind low novelty. Use hard safety rules where consequence dominates ranking.
Find the onset and growth of failure
In repetitive automation, failure often grows through small pose error, contact mismatch, retries or timing drift. Label onset, first detectable cue, first corrective opportunity and terminal outcome separately.
This structure supports early detection and recovery learning. It also prevents every frame after an unavoidable failure from becoming a redundant training sample.
Cluster by mechanism rather than visual similarity
Embedding and trajectory clustering can reduce duplicate review, but two visually similar frames may represent different causes such as slip, calibration bias or delayed command. Add action, force, progress and error-code features where relevant.
Review representative, boundary and outlier episodes from each cluster. Preserve small severe clusters instead of merging them into a common cosmetic category.
Use runtime detectors as candidate generators
The 2025 RSS FAIL-Detect paper frames failure detection from successful training data as sequential out-of-distribution detection and evaluates learned and post-hoc signals with conformal thresholds.
Its alerts remain model outputs under reported tasks. Human review and outcome evidence are still needed before labeling root cause or changing a deployed safety response.
Design human review for information gain
Show reviewers synchronized windows, predicted scores, task phase and prior cluster examples. Ask structured questions about failure, cause, recoverability, severity and useful next collection, with an uncertain option.
Measure inter-reviewer agreement and route disagreements to adjudication. Do not force complex physical events into an overconfident single label.
Keep hard negatives near the decision boundary
Hard negatives include compliant contact, slow progress, occlusion or unusual lighting that resembles a failure but completes safely. Without them, monitors trigger constantly and policies avoid valid rare states.
Pair each failure cluster with relevant normal variants where possible. Evaluate false alarms by task phase and operational cost.
Collect new data inside a protected envelope
Active learning should not blindly execute the most uncertain action. Filter candidates through collision, force, workspace, human-access and reset constraints, then use simulation, fixtures or teleoperation when direct autonomous reproduction is unsafe.
Specify stop criteria and independent safeguards. The learning objective never authorizes expansion of the physical safety envelope.
Allocate budget across diversity and expected value
Choose episodes to label or reproduce by cluster coverage, consequence, expected error reduction, cost and feasibility. A large common cluster may need fewer representatives than a small severe mechanism.
Track actual label, collection and reset cost. Use the mixture guide so newly mined data does not overwhelm normal behavior during retraining.
Retest on unseen failures and false alarms
After retraining, evaluate a frozen holdout containing unseen episodes, new instances and hard negatives. Do not score only the candidate pool used for selection because the loop explicitly optimizes toward it.
Measure detection delay, recovery opportunity, task success, unsafe events and false-alarm cost. Compare against a fixed baseline and preserve every intervention.
| Loop stage | Artifact | Acceptance check | Leakage risk |
|---|---|---|---|
| Mine | Candidate windows | Context and timestamps complete | Terminal frame only |
| Cluster | Mechanism groups | Rare modes retained | Visual shortcuts |
| Review | Labels and decisions | Agreement and uncertainty | Forced certainty |
| Collect | New episodes | Safety and diversity | Unsafe uncertainty chasing |
| Retest | Frozen holdout | Unseen failure improvement | Candidate reuse |
Release with a failure-mining lineage
Link candidate score, source rollout, reviewer decision, cluster, recollection request, new dataset version, model update and holdout result. Preserve rejected candidates so later audits can understand the sampling policy.
Close validation with the following checklist.
- Mine contextual windows with precursor and consequence.
- Calibrate uncertainty, novelty, impact and cost separately.
- Cluster mechanisms and retain hard negatives.
- Collect only inside independent safety controls.
- Retest on frozen unseen failures and normal boundary cases.
Frequently asked questions
Should every failed episode be added to training?
No. Deduplicate, verify labels, preserve mechanisms and balance against normal and recovery data.
Does high uncertainty mean the robot failed?
No. It is a candidate signal that needs calibration, context and outcome evidence.
Can mining operate on individual frames?
Frames can seed search, but root cause and recovery usually require a synchronized episode window.
Does active learning intentionally run dangerous states?
It should not. Use protected filters, simulation, fixtures or human control when reproduction exceeds the safe envelope.
How is retraining value proven?
Use a frozen holdout of unseen failures and hard negatives with delay, success, safety and false-alarm metrics.
Candidate-Alarm and Verified-Failure Boundary
Failure scores prioritize review; they do not create truth or hardware authorization. Preserve context, uncertainty, safety constraints and a separate unseen holdout.