Robot Policy Uncertainty and Abstention

Robot policy uncertainty and abstention address what happens when a policy should not continue autonomously. The system estimates a defined risk, crosses a calibrated threshold early enough to act and enters a safe hold, help request, teleoperation or recovery state.

A softmax score, action variance or model disagreement is not automatically a failure probability. Each signal responds differently to ambiguous demonstrations, unfamiliar observations, stochastic futures and model error, so calibration must match the event and time horizon that operations care about.

Pair this guide with the Physical AI safety-layers guide and human-intervention data guide. Report risk and autonomous coverage together.

Define the failure and prediction horizon

Start with an observable event such as collision, dropped object, workspace exit, unrecoverable pose, instruction mismatch or operator takeover. Define whether the score predicts the current state, the next action or failure within a future horizon.

A threshold cannot be calibrated when labels mix different events and lead times. Preserve onset, consequence, recoverability and censoring for episodes that end before the horizon.

Separate sources of uncertainty

Aleatoric ambiguity reflects multiple plausible actions or noisy observations. Epistemic uncertainty reflects insufficient model knowledge. Out-of-distribution detection asks whether the input differs from reference data, while failure prediction estimates an operational event.

One score may correlate with several categories without identifying them. Label the intended meaning and test counterexamples such as familiar hard states, novel but safe backgrounds and multimodal demonstrations.

Signal	Possible meaning	Strength	Primary failure
Max probability	Model concentration	Cheap	Overconfidence
Entropy	Predictive spread	Simple	Multimodality confused
Ensemble disagreement	Model uncertainty	Useful comparison	Compute and shared bias
Distance or OOD score	Novel representation	Detects shift	Novel can be safe
Failure predictor	Declared future event	Operational target	Label and horizon dependence

Do not equate softmax with failure probability

Classification confidence is shaped by training objective, class balance and temperature. A policy can assign high probability to an action because alternatives received lower logits even when all available actions are unsafe or outside training support.

Measure reliability against the declared event using held-out deployment-like episodes. Keep action confidence, OOD score and future-failure probability as separate fields when they drive different fallbacks.

Calibrate on independent deployment-like data

Fit thresholds or calibration mappings on validation data collected under representative robots, sites, operators and action policies. Calibration can break when the policy, camera, controller or intervention protocol changes.

Report reliability diagrams, expected calibration error with bin choices, Brier or log loss where appropriate, and event rates. Recheck high-risk subgroups rather than trusting one global number.

Choose thresholds from risk and coverage

Lower thresholds can reduce autonomous risk by sending more cases to a fallback, but they also reduce coverage and increase human workload. A system that attempts almost nothing can appear safe while providing little autonomous value.

Plot risk versus coverage and include confidence intervals, failure severity and fallback capacity. Select risk-specific thresholds rather than one score for every task and consequence.

Require enough warning lead time

A correct alarm arriving after contact cannot prevent the event. Measure time or control steps between threshold crossing and failure, then subtract communication, operator-response, braking and recovery latency.

Evaluate detection at multiple horizons and report late alarms separately from misses. Thresholds that improve recall by firing only at the terminal state are not useful for prevention.

Map abstention to executable robot states

Abstention must select a physically valid action: hold position, controlled stop, retreat, place an object, request clarification or transfer authority. The safe choice depends on payload, contact, balance, slope and nearby people.

Specify entry conditions, acknowledgments, timeout, control ownership and fallback failure behavior. A user-interface message without a verified motion state is not a complete abstention mechanism.

Design human handoff and safe re-entry

The help request must give the operator enough context to understand goal, observation, uncertainty reason and current control authority. Preserve pre-alert history and ensure commands cannot be issued simultaneously by human and autonomous controllers.

Define when autonomy may resume: after a human label, a completed teleoperation segment, return to a recognized state or explicit approval. Prevent rapid oscillation between autonomy and help with hysteresis or a state machine.

Use research evidence within its evaluated setting

The KnowNo project studies language-model uncertainty and robot help-seeking in its reported tasks. The related paper provides its experimental scope, while other selective-prediction work tests different models, labels and environments.

Treat reported improvements as evidence for those setups, not a universal threshold. Recreate calibration with the target policy, robot, cost of help and failure definition.

Mine hard negatives and shifted failures

Uncertainty datasets need safe unusual states as hard negatives and familiar-looking failures as positives. Otherwise a detector learns background novelty or operator identity instead of precursors to a bad outcome.

Connect false alarms and misses to the robot failure-mining guide. Recalibrate after policy updates because the states visited and failure boundary can change.

Evaluate assisted and autonomous outcomes separately

Measure failure at accepted coverage, selective risk, false-alarm workload, miss rate, warning lead time, help latency, handoff success, fallback success, re-entry outcome and total task completion. An assisted completion is not an autonomous success.

Use the SUSIE study and uncertainty-aware imitation-learning work as scoped research references, then validate the full operational loop on independent trials.

Outcome layer	Measure	Warning	Interpretation
Calibration	Predicted versus observed risk	Overconfidence	Refit or reject
Coverage	Autonomous fraction	Nearly zero	Low utility
Timing	Warning lead time	Alarm too late	Cannot prevent
Handoff	Resolved requests	Operator overload	Fallback insufficient
Task	Autonomous and assisted success	Metrics merged	Claim inflated

Release a calibrated abstention contract

Version policy, uncertainty model, calibration set, event definition, horizon, thresholds, fallback state machine, operator interface and re-entry logic. Store every alert with score components, state age, action, human response and final outcome.

Close release review with the following checks.

• Define the predicted failure and horizon.
• Calibrate on independent deployment-like data.
• Report risk and autonomous coverage together.
• Verify fallback, handoff and re-entry states.
• Recalibrate whenever policy or operating distribution changes.

Frequently asked questions

Is policy confidence a probability of success?

Not automatically. It must be calibrated against a defined operational outcome and horizon.

Should a robot stop whenever uncertainty is high?

No. It needs a risk-specific, physically safe fallback such as hold, help, teleoperation or controlled stop.

What is a risk-coverage curve?

It shows observed risk among accepted autonomous cases as the system abstains on progressively more cases.

How is an assisted success reported?

Report it separately from autonomous success, together with help latency, operator workload and fallback outcome.

When should thresholds be recalibrated?

After policy, sensor, robot, controller, task mix or intervention-protocol changes and whenever drift is detected.

Calibrated-Risk and Handoff Boundary

Abstention is a calibrated control-transfer policy, not a confidence label. Its value depends on risk, coverage, lead time, fallback success and safe return to autonomy.