SAFE turns a sequence of VLA internal features into a failure score, then compares that score with a time-varying upper band calibrated from successful rollouts. The band is designed to limit the chance of raising any alert during a successful episode under the calibration assumptions. The calibrated quantity is the false-alert frequency among successful episodes. Applying the band to a new task also requires examining how that task changes the feature distribution.
The SAFE project describes this as multitask failure detection. Paper version 2, particularly Sections 4.2–4.3, 5.4 and Appendix C.2, explains the score, calibration and distribution-shift limitation. The sources were checked October 15, 2026. No detector, calibration procedure or robot response was executed for this article.
The score is learned before the alert band is calibrated
SAFE extracts hidden-state features from the final layer of a VLA, before those features are decoded into action-token logits or a velocity field. It trains a small MLP or LSTM detector using both successful and failed rollouts. The training labels identify whether each whole episode succeeded; the method does not require annotations marking the exact instant at which failure began.
The two detector variants give their scores different numerical meanings. SAFE-MLP applies an MLP and sigmoid to each timestep’s feature vector, then accumulates those outputs. Its score can grow beyond one as the episode progresses. SAFE-LSTM processes the feature history recurrently and produces a sigmoid-normalized score between zero and one.
This is why “a score of 0.8” is not a portable alarm rule across the variants. The MLP score is not a probability. The LSTM output is a learned normalized failure score, but the conformal calibration described next is a separate operation; it does not turn every individual score into a verified posterior probability of failure.
Episode-level training labels also limit the interpretation of timing. The first alert is a detector decision, not an annotation establishing the physical onset of failure. A warning may precede an obvious dropped object, occur after recoverability has been lost, or never appear. Those are different outcomes even if the final episode label is the same.
Successful trajectories define the time-dependent boundary
The detector sees both outcome classes during training, but the functional conformal prediction step uses successful rollouts from a held-out seen-task calibration set. The object being calibrated is an entire score trajectory, rather than an isolated frame.
The paper writes the one-sided band as an upper limit upper(t) = μ(t) + h(t), with no finite lower limit. Here, μ(t) is a time-dependent mean and h(t) is the bandwidth. The alarm threshold is that upper limit: a score crossing it raises a failure flag.
A moving band lets a score be interpreted against the expected stage of a successful rollout. As an invented example, a score of 0.6 would cross a band whose upper limit is 0.4 at an early timestep, but would stay below a later upper limit of 0.8.
Under the stated exchangeability assumptions, the conformal result concerns a new successful rollout staying inside the band at all times. Equivalently, the probability that a successful rollout produces at least one false alert is bounded by the chosen significance level α. This is an episode-level, any-time event. It is not a separate allowance of α false alerts at every frame.
That direction of the statement matters. Limiting alerts among successful episodes does not establish that an alerted episode has a 1 − α probability of failure. The latter question also depends on how often failures occur and how effectively the detector catches them. Nor does this false-positive bound specify a deadline by which a failed episode must be detected.
Unseen tasks can move the successful-score distribution
SAFE is intended to generalize across tasks, but its conformal bands are calibrated on seen tasks and then evaluated on unseen tasks. A successful execution on a new task may produce a different feature and score trajectory from the successful calibration examples. In that case, the statistical assumption supporting the nominal band coverage can fail.
Appendix C.2 reports this limitation directly. Some benchmark configurations have true-negative rates near the nominal 1 − α line; others fall below it. The baseline methods also fail to follow that nominal line consistently across all benchmarks. The authors attribute these departures to calibration and evaluation scores potentially coming from different distributions.
Thus, a chosen α is a calibration input, not a measured deployment false-alert rate. For an illustrative α = 0.1, the nominal claim concerns at most a 10% chance of any alert on an exchangeable successful rollout. It does not establish a 10% rate for an unseen task population merely because the same band is reused there.
A policy change can matter for the same reason: SAFE reads internal VLA features, so a detector and band are attached to a particular feature-generating system. Replacing that checkpoint or observation pipeline changes the system that supplies the detector’s inputs, so calibration must be evaluated for that combination.
ROC-AUC and detection time answer different questions
The paper’s episode ROC-AUC uses the maximum score reached over the entire rollout. It measures how the resulting episode scores separate successes and failures across threshold choices. It does not select the time-varying band used during execution, and it does not say when the first useful warning arrived.
The conformal-band evaluation makes an actual alert decision. It reports true-positive and false-positive rates, balanced accuracy and detection time. Its detection-time measure is the relative timestep of the first band crossing; if a failed rollout never crosses, the value is set to one. The average is taken over all ground-truth failed rollouts, so missed failures remain in this reported timing measure.
That convention is worth preserving when comparing results. An alternative average calculated only over detected failures would answer a different question and could appear faster by excluding missed cases. Even the paper’s normalized timing metric is not a measured warning lead time before a specific physical hazard or loss of recoverability.
The VLA evaluation guide covers whole-system outcomes and intervention accounting. For SAFE, the narrower decision is to read the score model, successful-rollout calibration band and observed timing together. When choosing a band, compare the measured false-alert rate and first-alert timing on the intended task alongside the episode ranking.