RoboFailRing detects a possible failure by comparing a task’s start/current image pair with stored start/failure-end image pairs. It searches memory entries for the current task, takes the highest similarity among the retrieved neighbors, and applies a threshold. The VLM receives a structured failure report after that decision, so detection and explanation occupy different stages of the pipeline.
The ACL 2026 paper makes this separation concrete in Sections 3.1–3.4. It combines a frozen image encoder and a searchable failure memory with language grounding. The account below follows the paper’s method and figures; no memory index or robot system was run here.
A memory entry preserves the beginning and the failed ending
The authors build their failure corpus from FailGen trajectories whose outcomes are not Success. For each trajectory and camera view, they select the initial frame and the final failure-state frame. Front, overhead and wrist observations become separate memory entries.
That choice matters when a failure is visible from only one direction. The paper’s basketball-in-hoop example describes a backboard blocking the front view and the arm obstructing the overhead view, while the wrist camera exposes the relevant scene. Keeping entries separate allows that informative view to be retrieved without averaging it together with occluded views.
Each frame is encoded with CLIP ViT-B/32 into 512 features. Equations 2–3 normalize the two frame vectors individually, concatenate them into a 1,024-dimensional representation, and normalize the result again. The beginning and ending therefore occupy distinct parts of the vector. A single-frame image of an empty gripper loses the starting context that helps distinguish what the robot was trying to do.
This is a compact two-state representation. The intervening sequence is compressed into the contrast between those states, rather than encoded as a complete video. A matching memory entry supplies a relevant failure example; the later reasoning stage interprets that example alongside its task and failure metadata.
Online retrieval asks about this task, at this moment
During execution, the detector retains the initial image and pairs it with the current image. It first restricts the memory to entries with the current task identifier, then retrieves K nearest neighbors. This task filter reduces interference between visually similar scenes from different manipulation tasks.
The detection score is the maximum cosine similarity among those K neighbors. Under Equation 7, a score greater than or equal to the threshold produces a failure state; otherwise monitoring continues with new observations. The reported workflow sends a termination signal to the controller when the detector fires, before invoking VLM reasoning.
The paper describes the online representation as using the same concatenated normalization scheme as memory construction. Its printed Equation 4 abbreviates the query using encoder outputs directly, whereas Equations 2–3 explicitly show per-frame normalization. The method description supports paired-frame retrieval; a numerical implementation should resolve that notation against the code it actually uses.
For a hypothetical empty-gripper scene, the task filter and initial frame provide two distinct pieces of context. “Put the basketball in the hoop” selects a different task subset from “open the microwave,” while the initial frame describes the particular starting scene. If the relevant failure appearance is absent from that task’s memory, the similarity-based detector has less relevant evidence to retrieve. Raising or lowering the threshold changes the balance between missed failures and unnecessary alerts; it does not add missing examples to the memory.
The report gives the VLM named evidence
RoboFailRing stores a compact symbolic failure label with each memory entry. A deterministic mapping expands that label into natural language at report time. For example, the paper maps a rotation-at-waypoint label to a description of incorrect gripper rotation around a named axis at a named stage.
The report in Equation 8 aggregates the retrieved neighbors’ task identifiers, grounded failure descriptions, camera viewpoints and reference start/end images. The VLM can therefore compare a specific historical deviation with an explicit task context, rather than infer everything from a lone current image and a generic question.
The threshold decision answers whether the current pair resembles a stored failure strongly enough to alert. The report and VLM then address the cause and a possible repair. Keeping those outputs separate also makes it possible to locate an error: a wrong retrieval match arises before reasoning, while a poor explanation can arise despite a useful retrieved example. The broader robot evaluation guide explains how task outcomes and intervention records fit around such intermediate decisions.
“Time saved” measures trajectory progress
The paper’s temporal saving ratio is (1 − t_det / L) × 100%, where t_det is the frame index of the first alarm and L is the failure trajectory’s total frame count. It locates the alert within the recorded sequence.
For an illustrative calculation, an alert at frame 40 of a 100-frame trajectory gives 60%. That means 60% of the recorded trajectory lies after the alarm. Wall-clock retrieval latency, VLM generation time and controller stopping time are separate durations. The ratio can show earlier detection along a rollout while leaving those implementation timings to their own measurements.
RoboFailRing’s central design is thus the connection between two representations: paired images for task-specific retrieval, and grounded descriptions plus reference images for explanation. Reading those stages separately reveals what the memory contributes before the VLM begins to reason.