REFLECT makes a long robot history easier to inspect by keeping three connected levels of detail: sensor-derived descriptions, selected events, and observations at the end of each subgoal. It first asks whether the subgoals were achieved. A failed subgoal sends the explanation process back into the event history; a task that fails despite completing its planned subgoals sends it toward the plan and final world state.
That distinction changes what evidence the system needs. An empty gripper during placement can result from dropping the object earlier. A pot resting on the wrong active burner can instead expose a mismatch in an otherwise executed plan. The REFLECT project and paper Section 3 use these kinds of examples to connect summary construction with failure explanation. This article follows those sources; no robot logs, language model or correction planner were executed here.
Turn sensor streams into objects, relations and events
The lowest level begins with RGB-D observations, sound and robot state. REFLECT uses detected objects and depth to construct a semantic point cloud. It then builds a scene graph whose nodes describe objects and their states, while edges describe relations such as containment, support and proximity. The robot has its own node, including a relation for an object being inside its gripper.
This representation can express more than a list of visible objects. A pot on a counter, a pot in the gripper and a pot on a burner contain the same object category but imply different next actions. For objects with changeable states, the method compares image embeddings with a predefined set of state labels. Spatial relations come from geometric heuristics, and the gripper relation combines robot state with object detections.
The graph is also informed by the task. REFLECT retains objects named by the original plan and objects spatially related to them. As new frames arrive, it aligns the new point cloud with accumulated observations and uses add, update, replace and delete operations to maintain that representation. The original plan therefore influences which parts of a busy scene enter the summary.
Audio contributes another stream of events. The method separates louder segments from the surrounding audio, embeds each segment, and selects the most similar label from a candidate list. A sound associated with a dropped object can add a cue when the object is outside the current camera view. A burner sound can indicate a state change hidden behind a pot. These labels come from the audio model’s comparison, just as the visual states come from their perception pipeline.
Keep the frames where the represented situation changes
Storing a graph for every frame creates many nearly identical records. The event level selects frames when any of three conditions occurs:
- The task-informed scene graph changes from the previous frame.
- An audio event begins or ends.
- Execution of a subgoal ends.
The selected frame becomes a text record with its time, robot action, visual observations and audio description. The visual part of that event caption uses objects visible in the current frame. The accumulated point cloud serves a broader role in maintaining the environment representation over time.
This selection rule makes the event history different from uniform video sampling. A long motion with little represented change can produce few events, while a brief drop or change in object relation can create a useful entry. The selected detail depends on the perception outputs: an unnoticed relation change will not create a graph-change event, while a fluctuating detected relation can create extra entries.
The top level compresses the record again. It retains observations at the end of each subgoal, such as reaching a toaster or placing bread inside it. These endpoints let the language model compare an intended high-level outcome with the observed result without first reading every low-level action.
A failed subgoal opens the earlier event history
The progressive explanation procedure checks subgoals in turn, asking for a binary judgment about whether each was satisfied. When it finds an unsuccessful one, it retrieves the event history up to that point and asks what happened and what caused the failure.
The paper’s pot example makes the value of looking backward concrete. At 00:44, the robot attempts to place the pot on a stove burner, but the pot is absent from the gripper. The relevant earlier event is at 00:36, when the pot was dropped during transport. An endpoint-only description locates the failed placement; the earlier events supply the proposed explanation for how the robot reached that state.
The three levels divide the work:
| Summary level | Information used in the example |
|---|---|
| Subgoal endpoint | The intended placement was not achieved |
| Event history | The object was lost earlier during transport |
| Sensor-derived representation | Object relations, gripper state and audio cues underlying those events |
The times and pot scenario come from the source’s explanation example. They illustrate the query route through one execution history, rather than a policy for treating every failed placement as a dropped-object case.
Completed subgoals can expose a planning error
REFLECT takes another branch when all planned subgoals are achieved but the overall task still fails. It then uses the task’s goal condition, the original plan and the final environment state to ask what was wrong with the plan.
In the paper’s second example, the robot places the pot on burner four but turns on burner two. Each instruction can be completed as written. Checking those instructions individually does not reveal the mismatch between the pot’s location and the active heat source; comparing the final state with the overall goal does.
For this analysis, the final state comes from the scene graph built from the accumulated point cloud at the last timestep. REFLECT removes view-dependent relations such as left, right and occlusion. The question concerns the enduring object states and relations relevant to the goal, rather than how those objects happen to appear from the last viewpoint.
The two branches consequently ask different questions of different evidence. Execution analysis looks into the history around an unmet subgoal. Planning analysis compares an achieved sequence with the final goal. Extending the event caption alone would not repair a plan that consistently names the wrong burner.
Construct the correction from the state that remains
The correction planner receives the failure explanation, the final state and the desired goal, along with the original task and plan. It starts from the state left by execution. In the burner example, the proposed correction switches off burner two and switches on burner four, using the fact that the pot is already on burner four.
REFLECT then maps each generated action description to the closest executable action available in the task environment using sentence embeddings. This connects the language plan to the environment’s action vocabulary. Which corrective actions can be selected therefore depends on that available repertoire as well as the explanation.
The robot data-quality audit covers the observation, state and timing relationships that feed this process. REFLECT adds a particular way to organize that evidence: use subgoal outcomes to locate the question, retrieve event detail when execution failed, and compare the final state with the plan when the completed steps missed the goal.