Guardian creates failure examples in three different ways: it changes a proposed plan, perturbs an action in simulation, or changes how an existing real recording is paired with a task description or outcome image. These operations do not provide the same evidence. A plan can be wrong before anything moves; an execution can fail even when the plan names the right object and destination.
The construction rules are described in Guardian paper version 4, Sections 3 and 4. They produce examples for a verifier that judges planning and execution. The labels describe plan validity or execution completion. This article examines the method checked October 15, 2026; no failure data was generated and no verifier was trained or run here.
Change the plan while keeping its goal visible
A Guardian planning example contains the task instruction, proposed plan and initial front-view image. The verifier asks whether that plan is appropriate for the goal and scene. It does not need a completed rollout to identify a plan that omits an indispensable step.
The authors construct five planning-error types. Two change a semantic element: the manipulated object, or the target state or placement of the correct object. An LLM receives the instruction, plan and visible objects, changes one element, and leaves the remaining subtasks fixed. The other three types use rules to change order, delete a required subtask or insert a contradictory one.
Consider an illustrative instruction to place a cup inside a closed drawer. A plan that names the wrong cup has an object-selection error. A plan that puts the cup beside the drawer has a destination error. A plan that tries to place the cup inside before opening the drawer violates a dependency. Deleting the opening step or inserting a conflicting close step creates other plan defects.
The initial image and instruction give those changes meaning. Reordering two independent actions is not automatically an error, and an object name that is different in text is not necessarily a different visible object. Guardian’s generation rules create intended labels, but the verifier still has to ground the changed element in the particular scene and goal.
Change an action and observe the simulated outcome
For simulation, Guardian starts from successful scripted RLBench trajectories and divides them into subtasks. It chooses a subtask and perturbs its action. The resulting execution example includes the task and subtask descriptions together with images before and after the action, using multiple views.
The four perturbation types are concrete: omit gripper closure despite reaching the grasp position; leave the correct object in the wrong state or place; manipulate the wrong object; or introduce an imprecise grasp or push. Unlike a plan edit, these changes affect the simulated execution that produces the visual evidence.
For example, retaining a correct “pick up the cup” subtask while the jaws never close creates a different learning target from writing a plan that selects the wrong cup. The before/after images can show that the intended subtask was not completed. The planning input alone cannot establish that an otherwise correct grasp command actually succeeded.
The model’s formal inputs reflect this distinction. Planning verification takes the initial scene, task and plan. Execution verification additionally needs the subtask and observations before and after it. A prediction about plan validity therefore cannot substitute for a judgment about execution completion.
Re-pair real evidence without inventing a new physical rollout
Guardian also uses BridgeDataV2 recordings with ECoT subtask and object annotations. The authors clean those annotations before constructing examples. Because the actions have already happened, they cannot perturb the original execution in the way they can in RLBench. Their real-data construction uses existing imagery instead of generating a new physically altered trajectory through image editing.
For a task-execution semantic mismatch, the recorded start and end images remain paired as before, while the subtask instruction is changed. An LLM or a rule-based preposition swap supplies a different intended outcome. A recording that successfully puts an object beside a container can become a failure example relative to an instruction requiring it inside. The physical event did not change; the condition against which it is judged did.
The revert-action construction makes a different change. It retains the instruction but replaces the end image with the start image, presenting no visible progress. This means it would be inaccurate to say that every real-data failure example preserves the original before/after image pair. The source images are recorded evidence, but their pairing has changed.
These constructions give the verifier two specific learning targets: unchanged visible state in a start/start pair, and a mismatch between the requested outcome and a recorded outcome. Recovering the control error that caused a real failed attempt would require the actual execution sequence.
A verifier label is useful within its construction limits
Guardian adds generated reasoning annotations to its examples. For planning, the generation procedure checks subtasks and then the overall plan. For execution, it describes the before/after observations before judging completion. Those reasoning annotations accompany the constructed input pairs.
The paper makes that distinction concrete with UR5-Fail. Its execution examples come from autonomous 3D-LOTUS++ rollouts on a UR5, with three-camera observations and manual success/failure labels. Its planning failures still use the plan-perturbation method. Even within that evaluation set, “policy-driven” does not mean that every kind of label was collected in the same way.
The reported error analysis also leaves semantic limits visible: wrong final states or placements are difficult execution cases, while identifying the wrong manipulated object remains difficult in planning. A visually obvious nonprogress example and a subtle goal mismatch should therefore not be treated as equally solved because both carry a failure label.
The robot foundation-model guide discusses the broader problem of heterogeneous data and transfer. Guardian supplies a specific example of why the construction matters: the useful training question is determined by what changed—plan, simulated action, instruction pairing or image pairing—and by what the corresponding observations can actually establish.