Code as Policies turns a language instruction into a program that calls robot perception and action functions. Its usefulness depends on those functions having precise meanings: the model can compose a movement routine, but the robot still needs an implementation that knows which coordinate frame, units, observations and control limits apply. Before trying a new prompt, check whether every generated call can be resolved to a defined operation on your robot.
The Code as Policies project demonstrates language models combining API calls, calculations and control flow to produce robot behaviors. It also introduces hierarchical generation: when a program refers to an undefined helper, the system can generate that helper recursively. This is a way to assemble a program, not evidence that any plausible function name is already a working robot capability.
A readable program can still have an undefined action
Consider the instruction “Put the red block beside the blue block.” A generated program might locate both objects, calculate an offset and call a pick-and-place function. Each line looks reasonable. Yet “beside” leaves a geometric choice, object localization may return image pixels rather than a robot-frame pose, and the movement function may expect an absolute target rather than a displacement.
Those are interface errors even when the Python syntax is valid. In this hypothetical example, a function returning (x, y) needs a documented frame and unit. A second function accepting a target needs a documented height, orientation and completion condition. Without those definitions, the program can pass a type check while placing the object on the wrong side or at the wrong scale.
| Generated operation | Definition needed before execution | A useful offline check |
|---|---|---|
| Find an object | Output type, frame, units and behavior when detection fails | Supply an empty detection result and confirm it cannot become a target |
| Calculate “beside” | Reference object, axis convention and allowed offset | Evaluate the geometry on a labeled synthetic scene |
| Move to a pose | Absolute versus relative command, orientation and workspace | Check target bounds without enabling motion |
| Close the gripper | Command range and observable completion result | Confirm that command acceptance is distinct from a successful grasp |
This table is an integration checklist, not a claim that the research implementation enforces each check automatically.
Resolve helper functions all the way down
Hierarchical generation is attractive because a short top-level program can express intent clearly. A helper such as place_next_to_object can hide repeated calculations and lower-level calls. The important inspection point is where this expansion ends. A generated helper must eventually reach existing, permitted APIs; it cannot acquire a new sensor or control capability by defining another name.
List the generated functions and their dependencies before running them. For each leaf call, identify the concrete implementation and its input/output contract. Reject unresolved names, unexpected imports and calls that can reach outside the intended robot interface. Keep the generated program available for inspection rather than executing text as soon as the model returns it.
This is ordinary software integration applied to generated programs. It is particularly useful when a prompt example assumes a simulator helper that your real robot does not provide. Replacing that helper with a function of the same name is insufficient unless the behavior also matches. The paper presents program synthesis with robot APIs; it does not establish a universal API shared by arbitrary platforms.
Decide whether a behavior needs feedback or a waypoint
A waypoint command says where to go. A feedback routine repeatedly observes a state and changes its command in response. Code as Policies demonstrates both styles, including reactive controllers and waypoint-based behaviors. The generated source makes the distinction inspectable: look for the observation used, when it is refreshed and what ends the loop.
For example, “move until the object is centered” needs a fresh image or position estimate inside the loop. Reading the image once and repeatedly moving from that stale estimate is a different algorithm. Likewise, a loop that ends when the command is sent does not establish that the object arrived. Specify the maximum number of iterations and what the routine returns if the observation becomes unavailable.
An offline harness can supply a sequence of mocked observations: the target starts off-center, becomes centered, then disappears. The expected program should stop on the success condition and handle missing observations explicitly. Passing such a harness checks the program’s logic under those supplied cases. It does not validate camera calibration, physical contact or robot dynamics.
Separate language errors from robot interface errors
When a task fails, preserve three items: the instruction, the generated program and the actual values exchanged with the APIs. If the code selects the wrong object, inspect language grounding. If it selects the correct object but passes pixels into a function expecting meters, inspect the interface. If the target pose is correct but motion fails, inspect the planner and execution layer.
This separation gives you a specific next change. Rewording the prompt is unlikely to repair an undocumented unit conversion. Adding demonstrations is unlikely to repair an unresolved helper. Once a short program works, the long-horizon task guide addresses the separate problem of maintaining state and recovering across many actions.
Use Code as Policies when inspectable composition over a known set of robot capabilities fits the task. Start with a narrow API, test its geometry and failure returns, and only then assess whether generated programs express the desired behavior. No generated policy, robot API or physical execution was tested for this article; the examples above describe an integration review you can perform.