Where2Act predicts where and how an articulated object could be manipulated with a specified interaction primitive. A high actionability score at a drawer handle does not, by itself, give a complete executable pull. You also need a proposed orientation, an assessment of that particular action and a robot motion that can realize it. Keep these outputs together when interpreting the model.
The Where2Act project studies elementary interactions such as pushing and pulling articulated object parts. Its model has separate components for point actionability, action-orientation proposals and scores for proposed actions. Training uses simulated interaction in SAPIEN. This makes it a useful example of affordance prediction conditioned on an action family, rather than a general guarantee that a visible point is safe or useful to touch.
The handle is a location; the pull is an action
Imagine a cabinet handle beside a wall. Several pixels may identify the handle correctly, but only some approach directions leave room for the gripper. Once contact is established, the drawer’s motion is constrained by its joint. A point detector can be right about the handle while a proposed action is wrong about clearance or direction.
In this hypothetical scene, distinguish three questions. Is this point worth considering for the chosen primitive? Which orientations are plausible there? How likely is a particular point-and-orientation proposal to achieve the primitive’s goal under the model’s assumptions? Collapsing them into a single heatmap hides the choice the robot still has to make.
| Prediction | How to interpret it | What it does not settle |
|---|---|---|
| Actionability at a point | A location is promising for the specified interaction | The best executable orientation for your gripper |
| Orientation proposal | A candidate way to approach or act at that location | Whether that candidate will succeed |
| Action score | The model’s assessment of a particular proposal | Calibration on a new object, robot or physical surface |
The model’s score is not automatically a calibrated real-world probability. Check its meaning and evaluation conditions before using a threshold to permit motion.
Name the primitive before reading the heatmap
Affordance depends on what the robot is trying to do. A surface that supports a useful push may not support a pull. Likewise, an interaction defined for one tool shape may be impossible for a wider gripper. Compare predictions only while preserving the primitive and the assumptions that give its score meaning.
The Where2Act paper frames the problem around short interactions with articulated objects. It does not supply a complete task planner for opening a cabinet, retrieving an item and closing it again. If your application needs that sequence, treat an interaction proposal as one component in a larger control system.
Before integrating a prediction, write down the intended motion, contact model, end-effector geometry and success measurement. For a drawer, success might require a specified displacement rather than any visible movement. That is your application criterion, not a result established by the paper. Keeping it explicit prevents a model’s training label from silently becoming your product’s completion rule.
Inspect a proposal in the robot’s geometry
A useful offline review overlays the candidate point and orientation on the observed object, then checks the corresponding 3D pose against the robot workspace and nearby surfaces. Missing depth or an uncertain surface normal should remain uncertainty; filling either with a convenient value can make an attractive visualization physically misleading.
Next, check the complete approach and retreat, not only the contact pose. The end effector may reach the handle while the wrist or arm collides with the cabinet. A planner can reject that motion even if the local affordance predictor assigns a high score. Preserve that rejection as an execution constraint rather than labeling it a perception error.
For a controlled physical evaluation, record the exact primitive, point, orientation, score, object configuration and actual outcome. Compare failures across object geometry and contact conditions. If highly scored proposals fail after touching the right part, investigate the proposed motion and contact assumptions before replacing the point detector. The Drake contact-model guide explains why the contact representation itself also matters when evaluating simulated interactions.
Read the real-image examples within their scope
The project’s real-world section visualizes predicted interaction information on real data, including 3D scans from Replica and a Google Scanned Object, plus a 2D real image. Those examples demonstrate predictions on such inputs; they should not be described as a physical robot completing those actions. The page also notes that its real-world examples lack the ground-truth movable-part masks available in simulation.
That distinction affects adoption. A convincing overlay on a cabinet photograph is evidence about visual prediction, not proof of grasp clearance, force control or successful articulation. Use Where2Act’s separation of location, orientation and action score to design those additional checks rather than treating the overlay as the final result.
Choose this approach when primitive-specific interaction proposals are useful to your system. Keep the proposed action, its score and the execution constraints connected all the way to evaluation. No Where2Act model, simulation or physical interaction was run for this article.