Open-vocabulary robot object grounding connects a free-form phrase such as the blue cup behind the box to a specific object that a robot can track and manipulate. Text-conditioned detection broadens candidate names beyond a fixed training taxonomy, but manipulation needs geometry and identity that a 2D label alone does not provide.
A deployable pipeline preserves linguistic ambiguity, proposes multiple visual candidates, refines instances with masks and depth, maintains 3D identity through motion and checks reachability, pose, grasp and prohibited regions immediately before action.
Use this guide with the VLA guide and 6D object-pose guide. Report perception and manipulation outcomes separately.
Preserve the user phrase and its ambiguity
Parse head noun, attributes, relations and requested action, but keep the original instruction and unresolved references. Normalization should not silently replace beside, behind or the other one with an assumed target.
When two objects remain plausible, return candidates and ask a targeted clarification. A fast arbitrary choice can turn perception uncertainty into an irreversible manipulation error.
Distinguish open vocabulary from guaranteed novelty
Open-vocabulary detectors accept text categories or referring expressions beyond a closed output list. They still inherit concepts, biases and visual coverage from pretraining and may fail on domain-specific tools, transparent parts or unusual viewpoints.
The OWL-ViT paper reports open-vocabulary detection results in its benchmark settings. It does not guarantee recognition or physical actionability for every unseen factory object.
| Stage | Output | Required evidence | Main failure |
|---|---|---|---|
| Language | Phrase and relations | Original instruction | Meaning lost |
| Detection | Candidate boxes and scores | Calibrated coverage | Target missed |
| Segmentation | Instance masks | Boundary and occlusion | Objects merged |
| 3D grounding | Points, pose and frame | Depth and calibration | Wrong geometry |
| Action check | Reachable grasp target | Fresh scene state | Unsafe contact |
Generate candidates before selecting one
Keep top candidates with text-conditioned scores, spatial evidence and an explicit no-match outcome. Thresholds should be calibrated on the target camera and object mix because score distributions shift across scenes and phrases.
Grounding DINO is an open-set detector that accepts category names or referring expressions, as described in its official paper. Use its reported benchmarks as method evidence, not a deployment threshold.
Attach an instance mask to the candidate
A bounding box may contain neighboring objects, background and multiple depth layers. A mask provides a better surface set for depth filtering, pose estimation and grasp generation, but its edges and occluded regions remain uncertain.
The Segment Anything paper describes promptable segmentation and zero-shot evaluations. SAM does not identify which named object the user intended; the detector, phrase and scene logic still provide that link.
Project pixels into a time-aligned 3D frame
Use valid depth, camera intrinsics, distortion model, timestamp and camera-to-robot transform to create an object point set. Reject or down-weight pixels near depth discontinuities and missing or multi-path measurements.
A 2D mask from time t and depth from a later moving-camera frame can create a plausible but incorrect 3D target. Enforce synchronization and frame validity before publishing the pose.
Maintain instance identity through motion
Repeated cups or parts require stable identities, not reselecting the highest-scoring box every frame. Fuse appearance, 3D position, motion, object state and occlusion logic into tracks with explicit creation, merge, split and retirement rules.
After robot or object motion, update the track and re-ground relational language. The object that was left of another may no longer satisfy the original description.
Represent relations in robot-centered 3D
Image left and right depend on camera viewpoint. For manipulation, express above, behind, nearest and inside using declared world, base, tool or object frames and tolerances appropriate to the scene.
Keep uncertainty and reference-object identity in the relation. If the reference is ambiguous or moving, the target cannot be considered resolved merely because one 2D ordering matches.
Check actionability beyond semantic match
A correctly named object may be unreachable, too heavy, attached, hot, fragile or inside a prohibited region. Grounding should return geometry and uncertainty, while a separate planner and safety layer decide whether an action is allowed.
Validate collision-free approach, grasp surface, tool compatibility, payload, visibility and expected post-grasp observation. Do not let a detector score override physical constraints.
Handle transparent, reflective and deformable objects
RGB-language matching may locate a transparent container while commodity depth misses its surface. Reflective metal can produce mixed depths, and deformable objects do not have one rigid pose.
Use task-specific sensing, multi-view evidence or conservative geometry when needed. Report these object families separately rather than averaging them into common opaque objects.
Revalidate immediately before contact
Between initial instruction and grasp, people, robots and objects may move. Refresh target track, frame transform, reachability, prohibited zones and grasp clearance just before approach and again when contact-sensitive control begins.
If confidence drops or identity changes, pause and ask rather than continuing from a stale pose. Connect this behavior to the policy-abstention guide.
Evaluate the pipeline in layers
Measure phrase-to-candidate recall, referred-instance accuracy, mask quality, depth validity, 3D pose or point error, track identity switches, clarification success, grasp planning and final task outcome. Include no-match and multi-instance scenes.
VoxPoser demonstrates language-conditioned 3D value maps for robot manipulation in its reported experiments. A new deployment still needs independent cameras, objects, instructions and safety evaluation.
| Layer | Metric | Hard case | Pass question |
|---|---|---|---|
| Language | Resolved references | Ambiguous phrase | Ask correctly |
| Detection | Candidate recall | Novel synonym | Target retained |
| Mask and depth | Valid target points | Occlusion or reflection | Geometry usable |
| Tracking | Identity switches | Repeated objects | Instance stable |
| Manipulation | Safe task success | Moved target | Action verified |
Release a grounding contract
Version text processor, detector, segmenter, thresholds, depth filters, camera calibration, frame transforms, tracking parameters, object restrictions and clarification policy. Log every candidate and decision with timestamps.
Close release review with the following checks.
- Preserve the original phrase and unresolved ambiguity.
- Keep candidates and a no-match outcome.
- Validate masks, depth and time-aligned frames.
- Maintain 3D instance identity through motion.
- Recheck reachability, restrictions and grasp geometry before contact.
Frequently asked questions
Is open-vocabulary detection the same as zero-shot detection?
They overlap, but open vocabulary emphasizes text-flexible labels while zero shot describes evaluation without target-class training examples.
Can I grasp the object with the highest CLIP score?
No. The score does not establish instance identity, 3D geometry, reachability or a safe grasp.
Does Segment Anything find an unseen named object by itself?
No. It produces promptable masks; language grounding must identify which mask represents the phrase.
How should transparent objects be handled?
Use sensors or multi-view methods suited to missing or unreliable depth and keep conservative uncertainty.
What if two candidates remain plausible?
Ask a discriminating clarification and keep control authority explicit rather than selecting arbitrarily.
Language-to-Manipulable-Instance Boundary
Open-vocabulary grounding expands the language interface, not the certainty of manipulation. A robot must connect words to a fresh, stable 3D instance and revalidate actionability before contact.