A 6D object pose is a rigid transform containing three-dimensional translation and three-dimensional rotation. It normally expresses an object frame relative to a camera or the inverse, but the six numbers alone do not reveal direction, timestamp, symmetry or uncertainty.
A robot grasp needs more than a camera pose. Camera intrinsics, depth scale, hand-eye calibration, robot state at acquisition, object-to-grasp geometry, reachability and collision checks all affect the executable tool pose.
Use this guide with the robot grasp-planning guide and robot coordinate-frame guide. Carry named transforms and time through the complete perception-to-motion chain.
Define translation, rotation and transform direction
Translation locates one frame origin in another frame. Rotation describes axis orientation. A matrix, quaternion, rotation vector or Euler angles can encode the same orientation, but each representation has conventions and singularities that must be documented.
Camera-to-object and object-to-camera transforms are inverses. Applying the correct numeric rotation and translation in the wrong direction can send a robot to a mirrored or distant pose. Use explicit frame names in variables, messages and logs.
Define whether the pose refers to a CAD object origin, visible instance frame, object center or task datum.
Choose the object-onboarding evidence
Model-based systems may use CAD geometry, rendered views or textured templates. Model-free systems may use reference images or an onboarding sequence. RGB, RGB-D and depth-only pipelines make different assumptions about texture, scale, missing depth and lighting.
The current BOP benchmark covers 6DoF pose and related tasks, including recent model-based and model-free challenges. Match evaluation tasks to the deployed onboarding and sensor conditions.
| Input evidence | Primary strength | Typical dependency | Failure pressure |
|---|---|---|---|
| RGB plus CAD | Texture and geometry | Accurate rendering and model | Appearance gap |
| RGB-D plus CAD | Metric surface evidence | Depth quality and alignment | Reflective or missing depth |
| Reference images | Fast object onboarding | View coverage | Unseen appearance |
| Depth or point cloud | Shape alignment | Initial pose and overlap | Symmetry and occlusion |
| Multiple views | More coverage | Timing and association | Moving object |
Calibrate camera intrinsics and depth scale
Focal length, principal point and lens distortion map image pixels to camera rays. An incorrect intrinsic model creates position and orientation error that varies across the image. Crop, resize or digital zoom can invalidate calibration if parameters are not transformed consistently.
RGB-D systems also need depth scale, depth-to-color registration and invalid-depth handling. Verify a known target at several distances and image locations.
Calibrate the camera-to-robot relationship
Eye-in-hand systems need the camera transform relative to the moving tool; fixed cameras need their transform to the robot base or workcell. Hand-eye calibration accuracy depends on robot pose accuracy, target geometry and diverse calibration motion.
Use a held-out physical target to check the complete base-to-object result. A small camera reprojection error does not guarantee a small robot approach error.
Build correspondences before solving pose
A common pipeline detects or segments the object, predicts 2D-to-3D correspondences or keypoints, solves one or more poses, then refines and scores them. Correspondence quality and spatial distribution matter; many clustered points can constrain pose poorly.
Preserve the inlier count, visible fraction and residual pattern. A high neural-network confidence cannot replace geometric consistency.
Use PnP with explicit camera geometry
Perspective-n-Point estimates camera-object pose from known 3D object points, their 2D image projections and camera intrinsics. Different solvers handle point count, planarity and initialization differently. Robust sampling can reject some outliers but needs a valid inlier structure.
The OpenCV solvePnP documentation lists available methods and conventions. Validate the exact solver, flags, point layout and returned transform direction in unit tests.
| Check | What it catches | Metric | Reject example |
|---|---|---|---|
| Reprojection | Image-geometry mismatch | Pixel residual | Large structured error |
| Depth consistency | Wrong scale or surface | Point-to-model distance | Missing visible surface |
| Symmetry set | Equivalent rotations | Symmetry-aware pose error | Arbitrary angle penalty |
| Transform chain | Frame or time error | Base-frame target residual | Stale robot state |
| Grasp trial | Task mismatch | Success and contact | Collision or unreachable |
Refine depth alignment without trusting a bad initial pose
Depth or point-cloud refinement can align observed surface with a model using ICP or another objective. It is a local optimization and can reinforce a wrong initial hypothesis, especially when only one side is visible.
Limit correspondence distance, reject invalid surfaces and compare the final pose with image evidence. The point-cloud registration guide explains local convergence and degeneracy.
Represent object symmetry in training and evaluation
A cylinder, repeated fastener or rotationally symmetric part can produce multiple poses that are visually and physically equivalent. Penalizing one arbitrary rotation as wrong creates misleading training and metrics.
Define discrete or continuous symmetry transforms from object and task geometry. Preserve multiple candidates when a downstream grasp or connector orientation breaks the visual symmetry.
Separate occlusion, truncation and instance ambiguity
Occlusion hides object surfaces, truncation removes parts at the image edge and similar instances create association ambiguity. Reflective or transparent materials can invalidate depth. These failures require different data and rejection policies.
Report visible fraction, image boundary contact, depth coverage and competing instance scores. Do not compress every failure into one confidence number.
Transform at the acquisition time
An eye-in-hand camera moves with the robot, and a target may also move. Use the robot joint state and frame transform corresponding to image or depth acquisition, not host arrival. Latency creates position error proportional to relative motion.
Apply the robot time-synchronization workflow and log the timestamp event. Predict motion only with a validated dynamic model and bounded horizon.
Evaluate pose metrics and grasp success separately
Translation, rotation, reprojection and symmetry-aware surface metrics describe different errors. Dataset averages can hide failure on one object, pose, occlusion or range. Report per-condition distributions and rejection coverage.
Then measure real grasp reachability, approach clearance, contact, lift and placement. A pose can satisfy a vision metric yet produce a poor grasp offset or collide with the bin.
Publish a pose hypothesis with operational context
Output frame names, transform direction, timestamp, object identity, symmetry hypotheses, uncertainty, visible evidence and rejection reason. Downstream motion software should know whether the estimate is new, predicted, refined or stale.
Release with a concise checklist.
- Name pose direction and object datum.
- Verify intrinsics, depth and hand-eye calibration.
- Preserve geometric residuals and symmetry candidates.
- Transform with the matching robot timestamp.
- Validate both pose metrics and executed grasps.
Frequently asked questions
What does the six in 6D pose mean?
Three translation coordinates plus three rotational degrees of freedom; it does not imply six independent sensor measurements.
Can a 2D detector alone support grasping?
It can localize an image region, but a general 3D grasp still needs depth or geometric and calibration assumptions.
Is RGB-D always more accurate?
No. Missing, noisy or misregistered depth can degrade pose; performance depends on material, range, calibration and method.
How should a symmetric cylinder be scored?
Use the task-defined set of equivalent rotations or a symmetry-aware surface metric rather than one arbitrary angle.
Why can a correct pose still produce a failed grasp?
Hand-eye, timing, grasp offset, reachability, collision, contact and object motion all remain downstream factors.
Pose Hypothesis and Executable Grasp Boundary
A 6D pose estimate is a hypothesis in named coordinates, not permission to move. Gate robot motion with calibration, time, reachability, collision and task evidence.