Use an eye-in-hand setup when the camera is rigidly attached to the wrist and an eye-to-hand setup when the camera is fixed in the cell. Before collecting data, write down the source and destination of every transform. The OpenCV calibrateHandEye API consumes gripper-to-base and target-to-camera pose sets and returns a camera-to-gripper transform. A plausible matrix from reversed inputs is still a wrong calibration.
Do not stop when a solver produces a number. Collect poses with rotations about at least two independent axes, reserve several observations for validation, and measure error at the robot's actual work locations. The robot calibration guide covers the wider taxonomy, robot frames and TF2 explains transform conventions, and the 6D pose guide owns target-pose estimation.
Freeze the physical arrangement and frame directions first
In an eye-in-hand arrangement, the camera moves with the end effector while the target remains fixed in the robot workspace. In an eye-to-hand arrangement, the camera remains fixed and the target usually moves with the tool. Both can be expressed as AX=XB, but the required inversions and the meaning of the result are not interchangeable.
Draw base, gripper, camera optical and target frames on one page. Record whether each API reports parent-to-child or child-to-parent, confirm the optical-axis convention, and normalize translation units. This diagram becomes the contract used by the data logger, solver, TF publisher and validation script.
| Arrangement | Fixed components | Moving components | Relationship to estimate | Common mistake |
|---|---|---|---|---|
| Eye-in-hand | Target and robot base | Gripper and camera | camera ↔ gripper | Treating gripper-to-base as base-to-gripper |
| Eye-to-hand | Camera and robot base | Gripper and target | camera ↔ base | Forgetting to invert a target transform |
| Both | Target geometry and frame definitions | Robot poses | Consistent SE(3) transform | Mixing millimetres, metres or rotation conventions |
Pass camera-intrinsic and target-quality checks before AX=XB
Hand-eye calibration cannot repair poor intrinsics. Lock the lens, focus, resolution, exposure path and any protective window in the configuration used during operation. Verify that CameraInfo matches the live stream and repeat intrinsic calibration after a lens, focus or enclosure change.
The OpenCV calibration-pattern guide describes supported patterns and practical detection considerations. Use a flat target with measured dimensions, check print scaling, and avoid frames where the target is tiny, blurred, clipped or severely oblique. Those pose errors propagate directly into the hand-eye estimate.
Pose diversity matters more than simply adding samples
Repeated translations with nearly identical orientation do not constrain AX=XB well. The MoveIt hand-eye tutorial notes that calculation starts after five observations, requires rotation around at least two axes, and often improves until roughly 12 to 15 samples. Treat those counts as guidance, not a guarantee.
Cover central and outer workspace locations, near and far target distances, and positive and negative rotations while preserving collision and joint-limit margins. Capture only after the robot settles, pair images and joint states with reliable timestamps, and reject observations with weak detection or motion blur before solving.
Compare solvers, but investigate the data when they disagree
OpenCV implements separable rotation-and-translation methods such as Tsai, Park and Horaud, plus simultaneous approaches including Andreff and Daniilidis. Similar results across methods are useful evidence of a well-conditioned dataset. A large disagreement is a reason to inspect frames, pose diversity and outliers, not permission to choose the most flattering output.
Run several solvers on the same clean set and record translation, rotation and residual differences. Repeat after removing one sample or bootstrapping the pose set to expose sensitivity. Select the release candidate only after it performs well on observations that were not used by the solver.
| Measure | Data used | What it supports | What it may miss |
|---|---|---|---|
| Target reprojection error | Image corners | Detection and intrinsic consistency | Robot-frame error |
| AX=XB residual | Training pose pairs | Internal equation consistency | Overfit or workspace bias |
| Holdout transform error | Unused poses | Generalization to a new view | Tool-centre-point offset |
| Pick or probe error | Independent work points | Task-level suitability | Dynamic delay and backlash |
Validate on unseen poses and the real task volume
Set aside holdout observations before tuning. Apply the estimated transform to place the observed target in the base frame and check whether its reported position drifts as the robot changes pose. Report maximum and directional bias as well as the mean; an average can hide a systematic rotation error.
Then approach known points with the actual gripper or a measurement probe. Test near, far, high, low and corner locations rather than one point at image centre. Independently verify the tool centre point so that a TCP error is not blamed on the camera transform.
Use the shape of the error to diagnose the fault
A nearly constant translation offset across all poses often points to target scale, TCP or a fixed-frame mistake. An error that rotates or changes direction with robot pose suggests a rotation estimate, transform inversion, timing issue or structural compliance. Error that grows near image edges points back to intrinsics or the lens model.
If accuracy degrades only while moving, inspect image and joint-state timestamps, exposure timing and rolling-shutter effects. Also check camera-bracket repeatability and cable forces. Avoid hiding a mechanical or synchronization fault with an empirical software offset that works at only one pose.
Release the transform with provenance and a recheck trigger
Store more than the final matrix: camera serial number, lens, resolution, target type and measured size, robot and firmware version, TCP, pose list, solver, rejected observations and validation results. Put source frame, destination frame and units in machine-readable metadata rather than relying on a filename.
Revalidate after the camera or tool is removed, a bracket changes, the robot collides, focus moves or the imaging pipeline changes. In production, observe a reference target periodically and stop the affected task when drift exceeds the project-specific limit. No universal millimetre threshold fits every camera range, gripper and process.
Frequently asked questions
How many poses are needed for hand-eye calibration?
A solver can start with a small set, but a practical plan is roughly 12 to 15 diverse poses plus separate holdouts. Rotations about at least two axes and reliable target detections matter more than a fixed sample count.
Does a low reprojection error prove the hand-eye result is correct?
No. Reprojection error mainly checks image observations and the camera model. You still need AX=XB residuals, unseen-pose tests and physical pick or probe measurements.
Can the same transform be used for eye-in-hand and eye-to-hand?
No. The fixed and moving frames, required inversions and output relationship differ. The same family of solvers may be used, but the transform contract must be rebuilt for the actual arrangement.
Official sources checked
- OpenCV calibrateHandEye API documentation
- MoveIt hand-eye calibration tutorial
- OpenCV calibration-pattern guide
Last checked: August 7, 2026