In-hand manipulation changes an object’s pose after it has been grasped, using contact motions inside the hand. Rolling, controlled sliding, finger gaiting and regrasp can all contribute. The task is harder than holding because the contact set and stability margin change during the motion.
A target object pose does not specify a feasible finger sequence. The controller must estimate object and contact state, respect joint and force limits, avoid finger collisions and recover before a slip becomes a drop. Object shape, friction and actuator compliance may be only partially known.
Read this with the robot tactile sensor guide and grasp stability guide. Evaluate the complete initial-grasp-to-target-and-release episode, including resets and safe aborts.
Define the object frame, target and success tolerance
Specify whether the task requires rotation, translation or both, and express the target in a named object or hand frame. A visually symmetric object may allow several equivalent orientations, while a keyed connector may not.
Success should include final pose tolerance, hold duration, contact condition and readiness for the next task. A pose reached only while the object is already falling is not a successful manipulation.
Distinguish rolling, sliding, finger gaiting and regrasp
Rolling changes the contact location on both surfaces without gross slip. Sliding moves contact across a surface under controlled friction. Finger gaiting relocates one or more fingers while others support the object. Regrasp may temporarily release and establish a new grasp.
Real motions often combine these modes. Give the controller explicit mode labels or latent-state evidence, and define transition conditions rather than assuming a smooth trajectory uses one contact model.
| Contact mode | Useful capability | Primary risk | Evidence to log |
|---|---|---|---|
| Rolling | Precise local reorientation | Unmodeled curvature | Contact path |
| Sliding | Large pose change | Runaway slip | Shear and velocity |
| Finger gaiting | Expand reachable orientation | Temporary weak grasp | Supporting contacts |
| Regrasp | Escape joint limits | Drop during release | Object pose and support |
| Hybrid | Flexible behavior | Mode ambiguity | Transition confidence |
Estimate object pose without hiding occlusion
External cameras can estimate global object pose, but the hand and object occlude each other exactly when contact becomes important. Multi-view vision, wrist cameras or model-based tracking may reduce the blind region without eliminating it.
The OpenAI Dactyl report describes multi-camera pose estimation and simulated training for real in-hand reorientation. Treat its design as evidence for that system, not proof that any object and hand can use the same sensors or policy.
Use tactile sensing for local contact and incipient slip
Tactile arrays can reveal contact location, pressure redistribution, shear and vibration that vision cannot see through the fingers. Sensor covers and soft fingertips change the relationship between raw signal and physical wrench.
Calibrate the installed finger and temperature range. Fuse tactile evidence with object pose and joint state, and preserve uncertainty when a fingertip is saturated, detached or outside contact.
Track grasp stability throughout the trajectory
A stable initial grasp can become unstable after a finger moves or the center of mass rotates relative to gravity. The contact wrench set, normal force and joint torque margin must be recomputed or approximated during the episode.
Limit object acceleration and finger motion when the predicted or sensed margin falls. More squeeze is not a universal recovery because it may jam the intended slide, saturate actuators or damage the object.
Plan within finger kinematics and collision limits
Dexterous hands have many joints but also small workspaces, coupled tendons, backlash and self-collision. A fingertip pose that is reachable in isolation may be impossible with the other fingers and object present.
Track joint-limit distance, tendon or actuator load and collision clearance. Plan a finger gait before the current contact reaches its limit, leaving a supported path to the next contact.
Bound force, velocity and fragile-object damage
Force limits should reflect the object, finger surface and contact location. Thin shells, bulbs and packaging can fail below the hand’s actuator limit. Fast finger motion can create impact even when the final commanded force is modest.
Measure peak and integrated load where practical. Include breakable surrogates and defined stop behavior before valuable or sharp objects enter the test set.
Stabilize contact-mode transitions
Noisy touch can cause rapid switching between stick and slip logic. Add hysteresis, minimum dwell and a confidence requirement. The controller should know which fingers currently carry the object before another finger unloads.
Test near the transition boundary with changing friction, small disturbances and sensor delay. A policy that works only with one fixed friction coefficient has not learned a robust transition.
Train with failure, intervention and recovery data
Success trajectories underrepresent the states that precede a drop. Collect incipient slip, poor initial grasp, joint saturation, occlusion, unexpected contact, safe abort and human intervention with synchronized labels.
Separate training, validation and test objects by the intended generalization claim. Random frames from the same object and episode do not establish transfer to new geometry or material.
Use simulation without pretending contact is exact
Simulation can generate many contacts and randomize mass, friction, compliance, delay and actuator behavior. It is useful for broad policy training, but contact and tendon dynamics may differ materially from hardware.
The rapid motor adaptation study demonstrates real fingertip rotation from simulated training with online adaptation. Reproduce the deployment envelope and measure where adaptation fails rather than quoting an object count as universal capability.
Evaluate paths, drops and recovery, not final success alone
Report angular and translational error over time, completion time, path length, peak force, joint margin, number of contact transitions, drops, resets and safe-abort rate. Slice results by object, initial grasp, target and disturbance.
A fair comparison keeps reset assistance and object tracking conditions explicit. Count trials that require manual repositioning, because they affect throughput and autonomy.
| Test factor | Levels | Metric | Failure evidence |
|---|---|---|---|
| Object | Shape, mass, friction, fragility | Success and damage | Object identity |
| Initial grasp | Contact and pose variation | Reachable target rate | Start state |
| Target | Axis and angle | Pose error and time | Symmetry set |
| Disturbance | Push, occlusion, sensor delay | Recovery rate | Event timestamp |
| Deployment | Temperature and wear | Drift and aborts | Hardware version |
Publish a recoverable manipulation envelope
Document the hand, fingertip skin, sensors, object set, initial grasp distribution, target definition, control rate, force limits, reset policy and excluded conditions. Keep videos linked to synchronized logs rather than presenting only selected successes.
Release with a concise checklist.
- Name the object and hand frames.
- Track contact mode and support set.
- Bound joint, collision, force and damage margins.
- Train and test sensor dropout and recovery.
- Report drops, resets and interventions with success.
Frequently asked questions
Is in-hand manipulation the same as regrasping?
No. Regrasp is one strategy; rolling, sliding and finger gaiting can change object pose while maintaining other contacts.
Can cameras alone estimate the object pose?
They can in some setups, but hand occlusion and visually ambiguous objects often require multiple views, motion models or tactile evidence.
Are more fingers always better?
No. More fingers add contacts and control authority but also sensing, calibration, collision, actuation and planning complexity.
Can a learned policy own the force limit?
A policy may propose actions, but independently enforced hardware and supervisory limits are still needed for the validated risk and damage envelope.
What should be measured besides success rate?
Pose trajectory, time, force, joint margin, contact transitions, drops, resets, interventions and recovery success reveal deployability.
Contact-Mode and Recoverable-Motion Boundary
Dexterous motion should fail predictably. Keep object damage, drop energy and finger pinch hazards inside an independently enforced envelope during learning and deployment.