Vision observes global object pose, scene context and motion before contact. Tactile sensing observes local pressure, shear, texture and slip after contact. Their strengths are complementary, but their measurements differ in rate, frame, field of view and failure mode.
Fusion is not simply concatenating two tensors. The system must define the common state, align acquisition events, calibrate geometry, represent sensor validity and expose uncertainty to control. Otherwise the model can learn a stable but physically wrong association.
Use this guide with the robot sensor-fusion guide and time-synchronization guide. Validate estimation quality and downstream task behavior separately.
Define the common state before choosing a network
A fusion target may be object pose, contact location, slip probability, material property, deformation, grasp success or the action itself. Each requires different labels and different spatial and temporal resolution.
Write the state and decision horizon first. A global camera pose at 30 Hz and a tactile vibration signal at kilohertz cannot be treated as interchangeable samples without an aggregation rule.
Map complementary evidence and shared failures
Vision can see the approach and non-contact surfaces but loses information under occlusion, glare or poor texture. Touch works through visual occlusion but only at contacted regions and can saturate, wear or change under a protective cover.
Some failures are shared. A loose tool mount changes both camera extrinsics and tactile geometry, while a dropped frame can desynchronize every stream. Redundancy claims should separate independent evidence from common infrastructure.
| Modality | Strong evidence | Blind spot | Quality signal |
|---|---|---|---|
| RGB vision | Appearance and global motion | Depth and occlusion | Exposure and feature support |
| Depth | Metric surface geometry | Reflective or missing returns | Valid-depth fraction |
| Tactile image | Contact shape and slip | Non-contact scene | Contact area and saturation |
| Force-torque | Net wrench | Contact distribution | Bias and overload |
| Proprioception | Joint and actuator state | Object surface state | Timestamp and limits |
Align acquisition time rather than host arrival
Camera exposure, tactile sampling, robot joint state and inference completion are distinct events. Queueing and transport jitter can pair pre-contact images with post-contact touch, teaching the model an incorrect causal sequence.
Prefer device or hardware timestamps when available and measure the remaining offset with a shared contact or light event. Record interpolation, window length and maximum accepted age in both training and deployment.
Calibrate pixels, taxels, tool and object frames
A camera observation lives in an image and camera frame. A tactile sample refers to a deforming sensor surface attached to a finger. Connecting them may require camera intrinsics, hand-eye calibration, finger kinematics, sensor placement and an object-frame estimate.
Use the robot calibration guide and coordinate-frame guide to name every transform. Verify residuals at working contact depths because a soft sensor surface is not a perfectly rigid point.
Choose early, feature or decision fusion deliberately
Early fusion combines closely aligned inputs and can model fine cross-modal interactions, but it is sensitive to missing or misregistered data. Feature fusion uses modality-specific encoders before interaction. Decision fusion combines estimates or actions and is easier to inspect but may discard detail.
The best architecture depends on state, data and latency. The Visuo-Tactile Transformers study is primary evidence for its tasks and setup, not a universal ranking of fusion mechanisms.
Preserve modality masks and uncertainty
The model should know whether a sensor is absent, stale, saturated or outside contact. Replacing missing data with zeros without a mask can make absence look like a real measurement. Per-modality quality can support adaptive weighting or an explicit reject state.
Calibrate probabilistic output or use task-level confidence gates where appropriate. Confidence must be checked against actual error and failure coverage, not only average classification accuracy.
Treat occlusion as task-stage information
The hand hiding the object is not only a vision failure; it often signals that contact or enclosure has begun. A fusion system can use that transition while reducing trust in the visual pose component.
Train approach, first contact, stable hold, slip and release as distinguishable stages when the task needs them. Keep contact labels synchronized with physical evidence rather than annotating from video appearance alone.
Detect slip from local and global motion
Tactile shear, pressure migration or vibration can reveal incipient slip before a large object displacement is visible. Vision can confirm global motion and separate object slip from finger or camera motion.
Define the slip event and lead time. A detector that fires after the object leaves the grasp may score well on frames but provide no control value. Measure detection delay and false actions under intentional finger motion.
Train missing-modality behavior explicitly
Random modality dropout can improve robustness only if it resembles credible failures. Test camera blackout, frozen frames, tactile disconnect, saturation, delayed packets and calibration drift separately. A model may silently rely on the easier modality despite nominal fusion.
Provide degraded actions such as slow, hold, reobserve or abort. Continuing the original task with lower evidence is not always the correct behavior.
Use ablation to prove fusion value
Compare vision-only, touch-only and fused systems under the same split, compute budget and control conditions. Add shuffled or delayed pairing tests to reveal whether the model uses meaningful cross-modal timing or merely more parameters.
The Cross-Sensor Touch Generation study also highlights that tactile representations are sensor-specific. Test hardware changes rather than assuming a model transfers across tactile designs.
Evaluate latency, accuracy and control outcome together
Report pose or slip accuracy, calibration, coverage and per-condition error. Then measure sensor-to-action latency, missed deadlines, grasp recovery, object loss and damage. A more accurate model that arrives after the control horizon can reduce task performance.
Replay synchronized logs for repeatability, but finish with hardware trials because contact changes when the controller responds. Preserve software, sensor and cover versions.
| Test | Comparison | Metric | Hidden failure exposed |
|---|---|---|---|
| Ablation | Vision, touch, fused | Task delta | Sensor dominance |
| Time shift | Correct and offset pairing | Error versus lag | Stale association |
| Frame shift | Correct and wrong extrinsic | Error versus residual | Geometry dependence |
| Dropout | Missing, frozen, saturated | Detection and recovery | Silent collapse |
| Closed loop | Estimator and controller | Loss, force, success | Late but accurate output |
Publish the fusion contract and failure policy
Document every input event, rate, frame, calibration version, preprocessing step, modality mask, fusion point, output uncertainty, deadline and degraded mode. Keep data windows and label construction with the model artifact.
Release with a concise checklist.
- Define the common physical state.
- Align sensor acquisition events and robot state.
- Calibrate rigid and deforming coordinate relationships.
- Expose modality validity and uncertainty.
- Test ablation, time shift, dropout and closed-loop recovery.
Frequently asked questions
Is concatenating camera and tactile features sensor fusion?
It is one implementation, but it is meaningful only after time, frame, validity and task-state relationships are defined.
Does tactile sensing replace the camera?
No. Touch is local and contact-dependent; vision provides global pre-contact and scene evidence that tactile sensors cannot observe.
Which fusion level is best?
There is no universal best level. Alignment quality, task state, data, latency, interpretability and dropout requirements determine the design.
Should the robot continue if one modality fails?
Only inside a tested degraded envelope; otherwise it should hold, reobserve, slow or enter a safe abort.
How is a fusion model evaluated?
Use per-modality ablation, timing and calibration shifts, dropout tests, uncertainty calibration, latency and closed-loop task outcomes.
Aligned Evidence and Missing-Modality Boundary
A fused estimate can look confident while its inputs describe different events. Enforce data age, frame validity and degraded behavior outside the learned model.