TACT uses the robot’s arm joint positions as an observation, but its demonstrated actions are human arm poses. The learned policy predicts future commands in that human-pose representation. A separate retargeting and locomotion layer converts them into targets for the humanoid’s inverse-kinematics controller.
That choice preserves the interface used during teleoperation. The operator supplies an intended upper-body motion; the lower controller handles how the robot realizes it. At deployment, the policy takes over the operator’s role at the same interface, while the retargeting and control machinery remains in place.
The TACT paper makes this distinction explicit in Section V-B. Understanding it explains both what the imitation model learns and where tactile feedback enters the complete system.
The demonstration has two different pose representations
During collection, the human operator wears trackers on the waist, elbows and wrists. The measured elbow and wrist poses are expressed relative to the human waist. Retargeting maps these measurements to target elbow and wrist poses relative to the robot waist.
Section IV-B describes a calibration step that accounts for body proportions. The operator holds forward, sideways and upward arm postures. The tracker geometry defines reference axes and a shoulder origin; measured arm-segment lengths are compared with the robot’s to obtain scaling factors. Thus, mapping an arm motion includes a geometric transformation rather than a direct assignment of human coordinates to robot joints.
The training record keeps several quantities with different roles:
| Quantity | Role in TACT | What it describes |
|---|---|---|
| Robot arm joint positions | Policy observation | The robot’s current articulated configuration |
| Head-camera image and tactile measurements | Policy observations | Scene and contact-related evidence |
| Human elbow and wrist poses relative to the waist | Demonstration actions and predicted commands | The upper-body motion supplied to retargeting |
| Robot commanded joint positions | Lower-controller output | The joint-space realization of the motion targets |
This separation matters when reading a dataset or reproducing the architecture. A row can contain robot joint positions without making those positions the action label. In TACT, the human-pose action sits upstream of the retargeting controller.
Retargeting and balance targets meet in QP inverse kinematics
The lower layer combines several targets before computing robot motion. Retargeting supplies the elbow and wrist targets. The biped controller supplies center-of-mass and foot targets using a linear inverted pendulum model and the walking command’s footstep sequence. The paper formulates the inverse-kinematics calculation as a quadratic program.
The retargeting path also smooths target updates with a first-order lag. Its role is to control how quickly the IK targets respond to changes in the input arm poses. This places responsiveness below the learned action representation rather than requiring the imitation model to recreate that filtering in robot joint space.
For a hypothetical box-lifting command, the policy can predict a sequence of future elbow and wrist poses. The retargeting layer maps each pose into the robot’s geometry, and IK combines the arm targets with the current balance-related targets. The sequence is therefore an instruction to the lower layer, not a prerecorded list of actuator positions.
The broader whole-body control guide explains why simultaneous arm, foot and balance objectives share a control problem. TACT provides a concrete example of how a learned manipulation policy can feed that problem through a human-pose interface.
Tactile information conditions the predicted motion sequence
TACT extends Action Chunking with Transformers, or ACT, with tactile input. In this work, the tactile modality includes proximity sensing as well as direct touch. The policy combines those measurements with vision and robot joint positions to predict a sequence of future actions.
Section V-A describes a conditional variational autoencoder with an encoder, decoder and condition block. The tactile measurements are flattened into a one-dimensional vector and linearly projected into a token. That token enters both the encoder and the condition block. Image features and the other inputs participate in the multimodal policy, while the decoder produces the action sequence.
The training and deployment paths differ in a precise way. During training, the encoder has access to the recorded actions and sensory data, and training minimizes action-prediction error together with a latent-variable KL term. During deployment, the encoder is omitted. The condition block receives the current sensor observations, and the decoder generates future human-pose commands using a zero latent vector.
The tactile input consequently changes the learned prediction of intended motion. It can inform where the arms should move next based on contact evidence. The separate retargeting layer then determines how that prediction becomes robot motion.
Learned tactile response and wrist admittance are separate paths
The paper also describes a lower-layer wrist admittance mechanism. When wrist pressure is detected, the corresponding IK target is shifted along the surface normal in proportion to that pressure. This is a direct controller-level adjustment of a motion target, distinct from the tactile token conditioning the learned policy.
The experimental scope of that component is specific. The footnote in Section IV-B says wrist admittance was checked to function on the real humanoid, but was applied during manipulation experiments only in simulation, owing to the robot setup. The real-robot manipulation results and this simulation use of admittance therefore need to be read separately.
The official project description includes a fragile paper-box task involving contact across the forearms and wrists. That task motivates a policy that can use distributed tactile information. It does not require interpreting every change in arm motion as a wrist-admittance response: learned motion generation and the direct lower-layer adjustment occupy different points in the architecture.
Why the human-pose action interface is retained
Section V-B gives two connected reasons for the choice. It keeps collection and deployment consistent, since both provide the same kind of arm-pose command to the lower layer. It also leaves joint-space feedback behavior, such as controller-level compliance, to that lower layer instead of using it as the policy’s action target.
For a hypothetical comparison, two demonstrations could ask for similar arm movement while the lower controller produces different joint commands in response to the robot’s state. Recording the intended human-pose command preserves what the upper layer requested; recording only the resulting joint command would mix that request with the lower layer’s response.
This article interprets the paper and project text; no tracker collection, model training or humanoid trial was performed. The useful reading of TACT is the complete chain: tactile, visual and joint-state observations condition future human arm poses; retargeting maps those poses to the humanoid; and IK combines the resulting arm targets with locomotion and balance targets. Each representation has a defined role in that chain.