TLA predicts three corrections after an unsuccessful insertion attempt: translation in x, translation in y, and rotation around z. Its input combines a short history of fingertip tactile images with a language instruction that describes the peg and the task. Its output is generated as text, then converted into numerical motion values.
The important connection is between physical correction labels and their token representation. The instruction provides task context, while the training targets come from peg–hole pose errors in simulation. Scaling and rounding those targets makes the numerical output easier to express through the language model’s tokenizer.
The Tactile-Language-Action paper describes this pipeline in its dataset and method sections. It provides a concrete case in which the text output represents an action vector rather than a description of how a contact feels.
A failed insertion creates the next correction input
Section III constructs the dataset in NVIDIA Isaac Gym. The gripper starts with a peg above its corresponding hole, with randomized misalignment in x, y and rotation around z. It then attempts a downward insertion.
If the peg collides with the hole during descent, the attempt is marked unsuccessful and the gripper lifts for another attempt. The tactile sequence around the collision is retained to infer the next correction. In the paper’s collection procedure, descent to a predetermined position without a collision supplies the success condition.
The simulator also supplies the peg–hole pose error for each attempt. The authors use that error to construct corrective action labels, with clipping to bound the adjustment. This separates the source of the training answer from the model input: tactile observations and the shape instruction are what the learned model receives, while simulator state supplies supervision during dataset construction.
A shape description can help condition how a contact pattern is interpreted. It is not the measurement from which the authors calculate the ground-truth offset. Keeping that distinction clear explains why this is action learning from a paired dataset, rather than asking a language model to invent an insertion correction from prose alone.
Tactile history becomes a visual input to the language model
The method uses four time samples from each of the left and right fingertip sensors. It combines the two aligned sequences into a single image for the visual encoder. This makes temporal variation available through a spatial arrangement of the samples.
TLA builds on Qwen2-VL. A vision transformer encodes the combined tactile image, and a learned projection compresses groups of visual features before they are supplied to the language model alongside instruction tokens. The contact sequence provides the multimodal evidence; the instruction identifies the task and peg context.
This representation explains a specific modeling choice: time is represented by where the sampled tactile images appear, rather than by a dedicated tactile sequence model. Section VI identifies that treatment of temporal information as an area for further development. The article’s focus is the subsequent transformation from those features into a three-component correction.
Scaling shortens the numerical target
Section IV-B describes retaining numerical text encoding instead of replacing rarely used vocabulary entries with dedicated action-bin tokens. The authors scale the raw correction components and round them to integers before using them as targets. The scale is a three-component vector, so each action dimension can have its own conversion factor.
A hypothetical encoding example makes the arithmetic explicit. Suppose translation is expressed in millimeters and rotation in degrees, and choose invented scale factors of 100, 100 and 10 respectively. These factors are for illustration; they are not reported TLA settings.
| Component | Raw illustrative correction | Rounded scaled value | Value after dividing by the same scale |
|---|---|---|---|
| x | 0.236 mm | round(0.236 × 100) = 24 | 0.24 mm |
| y | −0.374 mm | round(−0.374 × 100) = −37 | −0.37 mm |
| z rotation | 0.183° | round(0.183 × 10) = 2 | 0.2° |
In this example, one integer step represents 0.01 mm for translation and 0.1° for rotation. Rounding changes the reconstructed values slightly. The chosen scale therefore connects token compactness with numerical resolution; it is part of the action representation, alongside the unit and component order.
The paper specifies scaling and rounding but does not provide the numeric scale vector in the method passage. Its decoder is described as converting generated text to floating-point actions, so a reproduction needs the concrete scaling and decoding configuration as well as the model weights.
Next-token prediction learns the correction sequence
During training, TLA freezes the tactile encoder and fine-tunes the Qwen2 7B language model using LoRA. The encoded action labels supervise next-token prediction. Earlier ground-truth tokens are supplied during training, while tactile and instruction tokens provide the conditioning input.
During inference, the model generates the action text sequentially with beam search until the end token appears. The action de-tokenization stage then maps the generated text into floating-point values. This is the point where the textual output becomes a numerical correction that the insertion loop can use.
The three output components retain their physical roles throughout that path. A predicted x value is a lateral correction, and the rotation value concerns the z-axis orientation. They are not tactile property labels. The Touch100k tactile-language label guide covers phrase and sentence supervision for tactile semantics; TLA instead trains text generation toward motion corrections.
Two source details matter when reproducing the representation
The published x and y label equations contain a concrete ambiguity: each printed piecewise expression gives positive-error conditions for both branches. That leaves the negative-error branch unclear in the displayed text. This article therefore describes the source of the labels and their clipping role without supplying a repaired piecewise implementation. The numerical encoding example above starts with an explicitly invented correction vector, independent of those ambiguous branches.
The tactile-encoder passage also combines a stated patch-token count with a description of merging 2 × 2 feature groups. Those statements do not clearly identify the count after merging. The mechanism used here is the documented feature compression followed by language-model conditioning; an exact tensor-shape reproduction requires checking the implementation.
These are separate from the central dataflow, which the paper describes consistently: collision observations and a peg instruction enter the model, bounded correction labels supervise training, and generated numerical text is decoded for the next attempt. The current study is simulation-based; Section VI places deployment in real environments in future work.
No dataset generation, model inference or insertion experiment was performed for this article. To connect a decoded correction to an actual assembly state machine, the peg-in-hole force-control guide explains contact, search and insertion behavior. TLA contributes a learned mapping at a particular point in that loop: using the contact from one failed attempt to predict the correction for the next.