HoMeR uses an absolute 3D end-effector keypose for long reaches and relative 6-DoF actions for fine contact. At the handoff, check both the selected mode and how its action is decoded: a relative delta still needs a coordinate convention. In our AI-assisted NumPy model of the released conversion, swapping rotation-composition order changed one synthetic target by 14.133° while identity and same-axis checks missed the difference. This was a matrix calculation, not a HoMeR policy or robot test.
The HoMeR authors’ project pairs this hybrid imitation policy with a whole-body controller that converts desired end-effector motion into coordinated base and arm commands. Its demonstrations use a holonomic mobile manipulator with a 7-DoF arm. The reported tasks show a concrete design, not a universal rule that every reach must be keypose and every contact must be dense.
A keypose is an absolute target, even when built from an offset
The distinction can be easy to misread. HoMeR’s keypose branch uses a third-person point cloud to predict a salient 3D point, then predicts a relative offset from that point to form the desired position. It also predicts end-effector orientation, gripper state and the next control mode. The resulting command is an absolute 6-DoF end-effector pose for long-range motion. The local offset used to construct that target does not turn the whole keypose action into a dense relative controller command.
The dense branch is different: it uses third-person and wrist RGB images to predict relative 6-DoF action deltas near objects. Both branches eventually pass through the whole-body controller, which handles base/arm coordination. The paper’s method separates these outputs explicitly. Keeping their reference frames straight matters for data annotation, training targets and rollout logs. An absolute pose should be interpreted in a declared scene or robot frame; a relative delta should be interpreted with its preceding state and execution interval.
| Task phase | HoMeR action route | Annotation or interface question | Typical diagnostic if it fails |
|---|---|---|---|
| Approach a visible handle | Salient-point-conditioned absolute keypose | Is the 3D point on the intended handle, and is its frame known? | Wrong point, depth or offset before contact |
| Grasp and move the handle | Dense relative pose deltas | Are wrist view and delta convention consistent? | Contact timing or small-motion error |
| Coordinate base and arm | Whole-body controller below either policy | Can the requested end-effector motion respect joint and collision constraints? | Feasible target in perception, infeasible execution |
This table is an editorial fault-isolation tool, not a measured failure breakdown from the paper. It keeps the learned phase choice separate from the lower-level controller’s ability to realize a requested pose.
A relative action still needs a declared coordinate convention
The real-robot rollout code at revision 672aed1 adds the predicted position delta directly to obs['arm_pos'] when delta_actions is enabled. It composes orientation as delta_rotation * current_rotation. The cabinet configuration enables that option. The inline source comment says absolute at the conversion branch; the statements are converting a delta prediction into an absolute command.
This does not mean that any tool-relative delta can be added unchanged. Position addition assumes that the delta and observed position use the same coordinate basis. For orientation, SciPy documents composition as matrix multiplication in the same order: delta * current corresponds to D @ R. Swapping those operands generally changes the requested orientation.
We executed an AI-assisted NumPy matrix model of those two operations on a CPU, using Python 3.12.14 and NumPy 2.3.5. We did not execute HoMeR’s policy, upstream rollout script, SciPy quaternion operations, controller or robot. The inputs below are constructed rotations; the comparison deliberately swaps composition order while preserving every other input.
| Current orientation | Predicted rotation delta | Difference between D @ R and R @ D |
|---|---|---|
| Identity | 10° about x | 0° |
| 90° about z | 10° about z | 0° |
| 90° about z | 10° about x | 14.133° |
| 90° about z | 90° about x | 120° |
Identity and same-axis checks can therefore miss an operand-order error. The last row is a stress case, not a proposed robot command. The angle is the separation between two orientations, not robot tracking error or task success.
Translation has a similar trap. With synthetic position (0.60, −0.20, 0.40) meters and delta (0.01, 0, 0), direct addition gives (0.61, −0.20, 0.40). Rotating that delta by a 90° z orientation before adding it instead gives (0.60, −0.19, 0.40). The two targets are 14.142 mm apart. This is a deliberately introduced convention mismatch, not a failure observed in HoMeR.
This NumPy snippet reproduces the table and positional separation. Its matrices model the inspected operations; it does not load upstream code or weights.
import numpy as np
def rot(axis, degrees):
a = np.deg2rad(degrees)
c, s = np.cos(a), np.sin(a)
if axis == "x":
return np.array([[1, 0, 0], [0, c, -s], [0, s, c]])
if axis == "z":
return np.array([[c, -s, 0], [s, c, 0], [0, 0, 1]])
raise ValueError("use x or z")
def separation(a, b):
cosine = (np.trace(a.T @ b) - 1) / 2
return np.rad2deg(np.arccos(np.clip(cosine, -1, 1)))
for r, d in [(np.eye(3), rot("x", 10)),
(rot("z", 90), rot("z", 10)),
(rot("z", 90), rot("x", 10)),
(rot("z", 90), rot("x", 90))]:
print(round(float(separation(d @ r, r @ d)), 3))
local_delta = r.T @ d @ r
assert np.allclose(r @ local_delta, d @ r)
p = np.array([0.6, -0.2, 0.4])
dp = np.array([0.01, 0, 0])
r = rot("z", 90)
print("targets:", p + dp, p + r @ dp)
print("separation mm:", round(float(np.linalg.norm(dp - r @ dp) * 1000), 3))
assert np.allclose(p + r @ (r.T @ dp), p + dp)
If a receiving interface requires right-composed, body-relative increments, the corresponding rotation delta is R.T @ D @ R; a positional delta in that basis is R.T @ dp. Applying those explicit conversions recovered the original targets in the examples and in 100 seeded frame checks, with maximum absolute numerical error below 1e-12. That verifies our matrix algebra only. Use the conversion only after establishing the receiving interface’s actual convention.
At the keypose-to-dense handoff, log the action mode, delta_actions setting, observed pose and decoded target together. Test nonaligned axes before investigating phase timing. These calculations do not validate quaternion serialization, calibration, contact behavior or the learned choice of when to switch modes.
Label the handoff, not just the destination
The authors label modes along demonstrations with a web interface. Their cabinet example marks the reach toward the handle as keypose and the grasp/opening period as dense. The real-world collection guide requires mode labels and salient 3D point annotations after teleoperation. Thus a dataset needs more than endpoint coordinates: phase boundaries and consistent action frames must also be available. A poorly placed handoff can leave a keypose too far from contact, or make a dense policy spend many tiny steps traveling across the room.
Consider a hypothetical cabinet with a handle partly hidden by an open door. A proposed diagnostic would log the selected salient point, predicted absolute pose, mode-switch time, first dense delta and achieved end-effector pose. If the robot reaches the wrong panel, inspect point selection and frame conversion first. If it reaches the handle but slips while pulling, inspect the dense contact actions and gripper state. If both predictions look reasonable but the base blocks the arm, inspect whole-body execution constraints. This cabinet variant and diagnostic were not run for this article.
The point representation has a nearby but different problem in RoboPoint’s 2D-to-robot-target guide: turning a predicted image point into a geometric target requires depth and calibration. HoMeR’s keypose branch works in 3D point-cloud space and uses a salient point plus an offset to construct an absolute pose. The PerAct/Act3D spatial-action comparison also concerns spatial target representation; HoMeR’s additional question is when to use that type of target and when to switch to dense relative motion.
Interpret the reported result within its task setup
The paper compares hybrid versus dense-only policies and whole-body versus decoupled base-arm control. The data are not identical across all control schemes: whole-body baselines use whole-body teleoperation, while decoupled baselines use separately teleoperated base and arm demonstrations. Preserve that distinction when interpreting the comparison. The HoMeR-Cond generalization variant additionally uses external salient points and point-cloud augmentation; those ingredients are not automatically present in the base policy. Manual mode and salient-point annotation remain real costs even when inference is autonomous.
For a new deployment, retain the phase labels, 3D point and offset targets, dense-action conventions and whole-body-controller configuration together. Evaluate success and the first failure within each phase. An end-to-end success rate alone cannot tell whether the keypose, switch, dense policy or controller deserves attention. The author demonstrations motivate this hybrid design; no dataset, model, simulator or mobile robot was run for this article.