Two FurnitureBench scores are comparable only after you match the skill interval and starting conditions. A trial beginning at a manually prepared intermediate assembly state asks a different question from a full assembly trial. Even the documented low randomness setting treats these starts differently: the full task uses a fixed end-effector target, while the skill benchmark adds a small perturbation.
Before ranking policies, identify what each robot was asked to complete, what was already assembled, and how the robot and parts were positioned. This article explains the original FurnitureBench/FurnitureSim documentation checked October 15, 2026. No installation, robot trial, or benchmark was run for this article, and these historical environment names are not a claim of compatibility with a current software stack.
Read the interval as completed work and remaining work
The official FurnitureBench environment guide uses from_skill and to_skill to select a skill range. Its convention is easy to misread: an index i represents completion of the ith skill and the start of the next skill. to_skill must be greater than from_skill; the default to_skill = -1 requests the remaining task from the selected start.
Consequently, from_skill = 0, to_skill = -1 and from_skill = 2, to_skill = -1 do not request the same work. The second configuration starts after the first two skills are considered complete. Both use “continue to the end,” but only the first begins at the beginning of the task. This is an interpretation of the documented indexing, not a result from either configuration.
For skill-specific initialization, the guide asks the operator to match the furniture parts to predefined poses using the GUI. The script then moves the end effector to that skill’s predefined pose, with noise determined by the randomness level. The prepared state matters: the tested policy did not necessarily perform the earlier operations that created it.
Record the actual furniture and skill bounds alongside a score. If one policy is evaluated on a selected interval and another on the full task, retain the results as separate evaluations. An interval result can locate a useful capability or failure without measuring end-to-end assembly.
“Low randomness” changes with the benchmark
The environment guide describes three randomness levels for initial furniture and robot configurations. Its explicit end-effector perturbations are:
| Evaluation setting | Documented end-effector initialization |
|---|---|
Full assembly, low |
Fixed predefined target pose |
Skill benchmark, low |
Target pose plus ±0.5 cm positional and ±5° rotational noise |
med or high |
Target pose plus ±5 cm positional and ±15° rotational noise |
These are end-effector figures, not a complete specification of all furniture-part distributions. The table also does not make med and high identical benchmarks merely because their stated robot-pose perturbations match.
For high, the documented high_random_idx selects among indices 0–2, with -1 selecting an index randomly. Record the selected index or the selection procedure when comparing trials. A label such as “chair, high, 80%” leaves out which starts contributed to that percentage, as well as the number of trials and the evaluated skill range. That percentage is an illustrative label, not a published score.
Suppose a report gives an 8/10 success count for a manually initialized interval and another gives 6/10 for full assembly. Those hypothetical values do not show that the first policy is better at assembling the furniture. They differ in the work required and may also differ in their starting-state distribution. To compare the policies on the interval, evaluate both from the same specified interval protocol; to compare full assembly, evaluate both from the beginning.
Keep the camera setup in the comparison
A visuomotor policy also depends on what it sees at the prepared start. The camera-calibration section says the demonstrations include front-camera pose perturbations across episodes and describes calibrating against an average camera pose for each furniture type. Its display indicates whether the deviation is within a threshold. The page does not establish a universal numeric tolerance for every setup, so do not invent one for a comparison.
Save the camera configuration and applicable calibration result with the run. If viewing AprilTag detection, record what was actually checked; seeing a marker overlay alone does not certify every camera’s calibration. Also record lighting changes that could alter the policy’s input. These are practical controls for interpreting a visual-policy comparison, not extra performance claims made by the tutorial.
Preserve the real or simulated task identity
FurnitureBench’s real-robot environment and FurnitureSim have separate environment identifiers and options. The FurnitureSim guide describes init_assembled, which starts with fully assembled furniture, and lists FurnitureSimLegacy-v0 as the deprecated environment used in the main paper. A result labeled only “FurnitureSim” leaves both the starting condition and the environment revision unclear.
An assembled start may be useful for inspection or another defined experiment. It does not demonstrate that a policy assembled the object. Similarly, do not copy the real environment’s skill-range arguments into an arbitrary simulator configuration and assume equivalent behavior. Preserve the exact environment ID, code revision, supported options, and success definition used by the evaluated implementation.
Before comparing two result rows, put these fields beside both:
- Furniture, real or simulated environment ID, and implementation revision.
from_skillandto_skillwhere supported, including what was already complete.- Randomness level, high-random index or sampling procedure, and manual initialization method.
- Observation/camera configuration, calibration evidence, and relevant lighting conditions.
- Success criterion, episode limit, interventions, and successes out of total trials.
The broader household-chore explanation covers why consecutive skills can fail at their handoffs. FurnitureBench adds a precise reporting question: did the policy inherit a prepared intermediate state, or did it have to reach that state itself? Keep that distinction attached to the score, and it becomes possible to choose a matched follow-up test instead of comparing different tasks under one percentage.