A LIBERO score says little about transfer until the task suite and training exposure are named. LIBERO-Spatial asks about changed spatial relationships, LIBERO-Object about object identity, and LIBERO-Goal about changed task goals. LIBERO’s 90-task pretraining set and 10-task downstream set address a different question: whether prior experience helps or hurts later lifelong learning. If a team claims “object generalization” after mixing these tasks into one success rate, the first fix is experimental design, not a new policy. The official LIBERO repository defines the suites and its evaluation entry points.
Match one suite to one transfer question
Imagine that a policy succeeds at placing a bowl but fails when the requested bowl changes position among similar objects. That is a spatial-selection problem. A test that instead changes the object type would measure a different transfer pressure. LIBERO’s controlled suites make this distinction useful: Spatial holds a related manipulation pattern while spatial relations vary; Object varies the target object; Goal varies the requested outcome while object arrangements remain fixed. The paper’s suite design explains why these correspond to different kinds of knowledge.
| Claim to examine | More relevant LIBERO suite | What the result still cannot establish |
|---|---|---|
| The policy uses a changed spatial relation | Spatial | Robustness to unseen physical camera or robot geometry |
| The policy transfers across target objects | Object | Generalization to arbitrary objects outside the suite |
| The policy can learn a new requested outcome | Goal | Retention of older goals after more training |
| Pretraining changes later lifelong learning | 90 pretraining tasks followed by the 10 downstream tasks | Benefit without an exposure-matched from-scratch comparison |
The last row has a naming wrinkle. The paper describes the 10 long-horizon downstream tasks as LIBERO-Long; the repository training interface calls the corresponding benchmark LIBERO-10. State which source and task list your run uses. Do not silently treat a 90-task pretraining result as a 10-task downstream transfer result.
Preserve the exposure boundary
For a defensible transfer test, write down what the model saw before each evaluation. A policy pretrained on LIBERO-90 demonstrations has prior task exposure; a from-scratch policy does not. If a downstream task, its demonstration, or a near-duplicate appears in both sides of your own data pipeline, apparent transfer may be memorization. The official split offers a starting boundary, but any added data, converted dataset, or checkpoint can cross it. Record task identifiers, demonstration sources, checkpoint lineage, and whether task instructions were available during training.
Suppose two models receive the same downstream demonstration budget, yet one starts from a 90-task checkpoint. If it learns the new task faster, that supports forward transfer only under the stated exposure and task order. If it also forgets earlier tasks, the new-task gain is not the whole lifelong-learning result. Conversely, a model that retains old tasks but adapts slowly should not be called better at forward transfer. LIBERO’s paper distinguishes forward transfer, negative backward transfer, and an area-under-curve measure for this reason.
Fix the starting conditions before comparing policies
The repository’s task example loads a named suite and task, resets the simulated environment, retrieves the suite’s fixed initial states, and sets a selected initial state before rollout. It also describes a sparse completion reward. For your comparison, retain the suite name, task ID, initial-state ID, environment seed, instruction, and rollout limit for every trial. Changing the starting-state mix between models can produce a score difference without any change in knowledge transfer.
This is especially important when one policy struggles only from awkward starts. A pooled completion rate could make an initialization change look like learning. Evaluate paired policies on the same initial-state identifiers, then show the per-task distribution. If starts are sampled or extended beyond the provided fixed set, label that as a new protocol and publish the membership. A reader should be able to tell whether the result is about policy learning or about the sampled states.
For lifelong learning, also save the exact task order and checkpoint after each stage. After learning task three, for example, keep the learning curve for task four and reevaluate earlier tasks to measure retention. An optional evaluation before task-four training is a zero-shot baseline; it does not replace the paper’s forward-transfer metric. Run more than one stated order or seed if claiming order robustness. Reporting only the final checkpoint erases the path that makes a lifelong-learning claim meaningful.
The related GIFT analysis asks whether action supervision supplies the right learning signal. LIBERO suite selection answers a separate question: which transfer claim the evaluation actually tests.
An auditable run sheet
Before training, put the question in one sentence: “Does pretraining on the 90 short-horizon tasks improve learning of the 10 downstream tasks at the same downstream data budget?” Then pre-register the two checkpoint lineages, target suite, task order, fixed initial states, demonstration budget, evaluation interval, and success rule. Keep the policy architecture and tuning budget comparable where possible. Publish the per-task learning curves and prior-task retention alongside the aggregate.
If the question is narrower, use the corresponding controlled suite and avoid translating its result into a general “robot transfer” headline. LIBERO is a simulated manipulation benchmark with human-teleoperated demonstrations; its result is evidence about that protocol, not physical deployment. No LIBERO experiment was run for this article. The practical payoff of suite choice is precision: you can identify which knowledge changed, what the model had already seen, and whether a measured gain survived the same starts and sequential training history.