For the 2026 BEHAVIOR Challenge, do not choose a checkpoint by repeatedly inspecting the score you plan to report. The current evaluation rules assign public instance indices 0–9 to reported results: one rollout per instance, with the evaluator’s default timeout. Public indices 10–19 may be used as a test set before the final policy, while a hidden set is reserved for final evaluation. As an experiment-design recommendation, freeze the policy and reporting procedure before the 0–9 run, then keep the complete outcome, including failures. The explicit challenge rule prohibits cherry-picking rollouts; this additional separation helps limit checkpoint-selection bias.
Suppose three checkpoints are trained on the released demonstrations. A team runs each checkpoint five times on index 3, keeps the best success, and then reports that value with single runs on the other indices. That number no longer measures the stated reporting protocol. The simulator can be nondeterministic, and the rules explicitly prohibit assembling favorable rollouts across runs, instances or tasks.
Assign each instance set one job
The challenge provides training instances and human demonstrations separately from extra public evaluation configurations. Before training, write down which data enter the policy, which configurations guide development, and which run becomes the reported result. The following division combines the official protocol, with an additional team-owned validation set inside the permitted training material if you need repeated checkpoint selection.
| Data or instances | Legitimate job | Keep out of the reported claim |
|---|---|---|
| Released training instances and demonstrations | Train the policy; carve out a documented internal validation partition if needed | Any statement that training success is the challenge evaluation score |
| Public indices 10–19 | Test a candidate before its final submission; record all uses | Presenting them as the official 0–9 leaderboard protocol |
| Public indices 0–9 | One reported rollout per instance at the default timeout | Repeatedly selecting the best checkpoint or rollout from their outcomes |
| Hidden final instances | Organizer-run final evaluation | Claims about unseen results before that evaluation occurs |
The table does not imply that index 10–19 testing is automatically unbiased after repeated tuning. Record every exposure to those instances too. If the team uses them to choose among many checkpoints, call them part of the development process and keep the official 0–9 reporting run separate. A privately held training-data split can help with frequent experiments, but it does not replace the challenge’s prescribed reporting set.
Freeze more than the checkpoint file
The current rules tell participants to use the v3.9.3 tag of BEHAVIOR-1K for challenge evaluation; the live evaluation document explicitly warns against using an older tag. Record the tag or commit, evaluator command, task list, robot configuration, observation wrapper, policy checkpoint hash and action schema before the reporting run. The evaluator’s default timeout is task-specific and equals 1.5 times the mean human demonstration length. If a team changes --max-steps, it must label the result as a different protocol rather than silently compare it with the official score.
The challenge track permits RGB, depth and proprioception as policy inputs. It excludes simulator-only privileged information such as ground-truth object state, target pose and full-scene point clouds during evaluation. Training may use additional information under the stated rules, so preserve the boundary at the point where the policy server receives observations. This is a practical leakage check: a policy may appear to work on the public instances because an evaluation wrapper passed information the challenge track would withhold.
For each rollout, retain the instance index, seed or reset record if available, observation mode, timeout, unedited success/partial-credit metrics, video and failure note. The official evaluator writes JSON metrics and supports rollout videos; the rules require videos in challenge evaluation outputs. Do not combine successes from separate attempts into a synthetic episode. If a run crashes, record the crash and the rerun decision for the whole protocol before viewing which tasks benefited.
Report the result the protocol actually produced
Publish the per-task results and the aggregate with the exact run inventory, then describe any separate diagnostic runs as diagnostics. A reader should be able to distinguish a checkpoint chosen during development from the frozen one scored on 0–9. The challenge page still describes an active event as of September 2026; no final winner or hidden-set outcome is established by these public rules. Recheck the live protocol before submission because the evaluator version and instructions can change. For the broader choice of what a benchmark measures, the LIBERO suite guide separates different transfer questions. No BEHAVIOR policy or simulator evaluation was run for this article.