RobotArena Infinity summarizes a rollout by averaging the VLM’s progress estimates over its final 30% of frames. This gives a different answer from recording the highest progress reached anywhere in the video: a policy that briefly completes a placement and then knocks the object away can score below one that finishes steadily. The difference comes from which part of the trajectory enters the calculation.
The RobotArena Infinity project evaluates policies in simulated scenes reconstructed from demonstrations. This article explains the evaluation method in the paper version checked on October 15, 2026 (UTC); it reports a source analysis, not a benchmark or VLM run.
The judge sees both the video and simulation state
Section 4.1 of the RobotArena Infinity paper supplies the evaluator with ordered video frames and corresponding object and robot state information. It also includes the initial state explicitly as the zero-progress reference. The VLM then assigns a progress score to each frame using both inputs.
The initial reference gives the judge a starting point for the particular task instance. The state sequence supplies information about configurations and interactions alongside what the images show. These are inputs to the evaluator; this part of the method describes the judge’s access to privileged simulation state, not an observation specification for the policy being evaluated.
For a placement task, for example, the evaluator can consider the visible motion together with the recorded object and robot states. That is the method’s reason for combining the two sources. A reader interpreting its scores should therefore picture a state-informed trajectory judge, rather than a model watching an isolated final screenshot.
A short calculation shows the effect of the final window
Consider two invented ten-frame sequences below. The 0–100 scale is used only to make this example easy to calculate; these values are neither paper results nor outputs collected from its evaluator. Suppose rollout A reaches a high-progress state in the middle and then loses it, while B continues toward the goal.
| Frame | Illustrative progress A | Illustrative progress B |
|---|---|---|
| 1 | 0 | 0 |
| 2 | 20 | 10 |
| 3 | 40 | 20 |
| 4 | 80 | 30 |
| 5 | 100 | 40 |
| 6 | 100 | 50 |
| 7 | 60 | 60 |
| 8 | 40 | 70 |
| 9 | 20 | 80 |
| 10 | 0 | 90 |
With ten frames, the final 30% consists of frames 8, 9 and 10. The calculation is:
A: (40 + 20 + 0) / 3 = 20
B: (70 + 80 + 90) / 3 = 80
Taking the maximum instead would give A 100 and B 90, reversing their order. Taking only the final frame would produce 0 and 90, also different from the tail means. A maximum answers how far the rollout got at its best moment. The final-window mean describes progress across a stretch of its ending.
The paper motivates this window by focusing on the terminal part of execution, where completion or failure is most apparent. Averaging includes several observations rather than giving the entire trajectory score to one final frame. In the example, A’s earlier peak lies outside the averaging window, while its declining end states lower the score.
The averaging window belongs to the recorded trajectory
Because the window is a fraction of the frames, the trajectory endpoint matters. If a recording continues after a placement and captures a later disturbance, those later observations can enter the tail. If it ends earlier, the same late disturbance is absent. This follows directly from the aggregation rule and is why the episode stopping rule and recorded sequence should accompany a comparison.
Frame sampling also determines which moments are represented in that tail. A report that says only “progress score” leaves out the temporal choice needed to reproduce the calculation. The useful description is the frame-level scoring input, the recorded sequence and the final-30% mean. Our ten-frame example makes the window an integer; Section 4.1 does not specify how an arbitrary non-integer frame boundary is rounded.
For comparison, the RoboArena paired-trial article describes its separate maximum-progress annotation. The two similarly named projects select different temporal summaries, which is why their progress values need their own definitions before being compared.
Human preference supplies a separate ranking
RobotArena Infinity also collects human judgments from pairs of execution videos with matching simulation environments, initial conditions and task instructions. Reviewers choose one policy or a tie and provide a written reason. This is a second evaluation route alongside the frame-scoring calculation.
The paper fits a Bradley–Terry model to the decisive human comparisons to obtain a global policy ordering. Ties are retained in the collected data but excluded from that ranking objective. The model’s relative ability parameters come from those paired choices; they are not the final-window progress averages under another name.
That distinction is useful when reading the project’s two result views. The VLM progress view follows how far the task has advanced near the end of a rollout. The human preference view aggregates which execution people favored across matched comparisons. Keeping these questions visible makes a difference between the two views interpretable without changing either metric’s calculation.