RoboArena compares two robot policies on the same task with closely matched starting conditions, then allows the next comparison to use a different scene. That makes the pair of trials the unit of control. Evaluators can contribute varied tasks across locations while keeping each A/B comparison focused on how two policies handle the same local situation.
Section 3.1 of the RoboArena paper describes the protocol. A central server samples two policies without revealing their identities to the evaluator. The evaluator then arranges a scene, writes a language instruction and runs the policies sequentially, stopping each rollout at completion or a fixed timeout.
Reset the pair; vary the next session
The initial conditions should be closely matched within a pair. Between pairs, the evaluator can rearrange objects, move the robot to another location or choose another task. Those are two distinct choices: matching a local comparison and expanding the population of situations being sampled.
Consider an illustrative task: move a mug from a shelf to a tray. Policy A starts with the mug near the front edge. If policy B starts with it pushed against the back wall, the comparison mixes policy behavior with a different reach and grasp problem. Restoring the mug and robot to closely matched starting conditions makes the two attempts answer the intended A/B question. The paper asks for close matching, so the reset procedure still matters in a physical scene.
After that pair, a different evaluator might ask the same policies to place a bottle in a drawer. There is no requirement that this new pair reproduce the shelf task. Each pair supplies a local comparison; the collection supplies diversity.
This changes how to interpret coverage. A set of comparisons concentrated on easy tabletop picks describes a different task population from one containing drawers, shelves and clutter. When comparing evaluation snapshots, examine which tasks and scenes entered each snapshot alongside the policy results. A larger number of trials can broaden coverage, or simply repeat a narrow set more often.
Progress and preference retain different information
The protocol collects three responses after both rollouts: a progress score for each policy, a binary preference between them and a free-text explanation of that preference. Progress runs from 0 to 100 and reflects the maximum progress achieved during the task. Zero means no progress, 100 means successful execution and intermediate values describe partial success.
Maximum progress is a property of the rollout history. Suppose an illustrative task requires opening a drawer and placing an object inside it. A policy might open the drawer, start moving the object and then lose its grasp. Its furthest achieved stage and its final scene describe different parts of that attempt. Keeping the recording makes the distinction visible when interpreting the score.
Preference captures a judgment that can remain even when progress scores match. The following is a hypothetical completed pair, not a reported RoboArena result:
| Feedback | Policy A | Policy B |
|---|---|---|
| Task result | Places the mug on the tray | Places the mug on the tray |
| Progress score | 100 | 100 |
| Observed behavior | Moves a neighboring object during the attempt | Leaves the neighboring object in place |
| Evaluator choice | Not preferred | Preferred |
The evaluator could explain the choice as “both finished, but B disturbed the surrounding arrangement less.” The paper leaves the basis of preference to the evaluator. The explanation therefore carries information about what drove the decision; preference is not a fixed formula that can be recovered from the two progress numbers alone.
A reader examining this pair can ask both how far the policies got and which behavior the evaluator valued. Averaging the progress scores preserves the first view. Keeping the preference and explanation preserves the second.
What travels from the robot back to the benchmark
The evaluator sends the instruction, pairwise feedback and recordings of observations and actions to the central server. Other evaluators can run their own pairs asynchronously. The two attempts in one comparison belong together even though the overall collection is distributed in time and place.
For a useful review of a pair, read the instruction and reset conditions before looking at the preference. Then use the rollout recordings to interpret maximum progress, followed by the evaluator’s explanation of the A/B choice. This order connects the judgment to the situation that produced it rather than treating a preference label as a complete account of the behavior.
The broader robot VLA evaluation guide covers fixed configurations, task outcomes and trial records. RoboArena offers a different way to organize those records: preserve the matched local pair while allowing the surrounding collection of tasks to grow. It is a useful design when the question concerns performance across varied evaluator-chosen scenes. A fixed regression suite remains useful for tracking the same prescribed tasks across revisions.
This article interprets the paper’s evaluation protocol, checked October 16, 2026. No robot trials or policy-serving software were run for this article.