PhAIL v1.0 measures three different things: how many items a policy moves, how quickly it moves them, and how much evaluation time accumulates per failed run. Its MTBF/A denominator is the number of runs whose outcome is anything other than Success. An operator-assistance count is a separate quantity, even though the metric’s name includes “assistance.”
The distinction becomes useful when two policies move the same fraction of items. One may finish more runs; another may repeatedly reach the time limit after partial progress. The v1.0 release protocol supplies the definitions needed to tell these behaviors apart. This explanation uses the published protocol, without running its robot or evaluation software.
Count items and runs separately
The v1.0 picking station uses one Franka Research 3 arm with a Robotiq 2F-85 gripper. Each episode starts with a known number of items from one object class. The policy moves them between totes, and the evaluator records both the outcome and the number successfully moved.
A run can therefore contribute useful item transfers while still counting as a failed run. For example, moving six of ten items before the timer expires contributes six completed items and one Ran out of time outcome. The episode timer in this protocol is 30 seconds multiplied by the initial item count.
The release defines completion as the fraction of items moved and UPH as units per hour, both averaged equally across object classes. It defines MTBF/A as total evaluation time divided by failed runs. The evaluator’s routine job of loading a tote is not an extra denominator event in that formula; neither is each unsuccessful grasp attempt within a run.
The same 80% completion can have different timing
Consider this hypothetical calculation, using four ten-item runs of a single object class for each policy. Every successful run takes 120 seconds. Every timed-out run reaches its 300-second limit.
| Quantity | Policy A | Policy B |
|---|---|---|
| Items in successful runs | 10 + 10 | 10 |
| Items moved in timed-out runs | 6 + 6 | 8 + 7 + 7 |
| Total moved / initially loaded | 32 / 40 | 32 / 40 |
| Evaluation time | 840 seconds | 1,020 seconds |
| Failed runs | 2 | 3 |
| Evaluation time / failed runs | 420 seconds | 340 seconds |
| Simple item/time rate | About 137 items/hour | About 113 items/hour |
Both policies complete 80% of the items. Policy A finishes more runs and spends less time moving the same total number of items. Its failed-run ratio is also higher. The last row is calculated as moved items divided by elapsed seconds, multiplied by 3,600; the published benchmark additionally specifies equal weighting across object classes.
MTBF/A is easiest to interpret beside those other rows. Its numerator includes evaluation time from successful and unsuccessful runs. Holding the failed-run count constant while making runs longer increases the ratio, which is why a larger value alone does not establish faster operation. With zero failed runs, the stated division has a zero denominator; the run count and exposure time then convey more information than a fabricated finite ratio.
Equal class weights answer a different question from a pooled total
PhAIL’s four object classes are towels, wooden spoons, scissors and batteries. Equal weighting gives each class the same contribution to the final completion and UPH summaries, even if their numbers of recorded runs differ.
An illustrative two-class example makes the effect visible: 100% completion on one class and 50% on another gives a class-balanced mean of 75%. Pooling every item first can produce a different answer if one class contributes more items. Class-balanced performance describes breadth across the benchmark’s objects; a production order mix would require its own stated weights.
The current overview uses a different presentation
The current PhAIL overview, checked October 15, 2026, describes Reliability and Speed. For Pick and Place, its displayed 120-second time limit applies to each item relative to the previous item. Reliability is the percentage placed within that limit; Speed is 60 divided by the mean time of the qualifying placed items, expressed in items per minute.
That differs from v1.0’s episode timer of 30 seconds × initial items and its total-evaluation-time/failed-runs metric. Select the definition associated with the result being discussed. Converting hours to minutes alone cannot reconcile a change in which events enter the calculation.
Throughput includes the inference arrangement
The v1.0 release also identifies where inference ran. ACT and SmolVLA used a notebook connected to the robot; GR00T used another machine on the same local network. OpenPI was served from Finland while the robot and notebook were in Cyprus, so each inference request included that round trip.
Moving a policy server closer changes the time between observations and returned actions. That is a concrete reason the same checkpoint can produce a different throughput on another installation. The distinction between a model’s output and the system that executes it is also covered in VLA versus VLM in robotics.
Read a PhAIL result as a joint description: object-balanced item completion, throughput under the recorded serving arrangement, and the distribution of run outcomes that supplies the failed-run denominator.