The VLA evaluation harness uses a shared recording database to connect a sharded run, its unfinished work and its exported results. Restarting a shard with the same eval_id resumes unfinished items. Export then selects the last committed attempt for each item and lists items that remain unfinished. The evaluation ID identifies the run; a worker process is only one participant in completing it.
This is the behavior described in the current official README, checked October 16, 2026. The explanation below follows its sharding and recording documentation; no evaluation, database recovery or model server was run for this article.
An episode belongs to the run, not permanently to one process
Episode sharding divides (task, episode) work across independent processes. With recording enabled, shards claim items from a queue in the shared recording database. That arrangement matters when one process stops: the evaluation still has an identity and a record of work beyond the lifetime of that process.
The database is written to <output_dir>/recording-<eval_id>.sqlite. It contains per-step rows, episode results and evaluation metadata. Sharing an evaluation ID and the corresponding recording store keeps workers associated with the same run. Creating a fresh ID when intending to resume would instead create a different run identity.
The README specifies a separate behavior for --no-save: sharding uses a fixed round-robin split, and recording is skipped entirely. Choosing that flag therefore changes more than disk usage. It removes the recording database on which the documented queue and export workflow relies.
This distinction also separates evaluation recovery from training recovery. A model checkpoint records learned parameters; this recording database tracks which evaluation items have been attempted and committed. Restoring a checkpoint alone does not tell an evaluation worker which episodes remain.
Export counts items after choosing their committed attempts
Imagine a small evaluation containing four items: A, B, C and D. In this hypothetical example, A and C have one committed attempt each. B has two committed attempts following a rerun. D has no committed result yet.
| Item | Recorded attempts in the example | Selection under the documented export rule |
|---|---|---|
| A | One committed attempt | That attempt |
| B | Two committed attempts | The last committed attempt |
| C | One committed attempt | That attempt |
| D | No committed result | Listed as unfinished |
There are four committed attempts in the example, but they belong to three completed items. Simply concatenating attempts into a results table would give B two entries and conceal the missing item D. The harness’s per-item export rule keeps those concepts separate.
“Last committed” is also a selection rule, not “best result.” If a later committed attempt differs from an earlier one, the documentation points to the later attempt. Choosing the most favorable outcome would answer a different question from exporting the run as recorded.
The robot VLA evaluation guide explains why trial counts and exclusion rules affect an aggregate score. Here the immediate practical step is narrower: read the unfinished-item list alongside the exported results before treating the run as complete. A result file can exist while some scheduled work is still outstanding.
Resume, unhealthy-worker handling and export do different jobs
The README tells operators to rerun a shard with the same evaluation ID to resume unfinished items. Its manual sharding example passes that ID to every vla-eval run invocation, together with the shard ID and shard count.
It also describes an early health rule: a shard whose first three episodes all error exits with code 3. The --requeue-unhealthy option gives that shard’s items to other shards. This addresses an unhealthy worker’s work allocation.
After workers finish, vla-eval export materializes per-episode JSONL and aggregate JSON from the database. Single-shard runs auto-export. For a manually launched sharded run, the documented sequence calls export once after all shards exit; the supplied run_sharded.sh helper launches shards and handles export automatically.
These stages answer different questions. Resume decides which work remains to be done. Worker-health handling decides where work can run. Export chooses the recorded attempt for each item and creates the result artifacts. Keeping the stages distinct makes an interrupted overnight run easier to interpret than a collection of worker exit codes alone.
A shared path must also support shared writing
Every writer needs access to both the database and its directory. The README explicitly requires shared storage to support SQLite locking and sync. A directory visible to every container is only the first part of that requirement; the filesystem must also provide the coordination behavior expected by the database.
Container ownership is another concrete failure mode. The documented default runs benchmark containers as root, which can leave output_dir root-owned. A later host-side export can then fail with Permission denied, even though the database was created. The README suggests docker.user: host or an explicit user/group identity in the evaluation YAML so the containers run as the invoking user.
That error belongs to result access rather than policy capability. It is different from an episode that reached its time limit or failed its task predicate. Keeping the original recording available allows the access problem to be addressed without treating the entire evaluation as a set of failed robot trials.
The dashboard follows the recording identity
Recording is enabled by default, while per-episode video is off unless requested. A dashboard is another layer: the harness can mirror aggregate metrics to W&B or Trackio, injecting the evaluation ID as the tracking run ID with resume allowed. Under sharding, aggregate emission is deferred to export; per-episode tracking belongs to the live execution path.
Consequently, dashboard activity and completed evaluation coverage are different observations. A worker can emit live information before all items finish, and an aggregate is produced through the export path. The database, selected per-item results and unfinished list explain what the aggregate actually represents.
For a sharded run that stopped overnight, the useful thread to follow is therefore concrete: the original evaluation ID, its recording database, the unfinished items, and the final export from that same record. That chain preserves the distinction between workers being restarted and evaluation items being counted.