A robot vision-language-action policy maps visual observations and a language instruction toward actions, but an evaluation observes the entire deployed chain. Cameras, prompt processing, policy inference, action conversion, controller, gripper and environment all contribute to the outcome. A success rate alone cannot identify which capability worked or failed.
A useful protocol freezes the configuration, defines tasks and initial states, stratifies familiar and shifted conditions, records trial-level outcomes and reports uncertainty. It also measures latency, intervention and physical side effects. This makes results comparable across model revisions without turning a research benchmark into an unsupported safety claim.
Use this guide with the diffusion policy guide and action chunking guide. Evaluate language grounding, perception, action generation and execution separately before attributing every result to a single model label.
Freeze the system identity before testing
Record model architecture, checkpoint hash, tokenizer, prompt template, image preprocessing, policy sampling settings and action normalization. Also record robot serial configuration, camera intrinsics and extrinsics, tool, payload, controller gains, firmware and software revisions. A model comparison is invalid when several surrounding components changed without attribution.
Define warm-up, reset and calibration procedures. Pin inference hardware and power mode because latency can affect behavior. Save environment assets and object identifiers, not only generic categories. If online adaptation, retrieval or memory is active, preserve its initial state and update history for every trial.
Tasks need observable start and end states
Write each instruction, allowed paraphrases, initial object region, robot start pose, timeout and success predicate. Replace judgments such as looks correct with measurable states: object inside a marked zone, drawer open past a distance or tool retained for a hold period. Define partial-progress milestones before observing results.
Specify what the policy may use. A hidden human correction, manual prompt rewrite or privileged object pose changes the task. Count retries and clarification requests. If an evaluator can stop unsafe motion, log the intervention as an outcome rather than deleting the trial.
| Evaluation layer | Question | Metric | Failure example |
|---|---|---|---|
| Language | Was the instruction grounded? | Object and relation choice | Wrong object |
| Perception | Was the scene represented? | Pose or state milestone | Occlusion error |
| Planning or policy | Was progress coherent? | Stage completion | Wrong sequence |
| Control | Were commands tracked? | Error and saturation | Overshoot |
| Safety behavior | Were limits respected? | Intervention and contact | Protective stop |
Stratify familiar and shifted conditions
Separate training-like combinations from new object instances, poses, backgrounds, viewpoints, lighting and language. Compositional tests can recombine familiar nouns, relations and actions, while harder tests introduce a concept absent from training. Label the intended shift instead of grouping every difficult trial as generalization.
Use enough repetitions within each stratum and randomize run order. Otherwise one easy object or one favorable camera angle can dominate the aggregate. Preserve a fixed regression set for version comparisons and a hidden set for model-selection leakage control.
Language grounding needs controlled contrasts
Test paraphrases that preserve meaning, then minimal pairs that change one object, attribute, spatial relation or action. Add irrelevant wording and plausible distractor objects. A policy that succeeds because only one graspable object exists has not demonstrated language-based selection.
Include instructions that should trigger clarification, refusal or no action when the target is ambiguous or unavailable. Score whether the robot avoids acting on an unsupported guess. Keep language tests culturally and physically interpretable in the scene; linguistic novelty alone should not silently change task feasibility.
Progress metrics make sparse success diagnosable
Break manipulation into stages such as approach, first contact, secure grasp, lift, transport and placement. Record the furthest valid stage, time to each stage and whether progress was later undone. Continuous distances can supplement milestones but should not replace task semantics.
Use a failure taxonomy agreed before testing: wrong target, missed grasp, collision, slip, stalled policy, timeout, controller limit, perception loss and evaluator intervention. Review ambiguous video blind to the model version. Report both task-level and failure-level counts so improvements have a mechanism.
| Condition | Minimum report | Uncertainty | Physical side metric |
|---|---|---|---|
| Matched tasks | Success count and trials | Binomial interval | Contact and intervention |
| Novel objects | Per-object outcomes | Stratified interval | Drop or damage |
| Language shift | Per-template result | Clustered by task | Wrong-object motion |
| View shift | Pose and lighting strata | Across-scene spread | Workspace excursion |
| Latency stress | Delay distribution | Percentile interval | Stale-command travel |
Trial counts prevent false precision
Ten successes in ten trials do not prove a 100-percent success probability. Report numerator, denominator and a suitable binomial confidence or credible interval. When repeated trials share the same object or scene, the observations are clustered; show per-object results instead of pretending every attempt is independent.
Plan sample sizes around the difference that matters operationally. Stop rules and excluded trials must be written in advance. If compute limits force small samples, describe the result as preliminary and retain raw outcomes. More decimal places do not create more evidence.
Latency and data age belong in the scorecard
Timestamp camera exposure, state sampling, inference queue, model completion, command conversion and actuator application. Report median and tail latency plus the age of each observation when its action reaches the robot. Average model throughput can coexist with rare stalls that dominate physical failure.
Use the ROS 2 real-time control guide to separate inference scheduling from a bounded hardware loop. Inject controlled delays and dropped frames, then measure stale-command travel and fallback behavior. The system should expose uncertainty or stop rather than silently execute an old plan.
Interventions and side effects are outcomes
Count emergency stops, protective stops, operator takeovers, manual object resets, collisions, drops and force or torque limit events. A rescued trial is not autonomous success. Report near misses and workspace violations even when the final object state satisfies the task predicate.
Coordinate contact evidence with the robot collision-detection guide. Capability evaluation does not establish functional safety. Protective functions, risk assessment and validation remain separate engineering activities under the standards and application requirements that govern the robot.
Generalist-policy examples need claim discipline
The Octo project describes a transformer-based diffusion policy pretrained on a large mixture of Open X-Embodiment data and evaluated across multiple real robot setups. It is a useful example of cross-embodiment policy work, but its results remain specific to reported tasks, robots and evaluation procedures.
The official Octo repository records implementation and license details. When adapting such a model, document observation and action adapters, fine-tuning data and controller differences. Do not transfer a paper-level average to an untested robot or describe a generalist policy as universal.
Publish an auditable result package
Release task definitions, configuration manifest, per-trial outcomes, seeds, timestamps, evaluator rules and failure labels. Retain synchronized video and telemetry where privacy and rights allow. Aggregate tables should be reproducible from trial-level data with a versioned script.
The Octo paper and Open X-Embodiment repository provide primary context for one generalist-policy ecosystem. Cite exact revisions and separate source claims from your measurements. A transparent negative result is more valuable than an unexplained success percentage.
- Freeze model, prompt, robot and controller identity.
- Define measurable tasks and partial progress.
- Stratify matched, compositional and shifted cases.
- Report counts, uncertainty, latency and data age.
- Count interventions and physical side effects.
Frequently asked questions
What does VLA mean in robotics?
It usually refers to a vision-language-action model that conditions robot action on visual observations and language. Architectures and action interfaces differ, so the label alone is not a specification.
Is task success enough to compare two policies?
No. Include trial counts, uncertainty, task strata, progress, latency, interventions and failures. Otherwise the same percentage can describe very different systems.
How should generalization be tested?
Name the shift, such as new object, pose, viewpoint, wording or combination, and report each stratum separately from training-like conditions.
Should a human rescue count as success?
No. Record the autonomous outcome and intervention separately. A rescued completion can be useful diagnostic evidence but is not autonomous success.
Does a high VLA score prove the robot is safe?
No. Capability benchmarks do not replace hazard analysis, protective functions, safety validation or application-specific operating limits.
VLA Evaluation Claim Boundary
VLA evaluation results apply to the recorded model, robot, controller, task distribution and test procedure. They should not be generalized to untested environments or used as a safety certification.