The reliable way to serve a VLA on a robot is not to select the fastest engine in isolation. Freeze the training-time input and output contract, then make ONNX conversion, TensorRT execution, preprocessing, action cadence and fallback a reproducible service. A successful export proves that an artifact was created; it does not prove equivalent actions or safe behavior.
Set a sensor-to-actuator limit with the robot inference latency budget. Track VLA quantization or distillation and LoRA adapter fine-tuning as separate model changes, each of which must repeat the deployment and closed-loop qualification below.
Define serving acceptance gates before changing code
Write a release contract for model version, camera count and resolution, language-token range, proprioception dimensions, action-chunk length, control cadence and latency limit. Accuracy gates should include offline numerical differences, task success, safety-constraint violations and human interventions rather than one aggregate loss value.
Fix temperature and power modes, competing processes, batch, warm-up count and sensor-replay data. Identical engines can behave differently under clock throttling, memory pressure and shape changes, so a latency result without its environment is not comparable or reproducible.
| Release gate | Required measurement | Example acceptance | Failure response |
|---|---|---|---|
| Contract | dtype, shape, normalization, action units | Exact match to reference | Stop the build |
| Parity | Output error and action difference | Within approved error budget | Inspect operator and precision |
| Performance | Cold/warm p50, p95, p99 and memory | Meets cadence and memory limits | Re-profile or redesign |
| Closed loop | Success, interventions, safety events | Passes test-plan thresholds | Roll back policy |
Freeze the model contract from pixels to action units
Extract image channel order, resize and crop rules, normalization constants, tokenizer version, padding, joint ordering, coordinate frames and action scaling from training code into one machine-readable contract. The Python reference and serving path should consume the same fixtures so a hidden preprocessing difference cannot survive review.
Name each input, dtype, tensor layout and dynamic axis, and state whether output means position, velocity, delta or torque. Timestamps, observation age and action validity are also API fields. Processing a stale observation quickly does not make the control loop real time.
Validate operators and shapes during ONNX export
Follow the PyTorch ONNX documentation, export with examples that represent the real input range, and pin exporter, opset, PyTorch and ONNX versions. Do not silently replace an unsupported operator with an approximation; compare intermediate and final outputs of the reference and converted graphs.
Establish parity with fixed shapes first. Then make only genuinely variable axes—such as batch, token length, camera count or image size—dynamic. Marking every axis dynamic widens optimization and memory uncertainty and allows unqualified input combinations into production.
Build TensorRT profiles and caches from observed ranges
Use the ONNX Runtime TensorRT Execution Provider guide and TensorRT documentation to record provider order, precision, engine cache, timing cache and build environment. Execution by another provider may keep a process alive, but it is not a validated safe-policy fallback.
Choose each optimization profile's min, opt and max from observed distributions as described in the TensorRT dynamic-shapes guide. Reject an out-of-range input or enter an approved conservative policy instead of reshaping it silently, and bind cached engines to model hashes and runtime versions.
Measure end-to-end action latency and peak memory
Measure from sensor timestamp through preprocessing, host-device transfer, inference, decoding, safety filtering and command transport to the actuator-command timestamp. Use the NVIDIA TensorRT benchmarking guidance, while reporting synthetic engine time separately from actual robot end-to-end time.
Test cold start, warm steady state, shape transitions, multi-camera bursts and concurrent logging. Report p50, p95 and p99 plus peak GPU, CPU and pinned memory. A fast average is inadequate when tail latency misses an action deadline; the queue policy and watchdog must react predictably.
Measure long enough to expose thermal throttling, allocator growth and cache eviction rather than ending after a short warm benchmark. Correlate every missed deadline with the active shape profile, queue depth, memory watermark and power state so the release team can distinguish a model regression from a system-level scheduling fault.
| Stage | Timestamp or metric | Typical failure | Observable |
|---|---|---|---|
| Input | Sensor creation and receipt | Stale frame or bad synchronization | Age, drop and skew |
| Pre/post | Start/end and CPU/GPU memory | Copy bottleneck or layout error | p95 and allocation |
| Engine | Enqueue and completion | Shape rebuild or OOM | Profile, cache hit, peak memory |
| Action | Decode, guard, send and apply | Missed deadline or stale action | End-to-end p99 and watchdog |
Progress from numerical parity to shadow and closed loop
With fixed fixtures, compare PyTorch, ONNX Runtime and TensorRT for absolute and relative output error, action ordering and decisions near control boundaries. FP16 or another precision is a separate candidate. A small mean tensor error does not guarantee the same action sequence over time.
Advance through recorded sensor replay, simulation, hardware in the loop, reduced-energy shadow mode and constrained closed loop. In shadow mode the new policy records proposed actions without commanding hardware. Closed-loop comparison then covers task success, intervention rate, constraint violations and recovery behavior against the reference policy.
Separate policy fallback, hardware stop and version rollback
Health checks should cover model hash, engine load, input age, shape range, p99 deadline, NaN or Inf, action bounds and heartbeat. A failure can select an approved hold pose, low-speed policy or previous stable version, but software fallback cannot replace independent safety PLCs, emergency stops and protective devices.
Bundle the model, tokenizer, preprocessing contract, ONNX graph, TensorRT engine, runtime, calibration and configuration in one deployment manifest. Define canary scope and automatic stop criteria, retain the prior manifest and caches, and make rollback a single approved operation that preserves evidence for reproduction.
Frequently asked questions
Does a successful ONNX export prove the VLA will act the same?
No. It proves artifact creation. The candidate still needs fixture-level parity, recorded replay, shadow operation, simulation and constrained closed-loop robot tests.
Is falling back to another execution provider enough for an unsupported operator?
It may preserve execution, but it is not a safe behavior fallback. Qualify output parity and latency for every provider path, then switch failures to a separately approved policy or safe stop.
Should VLA latency measure only inference?
No. Measure sensor age, preprocessing, transfer, inference, decoding, safety filtering and command delivery, including cold start, shape changes, p95, p99 and peak memory.
Official sources checked
- PyTorch ONNX documentation
- ONNX Runtime TensorRT Execution Provider
- NVIDIA TensorRT documentation
- TensorRT dynamic shapes
- TensorRT benchmarking
Last checked: August 7, 2026