Do not choose a visual robot simulator from a single FPS number. For a vision policy, the useful comparison holds the task, environment count, cameras, image size, observation channels, physics settings and hardware constant, then measures time and memory for the work the policy actually needs. ManiSkill 3’s performance benchmarking page is unusually useful because it separates state-only simulation from rendered RGB and RGB-plus-depth workloads. It also warns that its published comparison is a work in progress, uses Cartpole, and is not strictly like for like with Isaac Lab’s different renderer.
Imagine a team training on three wrist and room cameras. A state-only throughput chart says the simulator can generate many thousands of steps per second, but the training job stalls as soon as all cameras render. The useful question is no longer “Which engine has the largest FPS headline?” It is “How many complete, correctly shaped observations and actions can this workload deliver within our memory budget?”
Define the workload before reading a benchmark
Write the observation contract on one line: task and scene, number of parallel environments, camera positions and count, image dimensions, RGB/depth/segmentation channels, control interval, and whether the policy runs during timing. Include object and geometry complexity. Keep the same robot, contact settings and episode termination rules if the comparison will guide a manipulation experiment. A benchmark on Cartpole cannot establish throughput for a dense kitchen scene; it can only show how the tested rendering and simulation paths behaved under its own load.
The ManiSkill page shows state-only, RGB and RGB-plus-depth cases, with camera payloads resembling real robot datasets. It reports frames per second and, for some figures, GPU memory on specified hardware. Its methodology section says reward and evaluation functions were deliberately left out. Those omissions are legitimate for isolating simulator throughput, but a training pipeline must add them back when estimating wall-clock progress.
| Measurement | What belongs in the timed path | What to report beside FPS |
|---|---|---|
| State-only simulation | Physics steps and state observation | Environment count, physics configuration, CPU/GPU and memory |
| Visual observation | Physics plus specified cameras, resolution and channels | Renderer, image payload, transfer path and peak GPU memory |
| Training or evaluation loop | The visual path plus policy inference, reward, reset and logging where used | End-to-end steps or episodes per hour and the slowest stage |
The rows answer different questions. State-only timing helps identify a physics bottleneck; visual timing helps size a camera-heavy simulation; the final row tells an operator whether a particular policy workflow fits the hardware. Do not present one as a substitute for another.
Keep the renderer difference attached to the result
The official comparison used SAPIEN rasterization for ManiSkill 3 and ray tracing for Isaac Lab’s parallel rendering. The benchmark authors explicitly describe that mismatch and show qualitative images alongside FPS. The result can help you form a test plan; it cannot prove that one renderer will preserve the appearance required by a real-world vision policy. A faster image with a different shadow, reflection or depth artifact can change the model input enough to alter policy behavior.
If the intended task needs contact-rich manipulation, reproduce a small representative workload in both systems. Freeze the scene assets and camera poses, choose equivalent output channels, then inspect the rendered model inputs before timing. Report any setting that cannot be matched. For example, if one renderer uses more samples to achieve usable reflections, that quality choice belongs next to its runtime number. Avoid hiding it inside an “equal resolution” label.
The current ManiSkill support table also matters before reserving hardware: it lists GPU simulation support on Linux with NVIDIA GPUs, while Windows and WSL do not have that support in the documented matrix. CPU simulation and rendering have different entries. Check the table for your target version rather than assuming that an available workstation can run the same benchmark path.
Make the comparison reproducible enough to change a decision
Keep the exact simulator version, command, device model, driver, scene, camera settings, warm-up rule, timed interval and peak memory. Run enough repeated trials to see whether the result is stable on the chosen machine, and retain raw logs rather than only a chart. If memory forces one system to use fewer parallel environments, report both the common configuration and each system’s feasible maximum. That shows whether throughput is limited by the renderer or by memory capacity.
For a policy-regression question, the Isaac Lab Arena analysis examines another evaluation workflow. A throughput comparison answers the narrower capacity question. The ManiSkill page does not measure end-to-end policy learning, and no benchmark was run for this article. The decision-ready output is a workload-matched table with quality differences, FPS, memory and omitted stages visible on the same page.