A MoveIt planner benchmark compares repeated attempts at concrete planning problems. Before interpreting a success rate or timing average, inspect how the benchmark expanded its start states, goals and constraints into requests. A configuration containing a few named inputs can produce many problems, and those problems may differ from the stored queries you thought were being replayed.
The first task is therefore to identify the denominator: which planning problems existed, which planner attempted each one, and how many attempts were retained. A chart becomes interpretable only after those identities are clear.
One benchmark problem can have several runs
The MoveIt benchmarking guide calls a particular scene, start state and constraint combination a benchmark. A run is one attempt by a planner to solve that problem. The guide also states that benchmark pipelines run directly with their configured request adapters, without using MoveGroup.
This matters when comparing a tuned planner against a baseline. Changing an adapter can change the request or resulting trajectory. The measured pipeline then differs in more than its planner name. Record those stages as part of the comparison.
Inspect the expanded requests
The pinned BenchmarkExecutor implementation expands goal constraints and stored queries with configured start states and path constraints. Its helper skips a start state whose name matches the benchmark request’s name. An empty list also has meaning: absent explicit start states preserve the request’s starting state rather than producing zero combinations.
Consider a deliberately simplified example with two goal requests, three distinct named start states, two path-constraint sets, one planner and five runs per problem. If every combination is admitted and no names trigger the skip, the configuration creates 2 × 3 × 2 = 12 problems and 60 attempts. Those are illustrative counts, not results from an executed benchmark.
Keep the generated request identities rather than reconstructing the count later from that multiplication. Real configuration branches and skipped combinations can change it.
| Record level | Information to retain |
|---|---|
| Scene | Robot model, collision geometry and relevant scene revision |
| Expanded problem | Actual start, goal, path or trajectory constraints and workspace |
| Pipeline | Planner configuration and ordered request adapters |
| Attempt | Status, elapsed measurement, available trajectory and failure information |
| Aggregate | Included attempts and the rule for handling failures |
Trajectory constraints follow a separate branch
In the inspected source, trajectory-constraint requests call the combination helper with an empty path-constraint list. They are not expanded with the configured path constraints in the same way as goal requests. Count that branch separately when it is present.
Workspace handling also deserves an actual-request check. The guide describes a configured workspace overriding existing settings, but the stored-query branch in the pinned code only replaces workspace bounds when all bounds are zero. A stored query with nonzero bounds can therefore retain a different workspace. Inspect the request passed to the planner instead of inferring it from one global setting.
These details can explain an apparently surprising comparison before any planner algorithm is investigated: the candidates may have received different geometric problems.
Preserve failures when summarizing speed
A timing average over successful attempts answers how long the solved subset took. It says nothing by itself about how often the planner failed or timed out. Report success and the timing inclusion rule together, and keep the per-problem breakdown when difficulty varies substantially.
For a proposed comparison, freeze the expanded request set and adapter configuration first. Change the intended planner parameter, rerun the same declared set, and inspect both solved and failed attempts. If a scene or constraint changes, give that comparison its own identity.
The existing parallel-planning explanation covers selecting among returned candidates. Benchmark construction comes earlier: it determines which problems generated the candidates being compared.
The guide and source were checked October 16, 2026; no planner benchmark was run here. The next useful artifact is an expanded request inventory with attempt records, not an isolated average detached from its problems.