ROS 2 callback groups decide which callbacks may run concurrently, while an executor schedules ready callbacks on available threads. A Multi-Threaded Executor can still behave serially when work shares one mutually exclusive group, and synchronous calls can deadlock when their completion callback cannot run.
Robot timing failures often appear as stale sensor data, late control timers or blocked services. The cause may be callback execution, wait time before execution, a lock, synchronous service use or middleware queues. Thread count alone does not identify the bottleneck.
Use this guide with the ROS 2 real-time control guide and ROS 2 QoS guide. Trace the deployed executor before changing its configuration. If the missed deadline is inside the read-update-write path, move to the ros2_control hardware-interface guide.
How executors and callback groups divide scheduling
Subscriptions, timers, services, clients and waitables become executable callbacks when ready. The executor waits for work, selects eligible callbacks and invokes them according to its implementation and callback-group rules. It does not automatically infer which work is urgent for the robot.
List every callback with expected rate, maximum execution time, shared resources and deadline. Include hidden callbacks used by synchronous clients or futures. An incomplete inventory makes deadlock analysis and response-time claims unreliable.
A single thread makes blocking visible
A Single-Threaded Executor executes one callback at a time. A long image callback, blocking disk write or service wait delays every other callback in that executor. The simplicity is useful for determinism analysis when the combined worst-case workload fits.
Measure readiness-to-start wait and start-to-finish execution separately. If a timer begins late, the timer callback may be short while another callback owns the thread. Average CPU utilization can remain low when one blocking call creates the delay.
| Symptom | Likely layer | First measurement | Common cause |
|---|---|---|---|
| All callbacks pause | Single executor thread | Running callback duration | Blocking I/O |
| Only one group serializes | Callback group | Group and thread trace | Default group |
| Service call hangs | Dependency cycle | Future and group ownership | Self-deadlock |
| Fresh messages arrive late | Queue or executor | Sample age at callback | Slow consumer |
| Deadline misses under load | Scheduling and contention | Tail wait time | Shared lock or CPU pressure |
Multiple threads do not guarantee parallel callbacks
A Multi-Threaded Executor can run eligible callbacks on several threads, but callback-group restrictions still apply. If all entities use the node’s default mutually exclusive group, they cannot execute in parallel with one another even when many executor threads exist.
Inspect group assignment at entity creation and confirm that the group remains associated with the executor. The current ROS 2 executor concepts should be read for the deployed distribution because implementation behavior can evolve.
Use mutually exclusive groups for protected sequences
A mutually exclusive callback group prevents its callbacks from running simultaneously with one another. This can protect non-thread-safe state without a separate lock and can preserve ordering assumptions within the group. It can also cause unrelated long work to delay a critical timer.
Group callbacks by actual concurrency requirement, not simply by node. Put device transactions that must serialize together, while separating independent telemetry or expensive processing when the shared-state design permits. Document why each group is exclusive.
Reentrant groups require thread-safe code
A reentrant group permits callbacks, including multiple instances of the same callback, to overlap. It can improve concurrency for independent work, but it does not make accessed libraries, buffers or devices thread safe. Race conditions can corrupt state while benchmarks appear faster.
Review shared variables, publishers, clients, device handles and third-party libraries. Use bounded synchronization and avoid holding a lock across I/O. Stress with repeated overlapping requests and a thread sanitizer or equivalent test where practical.
| Group choice | Concurrency | Useful for | Primary risk |
|---|---|---|---|
| Mutually exclusive | One callback in group | Ordered device access | Hidden serialization |
| Reentrant | Overlapping callbacks | Independent stateless work | Data races |
| Separate exclusive groups | Parallel across groups | Independent serialized devices | Cross-group shared lock |
| Dedicated executor | Separate thread pool | Critical path isolation | Coordination overhead |
| Non-ROS worker | Explicit handoff | Blocking or batch work | Queue ownership |
Avoid synchronous self-deadlock
A callback that makes a synchronous service request may wait for a response callback that must run in the same mutually exclusive group. The waiting callback holds the group’s eligibility, so the response cannot execute. A similar cycle can occur with futures and actions.
Prefer asynchronous flow inside callbacks, place dependent callbacks in compatible groups or use a carefully designed separate executor. Draw the wait-for graph and test timeouts. Adding threads does not break a group-level exclusion cycle.
Size from execution and wait distributions
For each callback, record invocation rate, execution-time percentiles, maximum observed time and readiness wait. Add CPU affinity, thread identity and lock wait when possible. A thread pool should be sized from concurrency and blocking behavior, not from processor core count alone.
Include serialization, middleware take, memory allocation, logging and cache effects. Run long enough to expose thermal throttling and background jobs. A high throughput result can coexist with unacceptable control-tail latency.
Priority is not automatic
Executor selection alone does not guarantee operating-system thread priority, callback priority or bounded preemption. Critical and best-effort callbacks can share a worker thread or lock. A ready low-importance callback may run before a control timer depending on executor behavior.
Where deadlines matter, isolate critical work, configure scheduling and affinity deliberately, remove blocking operations and measure under worst-case contention. The ROS 2 real-time demo describes supporting practices, not a certification of an arbitrary application.
Separate QoS delay from executor delay
QoS affects compatibility, retention and transport behavior. The executor affects when an already-ready callback runs. Both can increase sample age. Changing Reliable to Best Effort may reduce a network backlog but cannot fix a callback blocked on a mutex.
Carry source timestamps and sequence numbers to the callback. Record receive readiness, callback start and consumption time. This locates the delay before or after middleware delivery and prevents tuning the wrong layer.
Diagnose with trace and controlled load
Trace callback ready, start and finish events, executor thread, group identity and relevant locks. Add one load source at a time: high-rate sensors, service bursts, slow I/O, CPU stress and network loss. Preserve a failing trace before changing the design.
Then move or split one callback group, replace one synchronous dependency or offload one blocking task. Repeat the same workload. Adaptive measurements provide stronger evidence than a complete rewrite that changes several timing layers at once.
Isolate the real-time path from convenience work
Visualization, parameter services, diagnostics, bag recording and model inference can interfere with control if they share executor threads, locks or memory. Use bounded queues and explicit ownership when crossing into a dedicated control thread or executor.
Isolation does not eliminate communication delay. Define the newest valid state, drop policy and watchdog at each boundary. Verify that best-effort work cannot fill a queue consumed by the critical path.
Accept by deadline and data age
Count throughput, but base acceptance on callback wait, execution tail, missed timer periods, sample age at use and recovery after overload. Test the same callback graph after software and hardware changes.
Publish an executor contract with groups, thread pool, scheduling, shared locks and worst-case workload.
- Inventory every callback and hidden dependency.
- Trace wait and execution time separately.
- Map callback groups and shared resources.
- Inject CPU, I/O, service and sensor load.
- Accept by deadlines and data freshness.
Frequently asked questions
Does Multi-Threaded Executor run every callback in parallel?
No. Callback-group rules, ready work, thread count and shared locks determine actual concurrency.
Can every callback stay in the default group?
It can, but the default mutually exclusive group may serialize work and defeat the intended parallelism.
Is a Reentrant group always faster?
No. It only permits overlap; contention, races and extra synchronization can reduce performance or correctness.
Will Best Effort QoS fix executor delay?
Not when the delay comes from callback scheduling, locks or blocking code. Measure both layers.
What should be measured first?
For each callback, measure source data age, ready-to-start wait, execution duration, thread and callback group.
Executor Timing Evidence Boundary
Executor timing depends on ROS distribution, RMW, operating system, hardware, callback code and workload. Measure the deployed graph; executor configuration alone is not a real-time guarantee.