A robot reward model converts human judgments, demonstrations or video progress into a score for states, actions or trajectory segments. It is useful when a hand-written reward misses qualities such as smoothness, care, acceptable recovery or instruction compliance, but the learned score remains a proxy for the feedback process.
The core risk is optimization outside the examples people reviewed. A model can predict held-out comparisons well and still reward camera shortcuts, repetition, long trajectories or unsafe behavior that correlates with preferred clips in the labeling data.
Use this guide with the human-intervention data guide and robot VLA evaluation guide. Keep task success, safety constraints and independent human judgment outside the reward model.
Define exactly what the reward model predicts
Choose whether the output represents pairwise preference, scalar quality, goal progress, success likelihood or a future-event estimate. State the observation window, action context, prediction horizon and evaluator population behind that label.
Do not combine comfort, speed, safety and task completion into one undocumented score. Preserve dimensions separately when operators may trade them differently across deployments.
Choose a feedback format that matches the question
Pairwise comparisons ask which of two segments is better and often reduce scale differences among raters. Ratings express magnitude but require anchors. Progress labels can provide temporal density yet assume the task moves monotonically toward a goal.
Success labels answer a narrower endpoint question. Select the format from the operational decision and record ties, both-bad cases and cannot-judge outcomes instead of forcing arbitrary winners.
| Feedback | Answers | Strength | Primary risk |
|---|---|---|---|
| Pairwise | Which is preferred | Simple relative judgment | No absolute quality |
| Rating | How good | Magnitude available | Rater scale drift |
| Progress | Closer to goal | Dense temporal signal | Non-monotonic tasks |
| Success | Goal achieved | Clear endpoint | Sparse and narrow |
| Correction | What should change | Actionable context | Teacher and control bias |
Write an observable labeling rubric
Define task goal, prohibited behavior, smoothness or efficiency criteria, tie conditions and unobservable factors. Include positive, negative and boundary examples from the target robot rather than relying on words such as good or safe.
Video raters cannot infer force, torque, delay, off-camera collisions or whether a controller limit fired unless the interface exposes that evidence. Mark missing information rather than letting appearance become a substitute.
Preserve annotator identity and disagreement
Measure inter-rater agreement by task and failure class, but do not erase legitimate preference differences. Operators, safety reviewers and end users may value different behavior because their responsibilities and information differ.
Store rater training, rubric version, timestamp and confidence with each judgment. Re-adjudicate systematic disagreements and report subgroup behavior rather than collapsing every label into a majority vote.
Sample comparisons near useful boundaries
Random pairs can be obvious and teach little. Active sampling can prioritize close scores, policy disagreements, novel states and high-consequence failures, while coverage floors preserve common operation and hard negatives.
The human-preference RL study trains from pairwise trajectory-segment preferences and reports results in its evaluated simulated locomotion and game settings. Reproduce the feedback loop for the target robot rather than transferring its label budget as a universal value.
Split by trajectory and collection context
Adjacent clips from one episode share scene, operator, object and outcome. If derivatives cross train and test, the reward model may identify a recording session instead of learning behavior quality.
Group splits by source episode and, when the claim requires it, by object, site, operator or policy version. Keep augmented, cropped and overlapping windows with their parent trajectory.
Use video rewards within embodiment limits
Human video can teach temporal progress or object relations without robot actions, but morphology, viewpoint and dynamics differ. A feature that recognizes a human hand approaching an object may not score a robot gripper correctly.
The HOLD paper studies reward learning from human manipulation video and transfer in its stated simulation experiments. Treat cross-embodiment transfer as a hypothesis and validate robot-specific observations, failures and contact outcomes.
Control trajectory length and reward aggregation
Summing per-frame scores can favor long episodes or repeated visually rewarding motion. Averaging can hide a brief severe failure, while terminal-only scoring provides little recovery guidance.
Define discount, segment overlap, terminal handling and failure penalties before policy optimization. Test synthetic trajectories that repeat, stall, reverse progress or end early to reveal aggregation exploits.
Search deliberately for reward hacking
Optimize the model in simulation or a protected environment and inspect the highest-scoring trajectories, not only policy success. Generate counterexamples that change camera angle, hide objects, repeat motion or exploit the episode boundary while leaving the real goal unsatisfied.
The ReQueST paper explores synthesized hypothetical behavior to find informative and unsafe cases in its evaluated environments. The approach motivates adversarial queries, but does not establish safety for an unrelated physical robot.
Calibrate scores only to declared outcomes
A pairwise preference logit is not automatically a probability of success. If operations need a threshold, calibrate the relevant output against an observable held-out event and report reliability, base rate and uncertainty by domain.
Recalibrate after changing the policy because optimization alters the trajectory distribution. A model accurate on behavior-policy data may be unreliable on the new policy it helped create.
Evaluate prediction, optimization and people separately
First measure held-out comparison accuracy, ranking and calibration. Second measure how optimizing the score changes task outcome and failure modes. Third ask independent evaluators to judge resulting trajectories without seeing reward scores.
Include task success, safety events, intervention, time, energy and subgroup preference. A higher learned reward with worse independent outcomes fails the deployment objective.
| Layer | Measure | Warning | Response |
|---|---|---|---|
| Labels | Agreement and unknown rate | Rubric ambiguity | Revise guidance |
| Prediction | Held-out ranking | Session shortcut | Resplit |
| Optimization | Score versus outcome | Reward hacking | Add challenges |
| People | Blind judgment | Score disagrees | Reject proxy |
| Robot | Success and safety | Offline-only gain | Do not deploy |
Release a reward-data-policy bundle
Version rubric, annotator protocol, feedback dataset, splits, preprocessing, reward checkpoint, aggregation, calibration, optimized policy and challenge set. Preserve exact examples that triggered model or human disagreement.
Close release review with the following checks.
- Name the judgment and prediction horizon.
- Preserve ties, unknowns and annotator disagreement.
- Split by source trajectory and context.
- Challenge the optimized policy for reward exploits.
- Compare reward, real success, safety and blind human judgment.
Frequently asked questions
Is a reward model the same as a success detector?
No. A success detector predicts a defined outcome, while a reward model may represent preference, progress or quality over trajectories.
How many human labels are enough?
There is no universal count; use coverage, agreement, learning curves and policy challenge results for the target distribution.
Can a VLM score be used directly as reward?
Only after task-specific validation; generic visual-language scores can miss contact, force, timing and safety.
Does a better reward model replace safety constraints?
No. Independent runtime constraints and risk controls remain necessary.
What is the fastest reward-hacking check?
Optimize the score in a sandbox and inspect the highest-scoring failures, repetitions and camera or episode-boundary exploits.
Preference-Signal and Goal-Alignment Boundary
A robot reward model is a learned proxy for a documented feedback process. Deployment requires independent task, safety and human-judgment evidence after optimization changes the behavior distribution.