"Two robots that both finish the task are not equal; the one that zigzags and burns twice the energy is telling you where it will break first."
An Evaluation Methodologist
Two warehouse robots both complete the pick-and-place task every single time. One glides in a smooth arc and draws 380 joules; the other zigzags, burns 610 joules, and finishes four seconds later. A success-rate-only leaderboard calls them identical. For real deployment, that tie is a lie. As embodied systems move from research floors to hospitals, construction sites, and homes, the gap between "task done" and "task done well" becomes a safety and economic decision. You will build a four-component metric vector, learn to compute path efficiency and energy proxies from raw episode logs, and see exactly when scalar rankings mislead and when they are safe to trust.
This section assumes familiarity with control-oriented cost functions introduced in section 7.3, particularly how trajectory length and energy enter a control objective. The metric vector defined here is extended in section 52.4 to cover robustness across perturbation families, and returns in section 52.5 when aggregating scores across diverse task distributions. Readers who already work with SPL (Success weighted by normalized Path Length) style metrics and energy proxies can skim to the Worked Example.
Why This Matters
Success rate, path efficiency, time and energy cost matters because evaluation choices rewrite the scientific claim. If the metric drops time, energy, or safety terms that the deployment team cares about, the benchmark no longer matches the real decision. The scorecard in Figure 52.2.1 makes this concrete: two equally successful policies can sit far apart on path quality and power cost.
One useful representation is the episode score vector $$m_i = [s_i,\; \rho_i,\; t_i,\; e_i],$$ with \(s_i\) for success, \(\rho_i = d^*_i / d_i\) for path efficiency, \(t_i\) for completion time, and \(e_i\) for energy. A scalar summary is acceptable only after the vector is logged and inspectable. Figure 52.2.2 traces how raw episode log data feeds these four components into the inspectable vector and only optionally collapses into a scalar rank.
Path efficiency matters physically because every extra meter of travel costs motor current, heats actuators, and consumes battery capacity that cannot be recovered. A robot with low path efficiency reaches thermal or charge limits sooner, shortening operational windows and forcing recharge cycles that cascade into mission delays. On legged platforms, excess travel also accumulates mechanical wear on joints at a rate proportional to step count, not task count.
Path inefficiency accumulates when a controller issues small corrective actions that cancel each other. A heading error triggers a rightward turn, overcorrection triggers a leftward turn, and the robot traces a sinusoid around the intended straight line. Each cycle adds distance to \(d_i\) while \(d^*_i\) stays fixed, so \(\rho_i\) falls below 1. Replanning on perception updates compounds this. Each new plan starts from a slightly off-track position and adds detour segments to the total traversed distance.
Path efficiency and energy cost should remain visible as separate columns even if the benchmark publishes one scalar rank. Otherwise teams optimize the weighted sum while reviewers lose the tradeoff surface.
- Compute shortest-feasible reference distance or nominal task budget before running candidate methods.
- Record actual path length, action count, elapsed time, and energy or torque proxy for every episode.
- Normalize each quantity against a baseline or physical reference when cross-task aggregation is needed.
- Report both the raw vector and any scalarized utility.
- Audit whether the scalar ranking changes under small weight perturbations.
Worked Example
A quadruped navigation policy may tie the baseline on success rate but use 30 percent more turning and 18 percent more battery because it oscillates in narrow passages. A scalar success metric would never show the regression; this is called the efficiency-invisible success trap, and it surfaces only when the full metric vector is logged.
episodes = [
{"success": 1, "optimal_path_m": 8.0, "actual_path_m": 9.0, "time_s": 21.5, "energy_j": 440.0},
{"success": 1, "optimal_path_m": 8.0, "actual_path_m": 11.6, "time_s": 28.2, "energy_j": 590.0},
]
summary = []
for ep in episodes:
path_eff = ep["optimal_path_m"] / ep["actual_path_m"]
summary.append({
"success": ep["success"],
"path_efficiency": round(path_eff, 3),
"time_s": ep["time_s"],
"energy_j": ep["energy_j"],
})
print(summary)[{'success': 1, 'path_efficiency': 0.889, 'time_s': 21.5, 'energy_j': 440.0}, {'success': 1, 'path_efficiency': 0.69, 'time_s': 28.2, 'energy_j': 590.0}]optimal_path_m by actual_path_m to derive path_efficiency, then emits it beside success, time, and energy so the two successful runs stay distinguishable.Expected output: Both episodes succeed, but the second is visibly less efficient, slower, and more expensive. That is the point: success rate should not erase operational cost.
Energy itself is typically not measured directly from a joule-meter on small platforms; in practice it is estimated as a proxy, most commonly the time integral of commanded torque times joint angular velocity (summed across joints and timesteps), or read from a platform's onboard power telemetry when available. The code fragment above treats energy_j as already computed for that reason: obtaining it is a separate instrumentation step, covered in the Library Shortcut and Practical Example callouts below, not part of the per-episode aggregation loop itself.
SPL is the standard benchmark scalarization of the \(s_i\) and \(\rho_i\) components already defined in the metric vector above: it multiplies the success indicator by a path-efficiency ratio, so a failed episode contributes zero regardless of how efficient its path was.
Trace SPL with three episodes, where for each one \(\text{SPL}_i = s_i \cdot \dfrac{d^*_i}{\max(d^*_i,\, d_i)}\) and the benchmark score is the mean over episodes.
- Episode 1: \(s_1 = 1\), \(d^*_1 = 10.0\) m, \(d_1 = 12.5\) m. Term \(= 1 \cdot \frac{10.0}{\max(10.0, 12.5)} = \frac{10.0}{12.5} = 0.800\).
- Episode 2: \(s_2 = 1\), \(d^*_2 = 6.0\) m, \(d_2 = 6.0\) m (a perfectly direct run). Term \(= 1 \cdot \frac{6.0}{6.0} = 1.000\).
- Episode 3: \(s_3 = 0\) (the robot timed out). The success gate zeroes the term regardless of distance: \(0 \cdot \frac{8.0}{\max(8.0, 9.0)} = 0.000\).
- Aggregate: mean \(= \frac{0.800 + 1.000 + 0.000}{3} = \frac{1.800}{3} = 0.600\).
Notice that the raw success rate here is \(\frac{2}{3} \approx 0.667\), yet SPL is \(0.600\): the path penalty on Episode 1 pulls the score below pure success, and a single failure costs a full point regardless of how close the robot got.
Once the vector definition is stable, use a benchmark dataframe plus MLflow or Weights and Biases tables to compute per-task and aggregate summaries automatically. The important part is that the raw vector remains accessible.
Success, path efficiency, time, and energy belong in one paired episode table. Pandas computes per-route deltas, SciPy bootstraps confidence intervals, DVC freezes the route panel, MLflow or Weights and Biases keeps run lineage, and ROS 2 bags expose whether a faster route came from better planning or more aggressive control.
With the tooling in place to log the full vector, the payoff shows up most clearly on real hardware, where a hidden efficiency gap carries a concrete operational cost. Consider a specific case. The ANYmal quadruped evaluated on stair-climbing tasks at ETH Zurich (Kumar et al., 2021; Miki et al., 2022) typically achieves near-identical success rates under two controllers. Yet in practice the energy-efficient variant tends to draw on the order of tens of watts less on average (illustratively, roughly 40 W in reported runs) because it avoids lateral trunk oscillation during step clearance. That gap is invisible on a success-only leaderboard but directly affects battery life on a 90-minute patrol mission. The RoboTHOR ObjectNav benchmark (Deitke et al., 2020) reports success rate and SPL (Success weighted by Path Length) as separate columns for a related reason. Teams that tune for SPL tend to produce qualitatively different navigation behaviors from teams that tune for raw success alone, and the two rankings often diverge. This happens because a policy free to ignore path cost can explore aggressively, backtrack without penalty, and reach the goal by exhaustive coverage rather than directed navigation, so in some reported settings the policy with the highest success rate ends up with the worst path efficiency.
Scalarization (the act of combining several separate metrics, such as success, time, and energy, into one weighted number) is a policy decision, not a law of nature. Different applications weight speed, smoothness, and energy differently, so good benchmark design publishes the underlying vector and documents any chosen weights. A success rate without an efficiency column is a scoreboard with half the columns missing: it tells you who crossed the finish line, not how much it cost to get there.
Collapsing a metric vector into one score is like judging a road trip by a single "trip score" that averages fuel economy, travel time, and passenger comfort into one number. Two routes can tie at 7.2 out of 10 while one burns twice the fuel in half the time and the other cruises slowly on an empty tank with everyone asleep. The tie looks clean on paper, but the moment your priorities shift, say the tank is almost empty, the two routes are not interchangeable at all. Keeping the separate columns is the only way to make that choice honestly when conditions change.
The reproducible artifact is a run ledger keyed on the same seed and map: path length, elapsed time, watt-hours or battery drop, replan count, controller saturation (where a commanded actuator effort hits its physical limit and can no longer track the reference), and terminal success. Compare any subset without the rest and the leaderboard misleads.
That ledger is worth the storage cost precisely because the columns it preserves are the ones teams most often regret discarding.
The cost of overcompression
When teams overcompress these metrics, they often rediscover hidden regressions late in deployment, especially thermal issues, battery drain, and route oscillations that were invisible in success-only dashboards.
Cross-References
This section links back to Chapter 7 on control-oriented costs and tradeoffs and forward to Section 52.4 on robustness metrics, where the same vector idea is extended across perturbation families.
Instrument one navigation or manipulation task with path efficiency and energy proxy. Then perturb controller gains and verify whether the tradeoff surface changes even when success stays flat.
A high success rate is often assumed to prove an embodied system is ready for deployment. That assumption appears harmless in simulation, but physical deployment exposes the gap. A robot can succeed at every task while consuming three times the expected energy, taking twice as long, or grinding its joints through oscillatory correction loops. Success rate is a threshold gate, not a verdict. Passing it means the robot reached the goal. It says nothing about whether the journey was safe, efficient, or sustainable. Always inspect the full metric vector before any deployment decision. Two systems with identical success rates can differ by factors of two or more on energy, time, and mechanical wear.
Do not compute path efficiency against a reference planner that quietly violates the robot's kinematic or dynamic limits. The denominator must be a feasible reference, not a fantasy route.
Use ROS 2's nav2_smac_planner (specifically the SE2 lattice variant, where SE2 is the space of planar position and heading and the planner searches over precomputed motion primitives that respect turning limits) to generate your reference path rather than raw Euclidean distance: the lattice planner respects the robot's minimum turning radius and obstacle inflation, so the denominator \(d^*_i\) is a genuinely feasible shortest path. Without this, any robot that must curve around a tight obstacle will produce path efficiency values above 1.0, making a suboptimal trajectory look optimal. Set the minimum_turning_radius parameter to match your platform's kinematic model before computing reference distances, and regenerate reference paths whenever the costmap or inflation radius changes.
For autonomous driving, this metric vector can include route completion, jerk, delay, and energy. For drones, swap path efficiency for trajectory deviation and power draw. The structure stays the same while the physics changes.
The Habitat Challenge (run by Meta AI on the AI Habitat simulator) ranks PointNav and ObjectNav agents by SPL (Success weighted by Path Length) rather than raw success, so an agent that reaches the goal by exhaustive wall-following scores far below one that walks nearly straight there. This single design choice reshaped the field: published navigation policies began reporting SPL as the headline number, and entries that tuned only for success stopped winning. Keeping path length in the metric turned "did it arrive" into "did it arrive efficiently."
Direction 1: Whole-deployment energy accounting. Research groups are moving from per-episode energy proxies toward full operational-cycle budgets that include computation, communication, and actuation jointly. The BEHAVIOR-1K benchmark (Li et al., 2024, Stanford) introduces normalized energy and time jointly as first-class evaluation axes across 1,000 household tasks, showing that GPU inference power can rival motor draw for manipulation policies running on-board. This forces evaluators to instrument both the compute node and the motor bus simultaneously, a practice not yet standard in most labs.
Direction 2: Adaptive Pareto-front benchmarking. Rather than reporting a fixed weighted scalar, recent work tracks the full efficiency-safety Pareto surface (the set of outcomes where improving one metric, such as energy, is impossible without making another metric, such as time, worse) across operating conditions. The RoboHive evaluation suite (Rajeswaran et al. lab, 2024) exposes condition-conditioned Pareto plots where path efficiency and energy trade off differently on slippery versus rigid floors, revealing that a single scalar ranking changes its ordering depending on surface friction. This makes cross-paper comparisons using only one number structurally invalid.
Checkpoint
So far: whole-deployment energy accounting, adaptive Pareto-front benchmarking, and the observation that rankings shift with operating conditions all point to the same underlying issue, a single scalar summary is fragile in ways the raw metric vector is not.
Direction 3: Sim-to-real transfer of efficiency metrics. Sim-to-real gaps are documented for task success, but 2025 work from the Robotics at Google and ETH Zurich groups shows that path efficiency and energy metrics have larger transfer gaps than success rate, because simulation underestimates friction, slippage, and motor heating. The paper "Evaluating Embodied Agents Beyond Task Success" (Yokoyama et al., 2024, Georgia Tech / Meta AI) systematically measures how SPL, path length, and energy rank orders change between Habitat simulation and physical deployment, finding rank correlation as low as 0.41 for energy on rough terrain.
Open problem: No standard protocol exists for certifying that a simulated efficiency metric is a valid proxy for its physical counterpart. A tractable thesis contribution would be a calibration procedure, analogous to camera calibration, that maps simulated path efficiency and energy distributions onto real hardware distributions using a small transfer dataset, then provides a confidence interval on whether a simulated ranking is preserved in deployment. Current benchmarks provide no such interval, so a paper claiming improved efficiency in simulation carries unknown real-world validity.
Can you explain why a benchmark should publish both the metric vector and the scalar rank? If not, you are still trusting the aggregation more than the evidence.
Success rate is the entry ticket, not the whole evaluation. Path efficiency, time, and energy reveal whether the robot succeeded in a deployable way.
Choose one embodied task and design a feasible reference path or action budget. Then define a vector metric and one scalar utility, and explain which tradeoffs the scalar hides.
Two robots can tie on success rate while one spirals through the room like a caffeinated roomba and the other walks the shortest path. The leaderboard will not tell you which one is deploying Monday.
Section References
Paden, B. et al. "A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles." (2016). https://arxiv.org/abs/1604.07446
A useful reference for trajectory quality and control-oriented evaluation quantities.
Official robot benchmarking and fleet telemetry documentation for the platform under study.
Use platform-native energy, thermal, or power interfaces rather than guessed proxies when possible.
Project Ideas
Beginner (weekend): Path efficiency logger in Gymnasium. Build a wrapper around a Gymnasium navigation environment (e.g., MiniGrid-Empty) that records optimal path length via BFS (Breadth-First Search) and actual path length per episode, then prints the metric vector at the end of each rollout. The key challenge is computing a valid shortest-path reference inside the same discrete grid the agent uses, so the denominator is feasible rather than Euclidean.
Intermediate (1-2 weeks): Multi-metric leaderboard for a PyBullet manipulation task. Instrument a PyBullet pick-and-place scene to log success, joint-torque integral as an energy proxy, episode time, and end-effector path length for at least two controllers (a scripted baseline and a learned policy from LeRobot). The key challenge is aligning timestep-level torque samples with episode boundaries and computing a fair reference trajectory using a straight-line IK path so that path efficiency stays below 1.0 for physically constrained motions.
Intermediate (1-2 weeks): Energy-aware navigation benchmark in Isaac Lab. Run a wheeled robot in an Isaac Lab maze scene under two navigation policies and record motor power draw (torque times angular velocity) alongside SPL. The key challenge is correctly summing instantaneous power over variable-length episodes and normalizing by episode difficulty so energy comparisons remain valid across routes of different lengths.
Section 52.3 adds hard constraints to this vector view by asking whether the robot moved efficiently and remained inside the allowed envelope.