Section 25.5: Evaluating offline policies rigorously

"A policy that scores well on the logged data and fails in the world has taught you one thing: the log was not the world."

A Disappointed Evaluator

Why Offline RL Is Different

A policy can earn a flawless 94% on every logged trajectory you own and still drive a robot arm into a cabinet the first hour it runs, because the score measured what the data already contained and the world contained something else. That single gap is what this section exists to close. Throughout, the behavior policy (often written \(\beta\)) is whatever controller actually generated the logged data, and the target policy \(\pi\) is the new policy we want to score; off-policy evaluation means estimating how \(\pi\) would perform using only data collected by \(\beta\). Figure 25.5A previews the core idea: a doubly-robust off-policy estimator (one that blends a learned value model with importance-weighted corrections) sits between a naive average of logged returns and the true on-policy performance, cutting variance by using the behavior policy as a control variate (a quantity whose known behavior is subtracted out to reduce the noise in an estimate, here the behavior policy's own logged returns). By the end of this section you should be able to audit a dataset's support envelope, decide when an off-policy estimate is trustworthy, and assemble an evaluation panel that holds a policy accountable for the states it will actually face.

Imagine a robot arm trained entirely on 50,000 logged pick-and-place episodes. At evaluation time, the learned policy proposes a wrist angle 15 degrees outside anything in the log. No simulator exists, the real arm cannot be queried during training, and the value function has never seen that configuration. The number it returns for that state-action pair is an extrapolation, not a measurement. Offline evaluation exists precisely to catch this gap before it becomes a hardware incident or a misleading benchmark number. A score on logged data is a record of what the behavior policy did; it is not a promise about what your policy will do.

Offline RL starts from a static dataset and must make the behavior policy, support envelope, reward labels, and candidate-policy update explicit before any robot rollout is trusted.

Coverage, Not Algorithm, Sets the Ceiling

Offline evaluation is hard because the learned policy can choose actions the logged data never contains. The D4RL "random" locomotion datasets (Fu et al., 2020) make the point sharply. Policies trained on sparse coverage reached only 2 to 5 normalized return (task return rescaled so 0 marks a random policy and 100 marks a trained expert, which lets scores be compared across tasks with different raw reward scales). The same algorithms on "medium-expert" data reached 90 to 110. One codebase, a tenfold gap, and the only difference was whether the data covered the good states. Coverage dominates algorithm choice so completely that in the robomimic manipulation study, behavior cloning on 300 proficient-human episodes beat CQL trained on 50,000 mixed episodes. More data made things worse here: the extra 49,700 episodes were suboptimal, and they diluted the support for the actions that actually mattered. A reliable evaluation plan therefore combines behavior cloning baselines, off-policy estimates where assumptions hold, simulator or real rollouts on held-out task panels, and qualitative failure labels.

Formal Contract

Coverage dominates the score, so the objective has to make explicit where support enters. The baseline objective is useful only after the data distribution and robot action scale are fixed; otherwise expected return can reward unsupported commands.

$$J(\pi) = \mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^{T} \gamma^t r(s_t,a_t)\right].$$

The practical objective needs a pessimism or support term because training cannot ask the real robot whether a novel action is safe.

$$\widehat{V}_{\mathrm{IS}}(\pi)=\frac{1}{N}\sum_{i=1}^{N}\left(\prod_t \frac{\pi(a_t^i\mid s_t^i)}{\beta(a_t^i\mid s_t^i)}\right)R(\tau_i).$$

The pessimism term should expose unsupported gripper poses, contact modes, saturation regions, missing viewpoints, or reset states rather than hiding them in one value estimate. Figure 25.5.B lays out the four-stage pipeline that keeps this pessimism honest: the dataset, the behavior-support check, the pessimistic critic, and the policy that finally acts, all scored on one shared evaluation panel. The critic in that pipeline is trained on Bellman targets, current reward plus the discounted value of the next logged state; the Algorithm box later in this section spells out that training step in full, so treat the diagram here as a preview of a mechanism defined precisely below.

Worked Numeric Trace

The importance-sampling estimator above looks harmless on the page. Run three trajectories through it and one weight takes over, which exposes its fragility. Code Fragment 1 for Evaluating offline policies rigorously compares candidate actions with dataset support and applies pessimism only after the support metric and robot action scale are explicit.

# Show why importance sampling can become unstable offline.
# The product of action-probability ratios can make one trajectory dominate.
import numpy as np

returns = np.array([1.0, 0.0, 1.0])
ratios = np.array([
    [1.1, 0.9, 1.0],
    [0.8, 1.2, 0.7],
    [2.5, 2.0, 1.8],
])
trajectory_weights = ratios.prod(axis=1)
estimate = np.mean(trajectory_weights * returns)
for idx, weight in enumerate(trajectory_weights):
    print(f"trajectory={idx} weight={weight:.2f} return={returns[idx]:.1f}")
print(f"ordinary_is_estimate={estimate:.2f}")

Practical Recipe

1. Start with behavior cloning (BC) and report it. If BC solves the task, offline RL must justify its extra complexity.
2. Write the dataset manifest: robot body, sensors, action units, operator source, split rule, reset distribution, episode horizon, reward source, and license.
3. Audit action support before training. Plot nearest-neighbor distances or behavior log probabilities for every proposed policy action.
4. Train CQL, IQL, or behavior-regularized actor-critic only after the support audit exists.
5. Report same-config evaluation: one task panel, one split, one seed policy, one artifact, and one failure taxonomy.