Section 25.2: Distribution shift and extrapolation error

"The value function was accurate everywhere the data reached. Then the policy stepped one state beyond the edge."

A Q-Value With Regrets

Why Offline RL Is Different

Offline RL starts from a static dataset and must make the behavior policy, support envelope, reward labels, and candidate-policy update explicit before any robot rollout is trusted.

The behavior policy is whatever controller actually collected the dataset: a human teleoperator, a scripted motion planner, or a prior RL agent. In embodied AI this choice is not abstract. A physical robot cannot recover from a bad grasp mid-episode the way a simulated agent can reset; the behavior policy's kinematic habits define the boundary of states the robot has survived, and any learned policy that ventures beyond that boundary enters territory with no demonstrated recovery path. A narrow behavior policy (one operator, one object pose) leaves large gaps in joint-space that the critic fills with guesses.

What happens when the robot encounters a wrist angle no human operator ever used during data collection? The critic has no evidence of failure there, so it fills the gap with optimism, and the policy follows.

How the support gap becomes an overestimate

The behavior policy defines the support set through the density of its logged state-action visits. During Bellman backup, the critic trains only on transitions the dataset contains. For state-action pairs with zero logged visits, the neural network interpolates from nearby samples and extrapolates a value that no real reward signal has ever tested. Neural networks stay overconfident in low-density regions, so those extrapolated values run high. Gradient ascent on the policy objective then steers the robot toward those inflated, unsupported actions. Figure 25.2A shows this split directly: actions near the behavior distribution carry accurate Q-estimates, while out-of-distribution actions receive overestimated values that mislead the policy update.

Distribution shift appears at two levels. State shift means deployment observations differ from logged observations, for example a warehouse robot facing a bin lighting condition or clutter arrangement it never saw during data collection. Action shift means the learned policy chooses actions the behavior policy rarely tried, for example a wrist angle no operator used. Extrapolation error is the critic's confident lie beyond the data edge. In illustrative offline RL evaluations (as of 2023-2024), policies trained without support constraints typically reach Q-values an order of magnitude above the true return for out-of-distribution actions; the exact magnitude varies by task and dataset, but the direction of the bias, toward overestimation, is consistent because nothing in training penalizes an unvisited action. Those overestimates are typically large enough to drive the robot directly into the failure region. A policy that trusts its own critic past the edge of the data is not acting from knowledge; it is acting from silence mistaken for permission.

Formal Contract

An unconstrained critic lies beyond the data edge, and the objective below makes that boundary precise: it turns "stay inside support" from a slogan into a quantity training can optimize.

The baseline objective is useful only after the data distribution and robot action scale are fixed; otherwise expected return can reward unsupported commands.

$$J(\pi) = \mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^{T} \gamma^t r(s_t,a_t)\right].$$

The practical objective needs a pessimism or support term because training cannot ask the real robot whether a novel action is safe.

$$\epsilon_{\mathrm{extra}}(s,a) = \left|Q_\theta(s,a) - Q^{\pi}(s,a)\right|, \quad a \notin \mathrm{supp}(\beta(\cdot\mid s)).$$

The pessimism term should expose unsupported gripper poses, contact modes, saturation regions, missing viewpoints, or reset states rather than hiding them in one value estimate. Figure 25.2.B traces the full pipeline that enforces this, from logged data through the behavior-support and pessimistic-critic stages to a same-panel evaluation, separating what the critic may believe from what the reported score means.

Worked Numeric Trace

The pessimism term in that objective stays abstract until we attach numbers to it, so the following trace measures support for concrete candidate actions and shows exactly which one the critic would be guessing about.

Code Fragment 1 for Distribution shift and extrapolation error compares candidate actions with dataset support and applies pessimism only after the support metric and robot action scale are explicit.

# Measure a simple action-support distance for candidate actions.
# Large distances identify actions where a learned critic is extrapolating.
import numpy as np

behavior_actions = np.array([-0.4, -0.2, 0.0, 0.2, 0.4])
policy_actions = np.array([-0.35, 0.15, 0.9])
distances = [np.min(np.abs(behavior_actions - action)) for action in policy_actions]
for action, distance in zip(policy_actions, distances):
    label = "inside support" if distance <= 0.1 else "extrapolation risk"
    print(f"candidate={action:+.2f} nearest_logged_distance={distance:.2f} {label}")

More data collection and larger critic networks do not fix extrapolation error on their own: a bigger network still has no reward signal for an action nobody logged, and more demonstrations from the same operator still leave the same joint-space gaps. Pessimism is the only principled fix because it is the only mechanism that acts directly on the gap itself, penalizing or discarding value estimates precisely where the dataset offers no evidence, rather than hoping the gap will shrink on its own.

Practical Recipe

Recognizing where extrapolation error strikes hardest is only useful if it changes what you do before training, so the steps below turn that awareness into an ordered workflow.

1. Start with behavior cloning (BC) and report it. If BC solves the task, offline RL must justify its extra complexity.
2. Write the dataset manifest: robot body, sensors, action units, operator source, split rule, reset distribution, episode horizon, reward source, and license.
3. Audit action support before training. Plot nearest-neighbor distances or behavior log probabilities for every proposed policy action.
4. Train CQL, IQL, or behavior-regularized actor-critic only after the support audit exists.
5. Report same-config evaluation: one task panel, one split, one seed policy, one artifact, and one failure taxonomy.