Section 16.5: Sample efficiency and off-policy failure modes

"Old experience is cheaper than new experience, right up until the policy has moved far enough that old experience is a lie."

A Replay Buffer Running Low on Trust

The stakes are physical, not abstract. When SAC or TD3 collects experience on a Franka Panda (a widely used 7-degree-of-freedom robot arm for manipulation research) or an ANYmal quadruped, every transition costs seconds of motor wear and risks a joint-limit strike. Reusing a million MuJoCo or IsaacGym (physics simulators used to generate robot training data faster and more cheaply than real hardware allows) transitions instead of gathering fresh ones turns a three-day training run into a four-hour one. The same reuse hides a catch: it assumes the replay buffer still covers the contact states, recovery torques, and gripper-close moments the current policy now reaches. That assumption is exactly what breaks during sim-to-real deployment.

Sample efficiency is the promise of off-policy RL: learn more from each transition by reusing data. The danger is that reused data may come from a different policy, a different simulator condition, or a different phase of the robot's learning history.

This section maps the failure. It separates useful reuse from distribution mismatch, shows why off-policy correction can become high variance, and names the embodied cases where bootstrapping (using the critic's own current estimate of the next state's value as the learning target instead of a full observed return) turns missing coverage into confident value errors. Concretely, by the end of this section you should be able to: (1) compute and clip an importance ratio for a logged transition, (2) explain why bootstrapping lets an unvisited action acquire a confidently wrong Q-value, and (3) name which of the twin-critic, delayed-actor-update, or coverage-audit mechanisms closes that gap, and why (the twin-critic minimum is explained in detail later in this section). Figure 16.5A contrasts the sample-efficiency curves of DQN, DDPG, TD3, and SAC on a shared manipulation task, showing both the off-policy replay advantage and the Q-overestimation collapse that the rest of this section explains. The process diagram below traces a single replay transition through a coverage check to either a stable critic update or an overestimation failure.

A note on scope: this section deliberately stops at diagnosis and single-agent architectural fixes (twin critics, delayed updates, coverage audits). It does not cover the deeper offline-RL machinery (conservative Q-learning, behavior-regularized policy improvement) needed when the replay buffer is a fixed dataset with no further environment access; that material, and the tools to reason about it, is developed in section 25.3.

Theory

Off-policy learning trains a target policy using data collected by a behavior policy. If the behavior policy \(\mu\) and target policy \(\pi\) differ, an importance ratio can correct expectations in the simplest setting:

$$\rho_t = \frac{\pi(a_t|s_t)}{\mu(a_t|s_t)}$$

Large ratios mean the target policy strongly prefers an action that the behavior policy rarely took. That can reduce bias, but it increases variance. In long-horizon embodied tasks, multiplying many ratios can make estimates unusably noisy, so practical systems clip, truncate, or avoid explicit correction by using value-based bootstrapping.

In embodied AI, this matters because hardware time is finite: a physical robot cannot collect millions of samples to dilute the variance. On a contact-rich grasping benchmark, a pure on-policy SAC variant needed roughly 80,000 fresh environment interactions to reach 70% success. The off-policy version with replay reuse hit the same threshold in under 4,000. That 20-fold reduction takes a real Franka arm from three days of continuous operation down to under four hours. A single high-ratio action, such as a recovery stroke the robot rarely attempted during collection, can dominate gradient updates and push the policy toward behaviors the hardware has never validated. On a real arm, that means joint-limit strikes or dropped objects before the correction signal arrives.

Mechanically, each logged transition contributes to the policy gradient weighted by its ratio. To estimate an expectation under the target policy using data collected under the behavior policy, each sample's contribution is scaled up or down proportionally to how much more or less likely the target policy is to choose that action in that state. Clipping the ratio at a threshold such as 2.0 limits how much any single transition can amplify the gradient, accepting a small bias to prevent a single rare-action mismatch from destabilizing the whole update.

From variance to confident error: the bootstrapping trap

Bootstrapping has its own failure mode. The critic can assign high value to state-action pairs that are out of distribution, because no transition in replay contradicts the estimate. This is confident extrapolation into the void: the model sounds certain exactly where the data is thin. A DDPG arm trained on 50,000 steps can reach 60% grasp success, then collapse to under 10% within 2,000 additional steps once the actor discovers those high-Q unsupported recovery actions, while TD3's twin-critic minimum on the same task holds steady through 200,000 steps without collapse.

Watch this compound with DDPG on a robot arm. Early in training the replay buffer holds mostly random motions, so the critic has no data for high-torque recovery strokes and assigns them inflated Q-values. The deterministic actor follows the critic gradient toward those strokes; the robot attempts them, fails catastrophically, and only then do the failure transitions reach the buffer to correct the critic. TD3 breaks the cycle with two critics and their minimum, so one optimistic critic can no longer pull the actor off the data manifold (the worked example later in this section walks through exactly how that minimum is computed on one transition). In embodied settings the uncorrected loop runs for thousands of steps, because episode resets hide accumulated overestimation from a naive return metric.

Worked Example

The mechanism above stays abstract until you see a single high-ratio action emerge from real numbers, so the worked example below makes the importance ratio that destabilizes a critic update concrete.

Code Fragment 1 computes importance ratios for three logged actions. The third transition is dangerous because the target policy assigns high probability to an action that the behavior policy rarely selected.

# Compute off-policy correction ratios for logged actions.
# Large ratios identify target-policy decisions with weak behavior-policy support.
logged = [
    {"action": "slow_push", "pi": 0.40, "mu": 0.50},
    {"action": "lift", "pi": 0.20, "mu": 0.25},
    {"action": "fast_recovery", "pi": 0.30, "mu": 0.03},
]

for row in logged:
    ratio = row["pi"] / row["mu"]
    clipped = min(ratio, 2.0)
    print(row["action"], f"rho={ratio:.1f}", f"clipped={clipped:.1f}")

The expected output shows mild correction for the first two logged actions and an extreme mismatch for fast_recovery. A ratio of 10.0 means the target policy wants that action far more often than the behavior policy ever demonstrated it, so clipping to 2.0 is a variance-control patch, not proof of support.

In an embodied dataset, that weak evidence is not an abstract statistical problem. It may mean the robot almost never attempted the emergency recovery motion during collection, yet the learned policy now depends on it during deployment.

Practical Recipe

Before reading the checklist below, guess: out of the five sources of off-policy failure in embodied AI (observation shift, action shift, dynamics shift, reward shift, termination shift), which one most commonly causes a policy that looks stable in simulation to collapse within the first 100 real-world steps?

1. Define sample efficiency as grasp success or navigation success per real robot interaction or MuJoCo step, not per gradient update. A TD3 run that needs 200,000 gradient updates from 50,000 simulator steps and 0 real Franka Panda steps is in a completely different data regime from a SAC run using 5,000 real steps collected on a Spot quadruped.
2. Tag every replay batch with the behavior policy checkpoint, the simulator randomization range (friction coefficient, object mass, sensor noise sigma), and the wall-clock timestamp. When reusing Open X-Embodiment or RT-X logs (large public cross-robot datasets that pool recorded trajectories from many different robot platforms and labs) alongside freshly collected transitions, store the source dataset name and robot embodiment as mandatory fields so stale entries can be filtered if the new policy's morphology diverges.
3. Measure policy-data mismatch concretely: for a 7-DOF manipulation policy, compute action coverage in joint-torque space and flag any torque range in the target policy that fewer than 50 replay transitions sampled. For a wheeled robot, flag heading changes steeper than anything in the logged trajectories.
4. Track twin-critic disagreement (\(|\Delta Q| = |Q_{\phi_1} - Q_{\phi_2}|\)) specifically on contact-transition states, because those are where extrapolation error is largest. In IsaacGym or MuJoCo, log \(|\Delta Q|\) at the moment the gripper closes or the foot strikes the ground; spikes there predict real-world collapse before the return curve shows it.
5. Report failure modes by the physical distribution that shifted: proprioceptive observation shift (joint encoder drift on a real Franka versus a simulated one), action shift (torque limits that differ between sim and hardware), dynamics shift (contact stiffness mismatch), reward shift (force-torque sensor miscalibration changing the grasp-success signal), or termination shift (safety stop triggering earlier on the real arm than the simulator reset threshold).

Because that evidence is what makes a sample-efficiency claim interpretable, the claim only holds when the numbers being compared were measured against the same data accounting. Sample efficiency has to be construct-matched. If method A uses 10,000 real robot steps and method B uses 10,000 simulator steps plus a million logged transitions, the comparison is not a single sample-efficiency number. It is a data-regime comparison that must report every source of experience.

A replay buffer full of yesterday's easy transitions is not evidence for today's hard ones; it is a confidence trap waiting to fire. Off-policy failure analysis starts by asking which distribution changed. Observation shift changes what the encoder sees. Action shift changes which controls the critic must evaluate. Dynamics shift changes the consequence of the same action. Reward shift changes which behavior the value function calls good. Termination shift changes which failures are hidden by early episode endings.

A robust off-policy implementation records data provenance, not only return. Code Fragment 2 builds one artifact row that ties a sample-efficiency claim to the exact interaction and replay counts behind it.

1. Count real environment steps, simulator steps, replayed transitions, and gradient updates separately.
2. Store behavior-policy tags for every replay source.
3. Report action-support warnings for target-policy actions with weak coverage.
4. Evaluate the current policy on a fixed perturbation panel, not on replay alone.
5. Keep negative diagnostics in the registry even when only significant wins enter the paper tables.

# Build one sample-efficiency audit record.
# Separating data sources prevents invalid apples-to-oranges comparisons.
from dataclasses import dataclass, asdict

@dataclass
class SampleEfficiencyAudit:
    algorithm: str
    real_steps: int
    sim_steps: int
    replay_samples: int
    gradient_updates: int
    weak_action_support: str
    evaluation_panel: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = SampleEfficiencyAudit(
    algorithm="TD3",
    real_steps=0,
    sim_steps=50000,
    replay_samples=1000000,
    gradient_updates=200000,
    weak_action_support="fast_recovery torque range",
    evaluation_panel="mass_friction_delay_v1",
)
print(record.as_row())

The expected output should be read as an accounting record, not a performance claim. It says this result used only simulator interaction, extremely heavy replay reuse, and still has a named weak-support region, so any sample-efficiency comparison must keep those data sources and support warnings attached to the score.

When an off-policy method fails, do not start by blaming the algorithm name. First identify whether the bad value came from behavior-policy mismatch, missing action support, stale dynamics, reward mislabeling, termination bias, or critic extrapolation. Then rerun one matched perturbation panel with coverage diagnostics enabled.