Section 14.4: Model-free vs. model-based; on- vs. off-policy

"Knowing the dynamics and using old data are separate questions. Confusing the two axes is how you spend twice the compute for half the insight."

Section 14.4

Figure 14.4A previews the core claim of this section: an algorithm's label is only useful once you know what it represents internally and whose data trained its last update.

Two RL papers can use the word "off-policy" to mean completely different things, and a third can call itself "model-based" while never planning a single step ahead: the labels collapse two independent decisions into one word, and the cost of the confusion shows up as a robot that fails for a reason you have misdiagnosed. This section draws on the control material in Chapter 7: Control for AI Practitioners and the environment interfaces in Chapter 10: Environments with Gymnasium and PettingZoo, then prepares the policy-gradient work in Chapter 15: Policy Gradient Methods and PPO. It separates two axes that are often blurred: whether the learner uses an explicit dynamics model, and whether it learns from data generated by the same policy it is improving.

Two axes set the technical contract: model-free vs. model-based learning and on-policy vs. off-policy data. Each axis gets a definition, then discounted occupancy quantifies how two policies see different parts of the same environment. Figure 14.4B lays out the resulting two-axis grid, placing a canonical algorithm family in each of the four cells.

Each cell in Figure 14.4B invites a distinct failure mode: model-free/on-policy pays in sample cost but rarely misdiagnoses a bad gradient; model-free/off-policy risks occupancy mismatch when the replay buffer under-covers the target policy; model-based/on-policy risks compounding model error even though the data itself stays valid; and model-based/off-policy stacks both hazards at once, so its audit checklist must cover model accuracy and replay coverage together. The remainder of this section works through each of these failure modes in turn, then the Algorithm Classification Checklist gives a step-by-step procedure for diagnosing which one is active.

The key question is practical: is the learner improving from its own fresh rollouts, from another policy's data, or from an explicit model of what the world will do next?

Theory

A model-free method estimates a policy, value function or action-value function without explicitly learning \(P(s'\mid s,a)\) for planning. A model-based method learns or uses a transition and reward model defined over the underlying Markov decision process, then chooses actions by planning, imagination rollouts, or dynamic programming. In robot tasks, model-based methods can reduce physical samples, but learned models exhibit compounding error across imagined horizons: a one-step prediction error of 2% can reach 33% after 20 steps, turning a planner's imagined trajectory into fiction.

An on-policy method updates a target policy using data that same policy generated. An off-policy method learns about a target policy \(\pi\) from data a behavior policy \(\mu\) generated. Off-policy learning is attractive because embodied data are expensive. Collecting 1 million transitions on a physical robot arm at a typical 1 Hz decision frequency takes roughly 300 hours of continuous operation, whereas a replay buffer runs the same gradient update in minutes. (Higher-frequency controllers cut wall-clock time but increase wear and safety risk.) In practice this gap separates 50,000 physical episodes to converge from roughly 300, because each stored transition feeds dozens of gradient updates instead of being discarded after a single pass. But you must measure the mismatch between \(\mu\) and \(\pi\) rather than wave it away.

A replay buffer trained by a cautious policy is not evidence for a bold one; it is evidence against it.

For physical robots, this distinction carries a safety consequence beyond sample cost. On-policy rollouts keep the data distribution aligned with the current policy, so gradient estimates stay valid. A robot that deviates from the current policy to collect exploratory data risks hardware damage from out-of-distribution joint velocities or unexpected contact forces. Off-policy data reuse avoids that exposure but introduces a silent correctness hazard. The stored transitions reflect a different controller's intent, possibly an earlier, weaker policy that never attempted the contact-rich behaviors the improved policy now requires.

That mismatch of intent is not just a conceptual worry; it leaves a precise mathematical fingerprint. Mechanically, the hazard manifests in the policy gradient estimator. The standard gradient \(\mathbb{E}_{\pi}[\nabla \log \pi(a|s) Q^\pi(s,a)]\) requires expectations under \(\pi\). When data come from \(\mu\), each term is weighted by the wrong distribution, biasing the gradient unless importance ratios \(\rho = \pi(a|s)/\mu(a|s)\) are applied. In regions where \(\mu\) rarely visits but \(\pi\) would, \(\rho\) can be very large, inflating variance and destabilizing updates.

Measuring the mismatch: discounted occupancy

The discounted occupancy measure makes this mismatch concrete:

$$d_\gamma^\pi(s,a)=(1-\gamma)\sum_{t=0}^{\infty}\gamma^t\Pr_\pi(S_t=s,A_t=a).$$

This quantity says how often policy \(\pi\) visits each state-action pair when near-term visits receive more weight. If a dataset has high occupancy for careful approaches but the target policy wants fast contact-rich grasps, the dataset may contain too little evidence for the target behavior.

Worked Example

Code Fragment 1 computes a finite-horizon approximation to discounted occupancy for two policies in a two-state robot task. The behavior policy is cautious; the target policy is more aggressive about contact.

# Estimate discounted occupancy for behavior and target policies.
# The mismatch shows where off-policy data may undersupport learning.
gamma = 0.9
states = ["approach", "contact"]
actions = ["slow", "fast"]
policies = {
    "behavior_mu": {"approach": {"slow": 0.8, "fast": 0.2}, "contact": {"slow": 0.7, "fast": 0.3}},
    "target_pi": {"approach": {"slow": 0.3, "fast": 0.7}, "contact": {"slow": 0.2, "fast": 0.8}},
}

def next_state(state, action):
    return "contact" if action == "fast" else state

for name, policy in policies.items():
    state_distribution = {"approach": 1.0, "contact": 0.0}
    occupancy = {(s, a): 0.0 for s in states for a in actions}
    for t in range(5):
        weight = (1 - gamma) * (gamma ** t)
        next_distribution = {"approach": 0.0, "contact": 0.0}
        for state, state_prob in state_distribution.items():
            for action, action_prob in policy[state].items():
                prob = state_prob * action_prob
                occupancy[(state, action)] += weight * prob
                next_distribution[next_state(state, action)] += prob
        state_distribution = next_distribution
    print(name, {f"{s}/{a}": round(v, 3) for (s, a), v in occupancy.items()})

The expected output shows the coverage mismatch directly in the occupancy mass. behavior_mu spends most of its discounted weight on cautious approach actions, while target_pi places much more mass on contact/fast, which is exactly where off-policy support becomes weakest.

The example also clarifies model-based learning. If a reliable model predicted the transition from `approach` to `contact`, the learner could plan contact behavior before trying every variant on hardware. If the model predicts contact poorly, planning amplifies the error.

Practical Recipe

1. Fix the observation space to what the hardware actually delivers: a Franka Panda arm with a wrist-mounted RealSense D435 gives 640x480 RGB-D at 30 Hz; anything faster is dropped frames, not real data. Write the action space as torque or velocity limits the real actuators enforce, not theoretical maximums.
2. Build a SAC baseline in MuJoCo or Isaac Lab with the default hyperparameters (replay buffer 1M, batch size 256, learning rate 3e-4) before adding model rollouts. Debugging a failed Dreamer world model is much harder than debugging a failed Q-function because model error and policy error compound together in the imagined rollouts.
3. Add MBPO-style short model rollouts (one to five steps) only after the model-free SAC baseline achieves above-chance performance in simulation. This confirms the reward signal is recoverable before committing to the extra complexity of a learned dynamics model.
4. Record failures with physical provenance: note whether the error occurred in simulation (Isaac Lab, MuJoCo, PyBullet), on hardware during teleoperation, or during autonomous rollout. A grasp failure in simulation that is absent on the Franka Panda almost always points to a sim-to-real contact-stiffness mismatch rather than a policy bug.
5. Before the sim-to-real transfer step, perturb the simulator's contact parameters (friction coefficient, object mass, actuator damping) by plus or minus 20% and recheck success rate. If performance drops below 50% under this domain randomization (deliberately varying simulator physics parameters across training so the policy does not overfit to one exact, and likely wrong, physics setting), the model-based or off-policy data assumptions are too brittle for physical deployment.

The two axes in this section answer different failure questions. If a model-based controller fails, inspect transition prediction, reward prediction, planning horizon, and model rollout error. If an off-policy learner fails, inspect whether the replay buffer actually covers the target policy's state-action occupancy.

For embodied agents, the strongest designs mix categories: a learned model for short-horizon contact prediction, a model-free value function for policy improvement, and off-policy demonstration data. The label matters less than the evidence that each source supports the update it feeds.

A robust implementation logs data provenance. Every transition should record the behavior policy, policy version, simulator or hardware source, reward version, and any model-generated rollout flag. Without those fields, an off-policy experiment cannot prove which distribution produced the evidence.

1. State whether the update is on-policy or off-policy.
2. State whether planning uses a learned, analytic, or simulator model.
3. Estimate occupancy coverage for important state-action regions.
4. Evaluate model rollout error separately from policy return.
5. Compare algorithms only on one environment panel and one metric definition.

When a model-based method fails, run one-step prediction checks before judging the planner. When an off-policy method fails, inspect the occupancy mismatch before tuning the loss. These two diagnostics isolate different root causes.