Section 16.1: Q-learning; deep Q-networks

"Tell me what every move is worth from here, and I will never need to know the destination in advance."

A Deep Q-Network Seeing Its First Frame

A robot can fumble a grasp under noisy exploration, never once execute the clean version, and still learn precisely what that clean grasp was worth. That single counterintuitive ability is what makes Q-learning off-policy: it recovers the value of the best possible behavior from data generated by a clumsier, more exploratory one. As Figure 16.1A shows, a deep Q-network turns that idea into a single forward pass that maps an observation to one value per action, with the argmax (the action whose value is highest, written \(\arg\max_a Q(s,a)\)) selecting the action to deploy. Reliable off-policy learning depends on fixing four things before any comparison with policy-gradient methods: replay semantics, the environment API, target computation, and GPU-scale batching.

Q-learning answers a specific embodied-agent problem: the robot often cannot wait for a full episode to learn whether a push, turn, or grasp was useful. It needs a local training signal after each transition, even when the reward is delayed and the next state is only partially observed.

That signal builds in three stages: the tabular Bellman update, the deep Q-network version that predicts action values from observations, and the stabilizers that make DQN usable when pixels, proprioception, and contact events keep changing the data distribution. A Q value that cannot survive noisy sensors and shifting replay distributions is not a value; it is a number waiting to mislead the policy at exactly the wrong moment. Figure 16.1B traces that full training cycle from collected transition to updated network.

Theory

In tabular Q-learning, the agent updates one state-action entry after observing a transition \((s_t, a_t, r_t, s_{t+1})\). The target is the reward now plus the best discounted value the agent currently believes is available next, a recursive relationship formalized by the Bellman equations:

$$y_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a')$$

The update moves the old estimate toward that target:

$$Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha \left[y_t - Q(s_t,a_t)\right]$$

The bracketed term is the temporal-difference (TD) error. In an embodied task, a positive TD error says the action produced a better consequence than expected, such as a door moving farther than predicted. A negative TD error says the action was overvalued, such as a grasp that looked good from the camera view but slipped under load.

How the discount factor shapes value

The discount factor \(\gamma \in [0,1)\) weights future rewards against immediate ones. In embodied AI this is not a mathematical convenience. A robot has finite battery, accumulates mechanical wear, and faces safety constraints that make distant rewards genuinely less reliable than near ones. Setting \(\gamma\) close to 1 tells the agent that a reward 50 steps away is almost as valuable as one now; setting it lower makes the agent prefer quick, certain payoffs. Mechanically, \(\gamma\) compounds with depth: a reward \(k\) steps ahead contributes \(\gamma^k\) to the current Q value. For \(\gamma = 0.9\), a reward 20 steps out is worth only \(0.9^{20} \approx 0.12\) of its face value. In practice, this exponential decay typically biases the learned Q values toward actions with near-term consequences over those with distant ones, since the distant reward's contribution to the target shrinks geometrically with every additional step of bootstrapping. Sensing a distant reward through repeated discounting is like trying to feel a 1-gram weight through a thick winter glove. The signal is physically present, but the insulating layers attenuate it so much that the network can barely distinguish it from noise. That attenuation is exactly why contact events 30 steps before a terminal success rarely drive reliable gradient updates without multi-step returns.

A Deep Q-Network (DQN) replaces the table with a neural network \(Q_\theta(o,a)\), usually trained by minimizing the squared TD error over replayed transitions:

$$\mathcal{L}(\theta) = \left(r + \gamma \max_{a'} Q_{\theta^-}(o',a') - Q_\theta(o,a)\right)^2$$

\(Q_\theta\) is the online network being trained. \(Q_{\theta^-}\) is a slower target network that supplies the bootstrap value, which prevents the target from chasing every small online update.

That completes the three-part promise from the start of this section: the Bellman update above is the tabular target derived from first principles, the algorithm box below turns it into a runnable DQN training loop, and the deadly-triad discussion that follows shows exactly why experience replay and target networks are both required to keep that loop stable.

The number of environment steps a DQN agent needs before its Q values stabilize on a simple Atari game depends almost entirely on the two stabilizers below, not on network size or reward scale.

DQN needs its two stabilizers for a specific reason. Function approximation (using a neural network to estimate Q values for states it has never exactly seen, instead of storing one entry per state-action pair), bootstrapping, and off-policy learning together form the deadly triad of divergence. Each ingredient alone is manageable, but all three at once cause value estimates to spiral. Target networks and experience replay break the correlation between consecutive updates. That correlation is what drives the divergence. The scale of the difference is concrete: without a target network, Q values on Atari games routinely diverge within 500,000 steps. With a frozen target refreshed every 10,000 steps, the same network trains stably for 50 million steps and reaches human-level scores on a majority of the 49-game suite (Mnih et al., 2015).

Worked Example

Code Fragment 1 traces a single TD update with small numbers. The robot chooses a forward nudge, receives a small penalty for contact force, but the next state has one action that looks promising.

# Trace one Q-learning update with concrete values.
# The TD target combines immediate contact cost with the best next action value.
alpha = 0.25
gamma = 0.90
old_q = 0.40
reward = -0.10
next_action_values = [0.20, 0.80, 0.35]

target = reward + gamma * max(next_action_values)
td_error = target - old_q
new_q = old_q + alpha * td_error

print(f"target={target:.2f}")
print(f"td_error={td_error:.2f}")
print(f"updated_q={new_q:.2f}")

This numeric trace is the same logic DQN applies at scale. The difference is that DQN computes the current value and the target value with neural networks, then uses gradient descent to reduce the TD error over many replayed transitions.

Before scaling to networks, it helps to watch that single update fire twice in a row so the backward propagation of reward becomes visible by hand.

Practical Recipe

1. Use Q-learning when the action set is discrete or can be discretized without hiding the control problem.
2. Define the reward so that near-term penalties, such as force spikes or collisions, do not disappear behind long-horizon success.
3. Track TD-error distributions, not only episode return. A widening TD-error tail often reveals bootstrapping instability.
4. Evaluate the greedy policy separately from the exploratory behavior policy.
5. Save per-transition logs with observation hashes, action ids, rewards, done flags, and target values.

The DQN design is a compromise between a clean Bellman equation and messy embodied data. Replay breaks short-range temporal correlation, the target network slows down the bootstrap target, and epsilon-greedy exploration keeps collecting non-greedy actions. Each stabilizer answers a specific failure: correlated frames, moving targets, and premature certainty.

For embodied agents, the hidden assumption is that the replay distribution contains enough coverage around the actions the greedy policy will later choose. If the robot learned mostly from safe, slow motions, the Q network may assign unreliable values to fast recovery actions that appear rarely but matter during deployment.

A robust DQN implementation starts with one inspectable Bellman update, then scales to replay and target networks. Code Fragment 2 records the fields that make a DQN run auditable: the online estimate, target estimate, TD error, and data-source label.

1. Log every transition as \((o,a,r,o',done)\) with episode id and seed.
2. Compute the target with a frozen or slowly updated target network.
3. Record the online Q value and the bootstrap component separately.
4. Plot TD-error quantiles by environment condition.
5. Compare greedy evaluation runs on one fixed perturbation panel.

# Build one audit record for a DQN target computation.
# Keeping target parts separate makes bootstrapping errors visible.
from dataclasses import dataclass, asdict

@dataclass
class DQNAuditRecord:
    transition_id: str
    action: str
    reward: float
    online_q: float
    target_q: float
    td_error: float
    source: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = DQNAuditRecord(
    transition_id="episode_014_step_032",
    action="nudge_forward",
    reward=-0.10,
    online_q=0.40,
    target_q=0.62,
    td_error=0.22,
    source="replay: low-friction block",
)
print(record.as_row())

When DQN fails, trace the bad action to one of five causes: perception error, reward misspecification, poor replay coverage, target-network lag, or overestimated bootstrap values. Rerun the same evaluation panel, saving frames, selected actions, max-Q values, and TD errors. Those four fields turn a vague weak-model complaint into a diagnosis.

Once those suitability conditions are settled and the algorithm is the right fit, everything in this section collapses into a single durable lesson about what makes the bootstrapped target trustworthy.