Section 16.3: Continuous control: DDPG, TD3, SAC

"Discrete actions ask which door. Continuous actions ask exactly how far to push, and the critic must answer in kind."

Section 16.3

Figure 16.3A previews the comparison this section works through in detail: a shared actor-critic interface with three different answers to the overestimation problem.

The figure's mention of "twin critics" is introduced ahead of its explanation: the Theory section below defines what a twin critic pair is and why TD3 and SAC both use one before any equation depends on the term.

Choosing among DDPG, TD3, and SAC for a continuous-control task turns on three questions: how each one controls critic overestimation, how it drives exploration, and which action-level logs prove a learned policy is safe to run on hardware.

Ask a discrete agent to pick a gripper velocity and it must first quantize a continuous dial into bins; ask it to balance a torque to the third decimal place and the table of possible actions becomes infinite, and the max over that table, the operation at the heart of tabular Q-learning, quietly stops being a lookup at all. A robot torque, steering angle, or gripper velocity can take infinitely many values, and no finite table can enumerate them.

DDPG, TD3, and SAC solve this by pairing a critic with an actor. The critic estimates value for a continuous action, while the actor proposes the action to evaluate or execute. This section explains how each method controls the bootstrapping error (the compounding error that arises when a value estimate is updated using another estimate rather than a true outcome) that appears when the critic and actor improve each other from imperfect off-policy data. Figure 16.3B below diagrams this actor-critic feedback loop before the per-algorithm mechanics are introduced.

Theory

DDPG uses a deterministic actor \(\mu_\phi(o)\) and a critic \(Q_\theta(o,a)\). The actor is trained to choose actions that the critic values highly, while the critic is trained from replayed Bellman targets drawn from the replay buffer and stabilized with target networks. This is efficient, but brittle: if the critic overestimates an action, the actor will move toward that action.

Concretely, the actor update follows the critic's gradient with respect to the action: \(\phi \leftarrow \phi + \alpha_\phi \nabla_\phi \frac{1}{N}\sum Q_\theta(o, \mu_\phi(o))\). There is no explicit max over actions, as in tabular Q-learning; the actor network itself plays that role by climbing whatever direction the critic currently reports as most valuable, which is exactly why a critic overestimate at one action pulls the actor toward it.

TD3 addresses that brittleness with three design choices. It trains two critics and uses the smaller target value. It delays actor updates so the critic has time to improve. And it adds small clipped noise to the target action so the critic cannot exploit a narrow spike in value. On a physical robot, a critic spike at one precise torque value is dangerous. The actor converges toward that torque, the joint exceeds its safe range, and the hardware protection trips before useful data is collected. Smoothing forces the critic to assign consistent value to a neighborhood of nearby torques, not a single fragile point. The mechanism clips Gaussian noise to a range \([-c, c]\) and adds it to the target actor's output before evaluating the critic. Because the target trains against this perturbed action, it cannot learn to score any single action far above its neighbors. The target commonly has the form:

$$y = r + \gamma \min_i Q_{\theta_i^-}(o', \mu_{\phi^-}(o') + \epsilon)$$

Where TD3 smooths the target action to keep a deterministic critic honest, SAC attacks the same brittleness from the actor side, by refusing to let the policy collapse to a single action at all. A deterministic policy that converges too early has not learned control; it has learned a brittle shortcut that the hardware will refuse. SAC changes the objective by rewarding both task return and action entropy, the maximum-entropy formulation developed in the next section. The policy is stochastic, so the agent keeps useful diversity in its action choices:

$$J(\pi)=\mathbb{E}\left[\sum_t r(o_t,a_t) + \alpha \mathcal{H}(\pi(\cdot|o_t))\right]$$

The temperature \(\alpha\) controls how much the policy values entropy. In contact-rich robotics, that entropy can help discover recovery actions that a deterministic actor would stop trying too early. On the MuJoCo HalfCheetah benchmark, as reported in Haarnoja et al. (2018), a vanilla DDPG run typically requires around 3 million environment steps to match the performance that SAC reaches in 300,000 steps: this is the entropy bonus accelerating exploration, and the gap widens further once the task involves unexpected contact.

Worked Example

Code Fragment 1 computes a TD3-style target from two critics. The smaller critic value is used because overestimated value is more dangerous than underestimated value when the actor is trained to chase high values.

# Compute a TD3 target with clipped double critics.
# The smaller critic value limits overestimation before the actor sees it.
reward = 0.3
gamma = 0.98
critic_1_target = 1.40
critic_2_target = 0.90
target_policy_noise = 0.05
smoothed_action = 0.62 + target_policy_noise

bootstrap = min(critic_1_target, critic_2_target)
td3_target = reward + gamma * bootstrap

print(f"smoothed_action={smoothed_action:.2f}")
print(f"bootstrap={bootstrap:.2f}")
print(f"td3_target={td3_target:.2f}")

The expected output shows the TD3 target being anchored by the lower critic value, not the optimistic one. Readers should interpret the td3_target of 1.18 as a deliberately conservative bootstrap built from a slightly perturbed target action.

The same target logic maps cleanly to embodied control. For a torque-controlled arm, target smoothing says the critic should value a small neighborhood of torques, not a single fragile torque vector that only works in simulation.

Practical Recipe

1. Use DDPG only when a deterministic actor is acceptable and the task is well shaped.
2. Prefer TD3 when critic overestimation or narrow action spikes appear in evaluation.
3. Prefer SAC when exploration, recovery behavior, or multimodal actions matter.
4. Log action saturation at actuator bounds, since saturated actors can hide critic problems.
5. Evaluate under mass, friction, delay, and sensor-noise shifts with the same action limits.

Each algorithm breaks in a characteristic way. DDPG's single critic overestimates value for actions the actor visits rarely. The symptom is a rising Q-loss curve alongside a policy that saturates actuator limits and stops improving. TD3 can mask overestimation without curing it. When both critics converge to similar but jointly wrong values, the clipped minimum offers no protection. The actor then chases a false peak. Look for low critic disagreement paired with falling episode return after a stable plateau. SAC fails most often through entropy collapse. The temperature \(\alpha\) decays to near zero, the policy becomes nearly deterministic, and the agent loses the ability to recover from novel contact states. The surprising reversal here is that SAC, the most sophisticated of the three algorithms, tends to break more catastrophically than DDPG at this point. DDPG was always deterministic, so the hardware team knew to plan around it. A collapsed SAC policy instead looks deterministic on evaluation logs yet was designed around diversity it no longer has. The observable signal is entropy dropping below 0.1 nat while evaluation success variance spikes.

With those failure modes in view, the three algorithms separate not by what they observe or actuate but by how much they trust their own value estimates. DDPG, TD3, and SAC differ less in their environment interface than in their attitude toward critic error. DDPG trusts one critic and one deterministic actor. TD3 distrusts overestimation enough to train two critics. SAC distrusts premature certainty enough to optimize entropy alongside return.

That distinction matters in embodied work because physical action errors have asymmetric cost. An underestimated value may slow learning, but an overestimated high-torque action can break contact, drop an object, or leave the training distribution entirely.

A robust continuous-control implementation logs the actor's action and the critic's evidence for that action. Code Fragment 2 records the fields that separate deterministic control, clipped double-Q control, and entropy-regularized control.

1. Record action vectors before and after clipping or squashing.
2. Log critic disagreement for TD3 and SAC.
3. Log entropy or temperature for SAC.
4. Track actuator saturation and safety-envelope violations.
5. Compare all methods on one perturbation panel with the same action bounds.

# Build one audit record for a continuous-control action.
# The fields expose actor output, critic disagreement, and safety limits.
from dataclasses import dataclass, asdict

@dataclass
class ContinuousControlAudit:
    algorithm: str
    raw_action: float
    executed_action: float
    critic_1: float
    critic_2: float
    entropy: float
    saturated: bool

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = ContinuousControlAudit(
    algorithm="SAC",
    raw_action=1.18,
    executed_action=1.00,
    critic_1=2.4,
    critic_2=1.7,
    entropy=0.42,
    saturated=True,
)
print(record.as_row())

The expected output is an audit row that immediately exposes two continuous-control risks: the actor asked for more torque than the actuator allowed, and the critics disagree materially about the value of that command. That combination should trigger closer inspection before anyone treats the rollout return as robust evidence.

When a continuous-control method fails, check four suspects: the actor left the safe action region, the critics disagreed, entropy collapsed, or replay lacked the new dynamics. Then rerun one perturbation, plotting action histograms and critic disagreement beside the episode video.