Section 6.5: Differentiable physics: what it buys you

"Once the simulator can tell you how a parameter change ripples into future states, you stop guessing and start steering."

A Gradient-Aware Trajectory Optimizer

Ask an ordinary simulator "your robot just missed the target by four centimeters" and it shrugs; ask a differentiable physics simulator the same thing and it hands you a precise answer: nudge this motor torque up, that approach angle down, and the miss shrinks. The difference is a single returned object. That object is the derivative of any output quantity, such as a task loss, with respect to its inputs: forces, control signals, masses, or shape parameters. It makes the simulator behave like a differentiable function you can optimize through. Figure 6.5A traces this idea: gradients flow backward through the simulator from a task loss to the policy parameters, marking which physics quantities are differentiable and which are not. The rest of this section builds the technical contract for differentiable physics into a usable mental model. First it defines the object of study, then connects it to the agent loop, then tests it with a compact implementation.

The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

Before differentiable simulators existed, engineers had three options for tuning physical parameters or trajectories: hand-derive the gradients (laborious and error-prone for contact-rich systems), run black-box optimization (sample-inefficient, requiring thousands of rollouts per gradient estimate), or iterate manually. The DiffTaichi benchmark (Hu et al., 2019) puts numbers on the cost: black-box search needed roughly 50,000 rollouts to find a stiffness distribution that gradient descent found in 200 steps. Without differentiable physics: 50,000 simulator evaluations, roughly 14 hours on a single CPU; with it: 200 gradient steps, under 4 minutes. Differentiable physics removes this bottleneck. The simulator becomes a differentiable function, so a single backward pass updates every parameter that influences the trajectory, exactly as it would update a neural network layer. This matters because a robot's policy, morphology, and world model all interact through physics. Gradient flow through the simulator lets all three be co-optimized together. Figure 6.5B traces this loop: a forward rollout turns parameters into a scalar loss, and a single backward pass returns the sensitivity of that loss to every parameter and control input at once. The constraint is that the technique works only where the physics is smooth, and establishing that smoothness takes most of the practical effort. A gradient computed inside the simulator is a promise about physics; a finite-difference check is the proof.

The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

Worked Example: Backprop Through a Rollout, Validated Against Finite Differences

A differentiable simulator exposes a discrete update \(x_{k+1} = f_\Delta(x_k, u_k, \theta)\) and a scalar loss \(L = \ell(x_T)\), then lets you compute sensitivities such as \(\partial L / \partial u_k\) or \(\partial L / \partial \theta\) by the chain rule through every step. The example optimizes the launch velocity of a projectile so it lands on a target, by differentiating the landing-error loss through a symplectic rollout. The non-negotiable discipline is to validate the analytic gradient against a centered finite difference before trusting it for optimization.

Reverse mode (also called backpropagation: computing a gradient by propagating sensitivities backward from the loss to the inputs, as opposed to forward mode, which propagates a perturbation forward from one input at a time) is the specific chain-rule strategy used throughout this section; the next paragraph explains why it is the right strategy for robots before the mechanics are derived. For a horizon of \(T\) steps the reverse-mode gradient accumulates local Jacobians (matrices of partial derivatives describing how a small change in one step's state affects the next step's state), \(\tfrac{\partial L}{\partial x_0} = \big(\prod_{k} \tfrac{\partial x_{k+1}}{\partial x_k}\big)^\top \tfrac{\partial \ell}{\partial x_T}\). The derivation below proceeds by hand to keep the mechanism visible, then checks it numerically.

Why reverse mode matters for robots. A real robot has dozens of joints but only one scalar task loss (energy consumed, landing error, grasp force). Reverse mode computes sensitivities of that single loss with respect to every parameter in one backward pass, regardless of how many parameters there are. Forward mode would require one pass per parameter, making it prohibitive for high-dimensional morphology or policy search. On physical hardware, this translates directly to sample efficiency: fewer rollouts needed, less wear on actuators, and faster convergence before a limited battery budget runs out.

How the backward pass works mechanically. During the forward rollout, each step stores the local Jacobian \(\partial x_{k+1}/\partial x_k\) (a small matrix relating how a state perturbation at step \(k\) propagates to step \(k+1\)). After the rollout, a single vector \(\lambda_T = \partial \ell / \partial x_T\) is initialized at the final loss, then propagated backward: \(\lambda_k = (\partial x_{k+1}/\partial x_k)^\top \lambda_{k+1}\). The chain terminates at \(\lambda_0\), which is the full gradient with respect to the initial state. Control or parameter gradients are read off at each step using the same stored Jacobians, costing one matrix-vector multiply per step rather than a full matrix product.

import numpy as np

g = 9.81
dt, T = 0.01, 300            # 3 s rollout
target_x = 6.0               # land here on the ground (y = 0)

def rollout(v0):
    """Symplectic-Euler projectile from origin. Returns landing x and loss."""
    vx, vy = v0
    x, y = 0.0, 0.0
    for _ in range(T):
        vy = vy - g * dt     # gravity on vertical velocity
        x  = x + vx * dt
        y  = y + vy * dt
    loss = 0.5 * (x - target_x)**2
    return x, loss

def analytic_grad(v0):
    """d loss / d v0 by differentiating the closed-form rollout.
    With constant vx: x_T = vx * (T*dt). Loss = 0.5*(x_T - target)^2."""
    vx, vy = v0
    x_T = vx * (T * dt)
    err = x_T - target_x
    dloss_dvx = err * (T * dt)        # dx_T/dvx = T*dt
    dloss_dvy = 0.0                    # horizontal landing x independent of vy here
    return np.array([dloss_dvx, dloss_dvy])

def finite_diff_grad(v0, eps=1e-6):
    grad = np.zeros(2)
    for i in range(2):
        vp = v0.copy(); vp[i] += eps
        vm = v0.copy(); vm[i] -= eps
        grad[i] = (rollout(vp)[1] - rollout(vm)[1]) / (2 * eps)
    return grad

v0 = np.array([2.5, 1.0])
print("analytic grad     :", analytic_grad(v0))
print("finite-diff grad  :", finite_diff_grad(v0))
print("max abs diff      :", np.max(np.abs(analytic_grad(v0) - finite_diff_grad(v0))))

# Gradient descent on v0 to hit the target.
v = np.array([2.5, 1.0]); lr = 0.05
for step in range(200):
    v = v - lr * analytic_grad(v)
x_final, loss = rollout(v)
print(f"optimized vx={v[0]:.4f}  landing x={x_final:.4f}  loss={loss:.2e}")

The analytic and finite-difference gradients agree to within numerical tolerance, which is the green light to optimize: gradient descent then drives the landing point onto the target. This rollout is smooth, so the gradient is typically exact to numerical precision. The cautionary half of the lesson is that adding a contact event (a bounce, a grasp, a foot strike) makes \(f_\Delta\) only piecewise smooth, and the same finite-difference check will then expose where the autodiff gradient is biased by contact smoothing or undefined across a mode boundary. Run the check on one smooth case and one contact case before optimizing a long horizon.

Practical Recipe

The worked example proved a gradient on one smooth rollout; turning that single check into a dependable workflow on a real arm means following a fixed order, starting from the parameters themselves and only later admitting contact.

1. Pin the physical parameters that will be differentiated before writing the first rollout: for a Franka Panda arm, this means joint inertias (0.35 kg m² at the shoulder), motor damping coefficients, and the stiffness of the fingertip contact patch. Differentiating unknown or unconstrained parameters produces gradients that are numerically valid but physically meaningless.
2. Build a smooth free-motion baseline first: a single rigid body or a single joint driven by a PD controller, no contact. Verify energy conservation and check the analytic gradient against centered finite differences. On a Franka Panda in MuJoCo (2 ms timestep, implicit integrator), the gradient of joint-angle error with respect to motor torque should agree with finite differences to within 0.1 percent on a 500 ms horizon.
3. Add contact only after the free-motion gradient passes. Introduce contact incrementally: one soft contact first (MuJoCo elliptic cone, solimp=[0.9, 0.95, 0.001], where solimp is MuJoCo's solver-impedance parameter that sets how softly a contact constraint is enforced), then stiffen it toward the real object. Re-run the finite-difference check at each stiffness level. Gradient bias under hard contact is visible as a growing discrepancy between analytic and numerical estimates before optimization diverges.
4. Record failures by physical cause: gradient vanishing over long horizons (horizon beyond 2 s at 2 ms timestep causes Jacobian products to lose three orders of magnitude), gradient bias from contact mode switches (a foot-strike during a Boston Dynamics Spot gait produces a gradient sign error on stance-phase force), or sim-to-real gap (a stiffness gradient optimized in MuJoCo at 10 ms timestep may not transfer to a 1 kHz real controller because high-frequency actuator dynamics are absent from the simulation).
5. Before deploying a gradient-optimized policy on hardware, run a single perturbation: shift the contact body mass by 20 percent and recheck the loss landscape. If the optimized solution lies in a narrow valley that disappears under this perturbation, that is evidence the gradient found a simulator artifact rather than a physical optimum, though the check is a necessary heuristic rather than a formal proof of transfer. This check takes one additional rollout and has saved multiple real-robot experiments from repeating the DiffTaichi hard-contact failure on physical hardware.

Production Pattern

With the recipe and its failure modes in hand, the last step is to place differentiable physics inside the larger robotics stack so its gradients answer a design question rather than float free of one.

Differentiable physics: what it buys you sits inside the Part II robotics contract: geometry defines where things are, kinematics defines what motion is possible, dynamics defines what motion costs, control defines how errors are corrected, and sensing defines what the agent can know on time.

Use differentiable physics when gradients answer a design question, and validate them against finite differences. This makes the section useful to students, builders, and researchers at the same time: the idea has an intuitive role, a formal interface, a runnable check, and a failure mode that can be reproduced.

When to reach for it

Differentiable physics is useful when the question is not only "what happened?" but "which parameter, action, or design choice would make the outcome better?" Gradients can tune masses, friction coefficients, control sequences, morphology parameters, or policy inputs, but only if the gradient is a faithful derivative of the simulation the reader intends to trust.

Consider a specific case. The DiffTaichi project (Hu et al., 2019) used differentiable simulation to optimize a soft robot's material stiffness distribution for locomotion speed. Starting from a uniform-stiffness body, gradient descent through a 500-step rollout converged to a non-uniform stiffness pattern in roughly 200 optimization steps. Black-box search found the same result only after tens of thousands of rollouts. The same gradient pipeline then failed on a gripper task once hard contact entered the model: the discontinuous contact normal force produced gradient estimates that pointed the wrong direction, so the team switched to a contact-smoothed approximation before optimization resumed. This contrast, fast convergence on smooth deformable bodies against unreliable gradients on hard contact, shows exactly where to reach for differentiable physics and where to fall back to sampling or trajectory optimization with explicit constraint handling.

Differentiable physics is the wrong choice when the dominant dynamics are discontinuous and no smooth contact model is acceptable (for example, high-speed impact or granular media), when the gradient horizon is so long that Jacobian products vanish or explode before reaching the early timesteps, or when the real-world system differs enough from the simulator that a gradient computed in simulation does not transfer. In those cases, model-based trajectory optimization with explicit constraint handling, or sample-efficient black-box methods such as CMA-ES (Covariance Matrix Adaptation Evolution Strategy), typically outperform differentiable simulation in practice.

Use this recipe when turning Differentiable physics: what it buys you into code, a simulator experiment, or a robot diagnostic. The point is not to use every library. The point is to keep the hand-built baseline and the maintained-tool path comparable.

1. Specify mass, inertia, actuator limits, contact model, timestep, and solver tolerance before running a rollout.
2. Run one free-motion test and one contact test with logged energy, constraint violation, and penetration depth.
3. Compare the hand calculation with MuJoCo, Drake, Pinocchio, or MJX on the same model and timestep.
4. Store solver settings, random seed, initial state, trajectory, and failure labels in one artifact.
5. Scale to Isaac Lab or GPU-parallel simulation only after a small model passes deterministic checks.

Distrust smooth simulation until the section-specific physical assumption has been stress-tested: timestep, contact stiffness, damping, friction, actuation, and energy behavior should each have a small diagnostic.