Section 21.3: DAgger and dataset aggregation

"The learner drove; the expert labeled where it actually went. One loop at a time, the map of recoverable states grew."

Section 21.3

Picture a self-driving prototype that handles the highway flawlessly until the first time it drifts a half-lane off center, then has no idea what to do because no human driver in its training data ever started from there: DAgger is the fix that lets the learner generate those scary states itself and asks the expert to label the way back. Figure 21.3A sketches the iterative loop at a glance: deploy the current policy, query the expert on visited states, aggregate, and retrain. The section defines the object of study, connects it to the agent loop, and tests it with a compact implementation.

Concretely, "aggregation" means the training set never shrinks or resets between rounds: round \(k\)'s dataset \(\mathcal{D}_k\) is the union of every prior round's expert-labeled pairs plus the newly labeled states from round \(k\), so the policy at round \(k\) is always retrained on the full history, not just the latest batch. This is the mechanical answer to "how do I actually use DAgger": collect a seed demonstration set, and at each subsequent round append (never replace) new expert-labeled corrections before retraining from scratch or fine-tuning on the growing union.

The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Two terms recur throughout this section and are worth pinning down before the theory: covariate shift (the mismatch between the state distribution a policy was trained on and the distribution it actually encounters once deployed) is the disease, and closed loop (the policy's own actions determine its next observation, rather than following a pre-recorded, fixed sequence of states) describes the setting where that mismatch compounds turn after turn.

Theory

Before the aggregation loop runs, fix the interface DAgger will iterate on so each round is comparable to the last. On a Franka Panda running robomimic's Can task in MuJoCo, that means pinning the observation (wrist-camera RGB plus 7-DoF joint positions), the action (end-effector delta pose at 20 Hz), the physics timestep (2 ms), and the intervention flag schema before round 0. If any of these shifts between rounds, the expert's corrections from round 1 were given under different dynamics than round 3 measures, and the intervention-rate curve stops being a clean signal.

Worked Example

Keep one concrete rollout in view. A sensor reading becomes an estimate, the estimate constrains an action, the action changes the world, and the next observation confirms or contradicts the assumption. The section's idea is useful only if it improves that loop.

from pathlib import Path

dataset_root = Path("robot_demos")
for episode in sorted(dataset_root.glob("episode_*")):
    print("inspect", episode.name)
print("next step: convert demonstrations to the LeRobotDataset format")

Expected output: the printed trace for DAgger and dataset aggregation should expose the method configuration, the measured evidence field, and the failure label. If one of those fields is missing or unchanged under the perturbation, the example is not yet an evaluation artifact.

# Track which learner version created each queried observation.
# This makes the aggregate dataset auditable across DAgger rounds.
rounds = [
    {"policy": "pi_0", "queried_states": 120, "expert_labels": 120},
    {"policy": "pi_1", "queried_states": 80, "expert_labels": 80},
    {"policy": "pi_2", "queried_states": 45, "expert_labels": 45},
]
total_labels = sum(r["expert_labels"] for r in rounds)
print("DAgger rounds:", len(rounds))
print("aggregate expert labels:", total_labels)

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

DAgger earns its keep only under a closed-loop contract: the observation stream, the state estimate, the action representation, the timing budget, and the evaluation artifact. Skip that contract and a model looks capable in a notebook, then fails the first time a sensor drops a frame or a controller saturates.

Separate the conceptual claim, the systems claim, and the evidence claim. A plausible mechanism, a clean interface, and a closed-loop result are different claims; the section should keep their evidence separate.

Start with a small baseline that logs inputs, outputs, units, timestamps, and termination conditions before moving to Gymnasium or PettingZoo. The library run should keep the same artifact schema, so the comparison remains a same-task evaluation.

1. Write a one-paragraph task contract with observation, action, success, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel.

When DAgger and dataset aggregation fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.