"The expert left no reward function, only trajectories. Inverse RL asks what you would have to care about to move like that, then optimizes the answer."
A Curious Reward Engineer
This section assumes familiarity with Markov decision processes and reward functions from section 14.1, and with the behavior cloning baseline from section 21.1. The reward-inference ideas developed here are extended in section 23.3, where preferences collected during teleoperation are used to refine the inferred reward, and in section 34.2, where language-conditioned reward signals replace hand-designed feature sets in vision-language-action models.
A surgeon's assistant robot trained by behavior cloning on 500 suture passes fails the moment the needle gauge changes. It copied wrist angles, not intent. Inverse reinforcement learning asks the deeper question: what reward function would a rational agent have to be maximizing to produce exactly these trajectories? Recover that reward and you can re-optimize it for new tools, new patients, new geometries. As robots enter open-ended real-world settings where demonstrations can never cover every situation, this shift from copying behavior to inferring goals is the central unsolved problem. This section develops a maximum-entropy IRL pipeline, traces exactly where reward ambiguity arises, and establishes the practical conditions under which IRL outperforms behavior cloning versus when its inner-loop cost makes it impractical. Figure 21.4A shows the overall shape: observed expert trajectories go in, a reward function that rationalizes them is recovered, and an RL agent is then optimized against that inferred reward.
Watch an expert pour water without spilling and you can copy the wrist motion exactly, yet still have no idea what the expert was trying to achieve. Inverse reinforcement learning reads intent off behavior. It recovers the reward an expert must have been maximizing rather than the motions they happened to produce. We first define that object of study, then connect it to the agent loop, then test it with a compact implementation.
The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?
A representation earns its place when it changes the measurable action interface. In inverse reinforcement learning, the reader should keep asking which decision becomes easier, safer, or more reliable.
Theory
The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.
The mechanism in Inverse reinforcement learning is the contract between representation and action. Name what enters the module, what leaves it, which assumptions make that transformation valid, and which log would reveal a bad handoff.
Worked Example
That contract between representation and action stays abstract until it is grounded in a single trace, so the worked example below makes the reward-inference loop concrete one step at a time.
Keep one concrete rollout in view. A sensor reading becomes an estimate, the estimate constrains an action, the action changes the world, and the next observation confirms or contradicts the assumption. The section's idea is useful only if it improves that loop.
from pathlib import Path
dataset_root = Path("robot_demos")
for episode in sorted(dataset_root.glob("episode_*")):
print("inspect", episode.name)
print("next step: convert demonstrations to the LeRobotDataset format")robot_demos directory, lists each episode folder in sorted order, and prints the target format (LeRobotDataset) that the raw episodes still need to be converted into before training.Expected output: the printed trace for Inverse reinforcement learning should expose the method configuration, the measured evidence field, and the failure label. If one of those fields is missing or unchanged under the perturbation, the example is not yet an evaluation artifact.
Use imitation, preference-learning, or custom IRL tooling for optimization, but log feature definitions, reward weights, planner settings, and counterexample tasks so the learned reward is not mistaken for ground truth.
Inverse Reinforcement Learning And Reward Ambiguity
Consider a concrete motivation. A behavior cloning policy trained on 200 demonstrations of a robot pouring water mimics the operator's wrist angle and grip force. When asked to pour into a smaller cup, it fails because it memorized motor patterns rather than goals. IRL targets exactly this transfer scenario. Instead of copying actions, it asks what cost the expert was minimizing, then re-optimizes that cost under the new geometry. The difference is measurable. Behavior cloning retrained from scratch for the smaller cup required 180 additional demonstrations to reach 80% success. An IRL policy re-optimized against the recovered reward reached 80% success in 12 fine-tuning rollouts, a 15x reduction in data cost (illustrative figures consistent with IRL transfer results as of 2024; exact ratios vary with task and feature set). The tradeoff is real: IRL requires solving a forward RL problem in the inner loop, which costs compute. The inferred reward may also be ambiguous when demonstrations are few or the feature set is incomplete.
Inverse reinforcement learning asks a different question from behavior cloning. Instead of directly fitting expert actions, it searches for a reward function \(r_\phi(s,a)\) under which the expert behavior looks optimal or near-optimal. Figure 21.4B traces this as a loop: expert demonstrations yield a feature expectation, the current reward drives a forward RL inner loop, and the gap between expert and policy feature expectations updates the reward until the two match. In maximum entropy IRL, a trajectory receives probability proportional to exponentiated return:
Checkpoint
So far: IRL replaces the behavior-cloning goal of matching expert actions with the goal of finding a reward function \(r_\phi(s,a)\) that makes the expert trajectories look optimal; that reward is found by an inner loop that solves forward RL, rolls out the resulting policy, and pushes the policy's feature expectation toward the expert's feature expectation until they match.
$$P_\phi(\tau) \propto \exp\left(\sum_t r_\phi(s_t,a_t)\right).$$
Step-Through: One MaxEnt IRL gradient step
Trace one iteration with a 2-feature reward \(\theta = [0.0, 0.0]\), learning rate \(\alpha = 0.5\). Features are (task progress, smoothness).
Step 1, expert feature expectation. Two expert trajectories give per-trajectory feature sums \(\phi^{(1)} = [0.9, 0.8]\) and \(\phi^{(2)} = [0.7, 1.0]\). Average: \(\hat{\mu}_E = [(0.9+0.7)/2,\ (0.8+1.0)/2] = [0.80, 0.90]\).
Step 2, forward RL with \(\theta = [0,0]\). Reward is identically zero, so every trajectory is equally likely. The policy is uniform, and rolling it out yields a policy feature expectation \(\hat{\mu}_\theta = [0.50, 0.50]\) (the average over the whole reachable set, lower on both features than the expert).
Step 3, gradient. \(\nabla_\theta \mathcal{L} = \hat{\mu}_E - \hat{\mu}_\theta = [0.80 - 0.50,\ 0.90 - 0.50] = [0.30, 0.40]\).
Step 4, update. \(\theta \leftarrow [0,0] + 0.5 \cdot [0.30, 0.40] = [0.15, 0.20]\). The reward now prefers both progress and smoothness, with a slightly stronger pull toward smoothness because the expert's smoothness gap was larger. Re-running the forward solve with \(\theta = [0.15, 0.20]\) pushes \(\hat{\mu}_\theta\) up toward \([0.80, 0.90]\), shrinking the next gradient. Convergence is reached when the two feature expectations match and the gradient hits zero.
Algorithm: Maximum Entropy Inverse Reinforcement Learning
Input: Expert demonstration trajectories \(\mathcal{D} = \{\tau^{(1)}, \ldots, \tau^{(N)}\}\); feature map \(\phi(s,a) \in \mathbb{R}^k\); learning rate \(\alpha\); reward parameter vector \(\theta \in \mathbb{R}^k\)
Output: Reward parameters \(\theta^*\) such that \(r_\theta(s,a) = \theta^\top \phi(s,a)\) rationalizes the expert demonstrations
- Compute the empirical feature expectation from expert data: \(\hat{\mu}_E = \frac{1}{N}\sum_i \sum_t \phi(s_t^{(i)}, a_t^{(i)})\).
- Initialize reward parameters \(\theta \leftarrow \mathbf{0}\) (or small random values).
- Repeat until convergence:
- Define the current reward \(r_\theta(s,a) = \theta^\top \phi(s,a)\).
- Solve the forward RL problem to obtain the optimal policy \(\pi_\theta = \arg\max_\pi \mathbb{E}_\pi[\sum_t r_\theta(s_t, a_t)]\).
- Roll out \(\pi_\theta\) to estimate the policy feature expectation \(\hat{\mu}_\theta = \mathbb{E}_{\pi_\theta}[\sum_t \phi(s_t, a_t)]\).
- Compute the maximum entropy log-likelihood gradient: \(\nabla_\theta \mathcal{L} = \hat{\mu}_E - \hat{\mu}_\theta\).
- Update reward parameters: \(\theta \leftarrow \theta + \alpha \nabla_\theta \mathcal{L}\).
- Evaluate the recovered reward by scoring at least one counterfactual (suboptimal) trajectory and confirming its return is lower than the expert's return.
- Return \(\theta^* = \theta\); record feature weights and counterexample trajectories as diagnostic artifacts alongside \(\pi_\theta\).
The intuition is useful but dangerous: many rewards can explain the same demonstrations. This is called the reward ambiguity problem, and it is what separates IRL from a mere curve-fitting exercise. A robot that carries a cup smoothly might be optimizing short path length, liquid stability, human comfort, or a hidden demonstrator habit. IRL becomes scientifically meaningful only when the learned reward is tested on new tasks, interventions, or counterfactual trajectories. A reward function that fits the demonstrations but fails the counterfactuals is not a recovered objective: it is a memorized alibi.
Imagine watching a seasoned chef add a pinch of salt to a dish and trying to deduce their goal from that single action. They might be seasoning for taste, controlling texture, slowing browning, or following habit. Every explanation fits what you observed. Only by presenting them with a new ingredient they have never cooked before, or by handing them a deliberately under-salted plate and watching their reaction, can you separate the true objective from plausible impostors. IRL faces exactly this problem: a finite set of demonstrations is consistent with many reward functions, and only counterfactual tests, situations the expert never encountered during training, can distinguish the intended goal from coincidental explanations.
A common assumption is that IRL recovers a single correct reward function that definitively explains what the expert was optimizing. This is wrong: the IRL problem is fundamentally ill-posed, because infinitely many reward functions are consistent with any finite set of demonstrations. In an embodied AI context this matters acutely, because the spurious rewards absorbed from operator style, hardware quirks, or dataset correlations will drive a real robot to physically unsafe behavior the moment those correlations shift in deployment. The correct mental model is that IRL narrows the space of plausible reward functions rather than identifying one; counterfactual ranking and held-out preference tests are required to shrink that space to something trustworthy enough for hardware.
For an embodied robot, reward ambiguity is not just a statistical inconvenience: it can cause physical harm. A reward that silently encodes "move fast" alongside "avoid collisions" will behave safely in the training environment but become aggressive when contact forces rise on a new surface. Unlike a simulated agent that can be reset, a real robot acting on a wrong reward may damage objects, injure bystanders, or saturate its actuators before any corrective signal arrives.
The standard fix is counterfactual ranking. After fitting reward weights, the engineer builds trajectories a correct reward would reject, such as shortcuts that skip safety margins, and checks that the inferred reward scores them below the expert demonstrations. When the ranking holds across a diverse counterexample set, the ambiguity shrinks to the rewards that agree on every tested contrast. Code Fragment 3 below is therefore not optional: it is the minimum diagnostic before trusting a recovered reward on hardware.
Concretely, counterfactual ranking answers the question this section opened with: what reward would a rational agent need in order to produce these trajectories and no others. A candidate reward is accepted only if it scores every deliberately-bad counterexample (the shortcut path, the corner-cutting maneuver, the unsafe grasp) below every expert trajectory. If even one counterexample scores higher than the expert, the candidate reward is rejected and the feature set or demonstration set must be revised. This is the concrete "how to use it" step that turns the abstract idea of reward inference into a pass or fail test an engineer can run before deployment.
Maximum Entropy IRL (Ziebart et al., 2008) assigns trajectory probabilities proportional to exponentiated return and fits reward weights by maximum likelihood; it works well when features are hand-designed and the environment is small enough for exact planning. Generative Adversarial Imitation Learning (GAIL; Ho and Ermon, 2016) replaces the inner RL loop with an adversarial discriminator that distinguishes expert from policy rollouts, scaling to high-dimensional continuous control without explicit reward features. Adversarial Inverse Reinforcement Learning (AIRL; Fu et al., 2018) adds a disentanglement structure (a network design that separates the reward, which should depend only on state and action, from a shaping term that depends on the dynamics, so changing the dynamics does not force the reward to be relearned) so the recovered reward is transferable across dynamics changes, making it the right choice when the robot body or physics will differ between training and deployment. The choice among these three depends on whether you need interpretable reward features (MaxEnt IRL), scalability without feature engineering (GAIL), or reward portability (AIRL).
When using AIRL (the airl algorithm in the imitation library), set reward_net_kwargs={"use_state": True, "use_action": False} if your transfer task changes the action space but keeps the same state observations; the default action-conditioned reward head will otherwise absorb spurious action correlations from the source embodiment and fail silently on the target robot. After training, always query reward_net.predict_processed on at least one deliberately suboptimal counterexample trajectory (for example, the "shortcut" path from Code Fragment 3) and confirm its score is lower than the expert's score before treating the recovered reward as transferable.
Modern robot reward learning often blends demonstrations, preferences, language feedback, and safety constraints. The open problem is identifiability: deciding which part of a demonstrated behavior reflects the task objective and which part reflects embodiment, operator style, or dataset bias.
Code Fragment 3 illustrates reward ambiguity with two reward weights that rank the same expert trajectory differently once a counterexample is introduced.
# Compare two plausible reward explanations for the same demonstration.
# Counterfactual trajectories expose ambiguity that imitation alone can hide.
import numpy as np
features = {
"expert": np.array([0.9, 0.8]), # task progress, smoothness
"shortcut": np.array([1.0, 0.2]),
}
smooth_reward = np.array([0.4, 0.6])
progress_reward = np.array([0.9, 0.1])
for name, phi in features.items():
print(name, "smooth-score", round(phi @ smooth_reward, 2), "progress-score", round(phi @ progress_reward, 2))shortcut smooth-score 0.52 progress-score 0.92
Real-World Application: Autonomous driving route preferences
Waymo and Uber ATG (now Aurora) have both published work describing maximum-entropy IRL used to learn driving cost functions from human traces: rather than hand-tuning weights for lane-keeping, smoothness, and headway, the planner recovers them from logged expert drives. Uber's CoverNet (a trajectory-set prediction model that scores a fixed library of candidate paths rather than regressing raw coordinates) and IRL-based motion planners infer the reward that ranks a candidate trajectory set, so the car is trained to prefer human-like paths it never saw during training. In principle the same counterfactual check from this section applies: a recovered cost should only be trusted in deployment after it scores deliberately reckless candidate trajectories below the demonstrated ones, though the published papers do not disclose the full extent of that validation.
Practical Recipe
The driving case shows IRL working at production scale, but reaching that point depends on a disciplined build order, so the following recipe distills the steps that keep an IRL pipeline debuggable from the first baseline onward.
- Write the observation, action, and success metric before choosing a model.
- Build a baseline that is simple enough to debug by inspection.
- Add the library implementation only after the baseline behavior is understood.
- Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
- Run at least one perturbation test before trusting the result.
The common mistake in Inverse reinforcement learning is to celebrate the component score before checking the closed-loop handoff. The failure usually appears at the boundary: stale state, wrong frame, delayed action, saturated actuator, or metric that ignores the real task cost.
A robot learning engineer applying inverse reinforcement learning starts by recording the robot body, camera setup, action units, operator source, and split policy for every episode. That record makes it possible to compare LeRobot with a baseline without changing the task definition midstream.
When inverse reinforcement learning feels abstract, ask what would be different in the next frame of video, the next robot state, or the next safety margin.
Language-conditioned reward learning (2024-2026). Rather than hand-crafting feature maps, recent work grounds reward functions directly in natural language. EUREKA (Ma et al., ICLR 2024; extended in follow-ons at CMU and NVIDIA) uses an LLM to propose and iteratively revise reward code from task descriptions, outperforming human-written rewards on dexterous manipulation benchmarks. The 2025 direction extends this to vision-language reward models that score robot states from image-text pairs without any simulator access.
Preference-based IRL at scale (2024-2026). Combining RLHF-style pairwise preferences with IRL closes the annotation bottleneck: instead of full trajectories, a human labels which of two clips is better. PEBBLE (an algorithm that pretrains a policy with unsupervised exploration, then fine-tunes a reward model from human pairwise preference labels) and its 2024 successors (including work from the Berkeley Robot Learning Lab) show preference-based reward learning reaches expert IRL sample efficiency with 10x fewer human labels. Active query selection, choosing the most informative clip pair, is now a mainstream research thread.
Transferable reward representations via foundation models (2025-2026). Groups at Stanford and Google DeepMind (including the RT-Reward line of work) are building reward models pretrained on internet-scale video and fine-tuned on a small set of robot demonstrations. These frozen reward encoders transfer across embodiments without re-running the inner RL loop, attacking the core computational cost of classical IRL.
Open problem for a PhD student. Reward identifiability under distribution shift: given demonstrations from multiple operators with different styles on the same task, how do you separate the shared task objective from individual operator habits, and how do you certify that the recovered objective is stable when the robot body or environment changes? Current theory gives no tight sample-complexity bound for this multi-source setting, and empirical methods lack agreed-upon benchmarks.
Can you name the observation, state estimate, action, success metric, and most likely failure mode for inverse reinforcement learning? If not, the system boundary is still too vague.
Inverse reinforcement learning becomes useful when it is tied to a closed-loop contract. That contract names the observation stream, the state estimate, the action representation, the timing budget, and the evaluation artifact. Without that contract, a model can look capable in a notebook while failing the first time a sensor drops a frame or a controller saturates.
Recap: a recovered reward is trustworthy only when it is bound to a closed-loop contract and survives counterfactual tests, not when it merely reproduces the demonstrations.
Separate the conceptual claim, the systems claim, and the evidence claim. A plausible mechanism, a clean interface, and a closed-loop result are different claims; the section should keep their evidence separate.
| Tool or Library | Role in IRL | Builder Advice |
|---|---|---|
Gymnasium (MuJoCo FetchPush-v2) | Forward RL inner loop for MaxEnt IRL and GAIL on manipulation tasks | Pin mujoco==3.1.6 and fix the physics timestep to 0.002 s; a mismatch between the demo timestep and the rollout timestep causes the inferred reward to absorb control-frequency artifacts rather than task intent. |
| imitation (AIRL / GAIL) | Adversarial reward learning without hand-crafted feature maps | For Franka Panda sim-to-real transfer, use AIRL with use_state=True, use_action=False; the action head otherwise absorbs joint-torque correlations specific to the simulated PD controller and will not transfer to the real robot's impedance controller. |
| ROS 2 (rosbag2) | Recording and replaying real-robot demonstration trajectories at hardware frequency | Record at the controller frequency (1 kHz for a Franka, 500 Hz for a UR10e) and downsample to your learning frequency (10-50 Hz) offline; downsampling online in the bag reader introduces variable-length gaps that corrupt feature expectations. |
| MuJoCo | Fast physics for the inner-loop RL solver during IRL training | Use mj_step in batch mode with a fixed seed; stochastic resets during inner-loop policy rollouts add variance to the feature expectation estimate and require 3-5x more samples to converge reward weights. |
| LeRobot (HuggingFace Hub datasets) | Sourcing cross-embodiment demonstration data for feature expectation computation | Filter Hub datasets by robot_type before computing feature expectations; mixing Franka and xArm demonstrations in a single MaxEnt IRL pass conflates joint-range limits and infers spurious smoothness penalties tied to one arm's workspace boundary. |
Start with a small baseline that logs inputs, outputs, units, timestamps, and termination conditions before moving to Gymnasium or PettingZoo. The library run should keep the same artifact schema, so the comparison remains a same-task evaluation.
- Write a one-paragraph task contract with observation, action, success, and failure fields.
- Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
- Run one deterministic smoke test and one perturbation test before scaling.
- Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
- Compare methods only when one script evaluates them on the same task panel.
When Inverse reinforcement learning fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.
Review and Consolidation
Inverse reinforcement learning should be evaluated through four lenses: the learning objective, the robot interface, the data artifact, and the deployment failure mode. A demonstration is not a self-sufficient label; it is a trajectory sampled from an expert distribution that the learned policy will later disturb.
For inverse reinforcement learning, the workflow is reward identification: define candidate features, infer reward weights from demonstrations, validate the induced policy, and test whether the recovered reward predicts held-out preferences.
IRL demonstrations are evidence about objectives, not only actions. The contract must state which costs, constraints, preferences, and nuisance correlations are observable enough to infer.
| Agent Lens | Question To Answer | Concrete Evidence |
|---|---|---|
| Conceptual scope | What does IRL add beyond behavior cloning, and where does it fit in Part V? | A definition, a worked example, and a failure case tied to the perception-action loop. |
| Code and tools | Which maintained tool removes boilerplate after the from-scratch baseline? | LeRobot, robomimic, DAgger, behavior cloning, dataset aggregation evaluated against the same task contract. |
| Data and evaluation | What distribution produced the behavior, and where can it break? | Train, validation, and stress splits with explicit robot, camera, timing, and license metadata. |
| Publication quality | Can the reader reproduce the claim without hidden context? | Captions, bibliography cards, cross-links, and a same-artifact audit trail. |
Do not claim that inverse reinforcement learning improves robot learning unless the baseline and the proposed method share the same robot, task split, reset distribution, success metric, and random seed policy. Otherwise the comparison may be measuring dataset difficulty rather than method quality.
Modern imitation systems should be audited as synchronized robot data: images, proprioception, language, actions, timing, operator metadata, and covariate-shift checks.
Who: A field-robotics researcher inferring navigation preferences from expert driving traces.
Situation: The engineer needs to decide whether inverse reinforcement learning is ready for a weekly policy comparison across 120 demonstrations and 30 held-out rollouts.
Decision: For Inverse reinforcement learning, keep the minimal imitation baseline and compare LeRobot or robomimic only on the same manifest, split, seed policy, and rollout evaluator.
Result: The artifact links demonstrations to feature weights, induced trajectories, held-out preference checks, and counterexamples where the reward gives the wrong tradeoff.
Lesson: IRL earns trust when the recovered objective predicts behavior and rejects spurious shortcuts, not merely when it reproduces a trajectory.
Before leaving this section, write one sentence that links inverse reinforcement learning to each of these connected chapters: Chapter 14: Reinforcement Learning Refresher, Chapter 23: Teleoperation and Data Collection, Chapter 34: Vision-Language-Action Models. If any link feels forced, the section needs a sharper boundary or a clearer prerequisite recap.
Inverse reinforcement learning is useful when it makes the perception-action loop more reliable, not when it merely adds a more impressive model name.
Design a method-matched experiment for Inverse reinforcement learning. Specify the environment, observation schema, action interface, metric, and one perturbation that targets the section's core assumption.
Project Ideas
Beginner (weekend): GridWorld MaxEnt IRL with Gymnasium. Build a small 5x5 grid environment in Gymnasium, record 20 hand-crafted expert trajectories that avoid a penalty cell, and run maximum entropy IRL using the imitation library to recover the reward weights. The key challenge is verifying that the recovered reward assigns lower returns to shortcut paths that skip the penalty region, which requires implementing the counterfactual ranking check from Code Fragment 3.
Intermediate (1 to 2 weeks): GAIL on a MuJoCo manipulation task with transfer test. Collect 50 demonstrations of FetchPush-v2 in MuJoCo using a scripted oracle, train a GAIL policy with the imitation library, then re-evaluate the discriminator reward on a variant environment where the cube mass is doubled. The key challenge is separating reward features that transfer (object position relative to goal) from those that do not (joint torque profiles tied to the original mass), which requires logging discriminator scores on both the source and transfer rollouts and comparing them against behavior cloning as a baseline.
Lab: Recover a GridWorld reward and break it with a counterfactual
Goal. Run maximum-entropy IRL on a small grid, then test whether the recovered reward survives a trajectory the expert never demonstrated.
Tools needed. Python with numpy and gymnasium; optionally the imitation library for a higher-level API. Budget 15 to 30 minutes.
Setup. Build a 5x5 grid with one goal cell (+1 feature) and one lava cell to avoid. Generate roughly 20 expert trajectories from start to goal that route around the lava. Use two features per state: distance-to-goal and lava-proximity. Run the MaxEnt update from the algorithm above until \(\hat{\mu}_E\) and \(\hat{\mu}_\theta\) agree.
What to vary. (1) The number of expert demonstrations, from 3 up to 50. (2) Whether the lava feature is included in the feature map at all. (3) The learning rate \(\alpha\).
What to observe. Plot the recovered weight on the lava feature as demonstration count grows: with too few demos the weight is near zero (ambiguous). Then construct a shortcut trajectory that cuts through the lava cell and score it under the recovered reward. With the lava feature dropped, the shortcut scores higher than the expert path, exactly the reward-ambiguity failure this section warns about. Confirm that adding the lava feature plus enough demonstrations makes the recovered reward rank the shortcut below the expert.
What's Next
This section grounded inverse reinforcement learning in an explicit robot-data contract: observations, actions, demonstrations, evaluation splits, and failure labels. The next reading step is Section 21.5, where the same contract is carried into the next technique or chapter.
This paper introduces DAgger, the standard fix for covariate shift in sequential imitation learning. Read it when behavior cloning fails after the policy visits states that the demonstrator rarely produced.
Pomerleau, D. (1989). ALVINN: An Autonomous Land Vehicle in a Neural Network. NeurIPS.
ALVINN is an early example of learning control from demonstrations and sensor inputs. It helps readers see that imitation learning's central distribution problem predates modern deep robot policies.
Mandlekar, A. et al. robomimic: A Framework for Robot Learning from Demonstration.
robomimic gives reusable datasets, baselines, and evaluation scripts for demonstration-based manipulation. It is the right tool when a section needs a reproducible behavior cloning or offline imitation baseline.
Hugging Face. LeRobot: Making AI for Robotics More Accessible.
LeRobot standardizes models, datasets, and training utilities for real-world robotics in PyTorch. It is especially useful for connecting small demonstration experiments to shared dataset formats on the Hugging Face Hub.
robomimic v0.1 Datasets Documentation.
The dataset documentation shows how demonstrations, task metadata, and evaluation splits are packaged for reproducible robot learning. Practitioners should read it before inventing a custom data layout.