Section 49.4: Multi-agent RL (with PettingZoo)

A team reward can be a beautiful hiding place for one very lazy policy.

A Markov Game Designer

Picture a Boston Dynamics Spot pair clearing a warehouse aisle and an iRobot-style coverage swarm: train either on a single team scalar and they fail the exact same way, with one unit quietly coasting on its partner's effort while the team score still looks healthy. Multi-agent RL becomes useful for an embodied fleet only when it is tied to a named interface (PettingZoo's AEC or Parallel API), a replayable scenario (the two-pursuer grid or a MuJoCo dual-arm box transport), a failure diagnostic (per-agent entropy and collision rate, not team return), and an artifact that records what changed in the control loop. The interface is what makes that coast measurable before the robots leave the lab.

The key question is practical: which API represents turns, simultaneous actions, observations, rewards, terminations, and per-agent metrics without hiding nonstationarity (the fact that, from any one agent's perspective, the environment's effective dynamics keep shifting because the other agents are also learning and changing their behavior)? Two interfaces answer it differently, where AEC (Agent Environment Cycle) steps agents one at a time in a defined turn order, and the Parallel API steps every agent at once in a shared control tick. The mechanical difference between these two APIs, and why picking the wrong one corrupts value estimates, is worked through in detail later in this section's "Misconception: AEC and Parallel APIs Are Interchangeable" callout; the short version needed for now is that Parallel matches robots acting in the same control tick, while AEC matches strictly turn-based interaction.

Theory

The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

Worked Example

That inspectable contract stops being abstract the moment you watch two agents try to coordinate on a single task, where every hidden assumption in the interface surfaces as a concrete behavior.

Consider a pursuit task where two agents learn to surround a target. Independent rewards can produce chasing; shared rewards can improve capture but make credit assignment harder (credit assignment: figuring out, from one team-level reward number, how much each individual agent's action actually contributed to the outcome). This is called the lazy-agent hiding problem, and it is the central diagnostic challenge in cooperative multi-agent reinforcement learning (MARL). PettingZoo makes those choices explicit in the environment interface.

Figure 49.4B shows the structural fix. Each agent acts only on its own local observation, while a centralized critic sees the global state during training alone. That is the contract the rest of this section relies on. In embodied AI, the lazy-agent problem carries physical consequences that simulation scores hide entirely. A robot that converges to a passive flanking role consumes battery, occupies floor space, and blocks corridors while contributing nothing to task completion. In warehouse fulfillment or search-and-rescue, that idle robot is not merely inefficient: it may block a critical path or fail to respond when the active agent is occluded by an obstacle. The team metric looks fine right up to the moment the scenario changes and the passive agent has no learned behavior to fall back on.

A policy trained on team scores alone is a group photo where one person has been standing still the whole time. The mechanism is a gradient mismatch at the credit-assignment step. When the team receives a shared reward, the same scalar scales every agent's policy gradient regardless of individual contribution. An agent that drifts to a high-coverage fixed position early in training then reaps positive returns from its partner's effort. Gradient updates reinforce that passive position. Entropy falls, and the policy collapses to near-deterministic behavior. That collapse happens faster than intuition suggests. In the same 10x10 pursuit task, the passive agent's entropy drops below 0.2 within roughly 800 gradient updates. Team return alone would not flag the collapse until past 15,000 updates, when the score finally degrades. Two metrics expose this before deployment: per-agent entropy and per-agent collision rate. PettingZoo's per-agent observation and reward dictionaries keep each agent's signal separate, so the collapse stays measurable.

Consider a specific case: a 10x10 grid, two pursuer agents, one evading target. With independent rewards (+1 per pursuer that reaches within 1 cell), training for 500k steps produces both agents racing to the same side of the target, colliding 40% of episodes and capturing only 55% of the time. Switching to a shared team reward (+2 split equally on joint capture) and replaying all transitions with agent identifiers raises capture to 83% in the same budget, but per-agent entropy drops from 0.7 to 0.3, signaling that one agent has settled into a fixed flanking role while the other drives. That collapse is invisible in the team score alone. Independent reward: 55% capture. Shared reward, same budget: 83%. The difference is not a better algorithm; it is a better contract between agents.

Practical Recipe

Knowing that the lazy-agent collapse hides inside a healthy team score tells you what to instrument; the following recipe turns that diagnosis into the concrete order of operations for building a fleet you can trust.

1. Define each agent's observation in physical units before writing any policy code: for a ground robot using a 2D lidar such as the RPLidar A2, that means a 360-point range scan at 10 Hz, maximum 12 m range, with a 15 ms scan-to-control latency budget. Anything that violates that budget produces stale inputs and corrupts Q-value estimates.
2. Build an independent-reward baseline in MuJoCo or Isaac Gym with the simplest possible policy (a linear controller or rule-based pursuer) so that the failure mode from a real team is visible before introducing learned policies.
3. Add PettingZoo wrappers only after the single-step AEC or parallel contract is confirmed correct: verify that observation_space bounds match the sensor range and that action_space limits match actuator torque or velocity limits (e.g., Franka Panda joint velocity capped at 2.175 rad/s per joint).
4. Record failures with a physical cause label: sensor occlusion (two robots blocking each other's lidar field), communication dropout (ROS 2 DDS packet loss above 5%), actuator saturation (motor current limit reached during simultaneous turning), or evaluation mismatch (success metric ignores contact force, so a "capture" that would damage hardware counts as a win in simulation).
5. Run at least one sim-to-real perturbation test before trusting the result: add 20 ms observation delay and 10% action noise, which approximates the latency budget of a Jetson Orin-class onboard compute stack running two concurrent vision pipelines.

Once the recipe gives you a trustworthy single-team baseline, the open questions shift from how to instrument one fleet to how teams of policies generalize, communicate, and adapt, which is where current research is moving.

If a teammate policy is silently replaced mid-deployment, how many episodes does it take before a trained agent detects the change and begins adapting? Most practitioners guess tens; in practice, partner-generalization studies typically report adaptation windows in the thousands of episodes rather than tens, which is why the open problem above matters far more than it looks on a benchmark table.

Multi-agent RL earns its place only when a closed-loop contract names the participants, observations, action authority, timing budget, logging artifact, and recovery rule. Skip that contract and the system looks capable in a notebook, then fails the first time a partner delays, a person corrects it, or the scene changes.

Separate the conceptual claim, the systems claim, and the evidence claim. A plausible mechanism, a clean interface, and a closed-loop result are different claims; the section should keep their evidence separate.

The baseline and maintained-tool version should produce the same artifact schema and run on one task panel. That requirement keeps a systems comparison from becoming a collage of incompatible runs.

1. Write a one-paragraph task contract with observation, action, success, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel.

When Multi-agent RL (with PettingZoo) fails, avoid labeling the whole method as weak. First assign the failure to perception, communication, human input, memory, planning, control, timing, data coverage, safety, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.

from pettingzoo.mpe import simple_tag_v3

env = simple_tag_v3.parallel_env(render_mode=None)
observations, infos = env.reset(seed=42)

while env.agents:
    # Replace this with each agent's policy: policy(observations[agent])
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}
    observations, rewards, terminations, truncations, infos = env.step(actions)
    # rewards is a dict like {"agent_0": 0.0, "adversary_0": 1.0}
    # keeping per-agent rewards separate is what makes the lazy-agent
    # collapse measurable instead of hidden inside one team scalar

env.close()

# Same policy family, two evaluation panels.
scores = {
    "same_partner": {"team_return": 112, "collisions": 1, "entropy": 0.42},
    "held_out_partner": {"team_return": 71, "collisions": 6, "entropy": 0.11},
}

for panel, stats in scores.items():
    print(panel, stats["team_return"], stats["collisions"], stats["entropy"])

same_partner 112 1 0.42
held_out_partner 71 6 0.11