Section 58.2: Generalist vs. specialist policies

"I am a generalist until the gripper asks for millimeters."

A Broad Policy At A Precision Test

The same robot that can name a screwdriver, describe its uses, and narrate a repair plan will strip the fastener head because it missed by four millimeters: by the end of this section you should be able to decide, for a given deployment, whether to run a single generalist policy, a set of specialists, or a router that blends them, and to justify that choice with a measurable contract rather than peak accuracy alone.

A set of specialists (no router at all, each deployed behind its own task selector or operator switch) is the right call precisely when the router itself would be the least reliable part of the system: a small, fixed menu of known tasks, a human or upstream planner that already knows which task is next, and a precision requirement high enough that even a well-tuned router's occasional misroute is unacceptable. The routing math worked through below tells you when a learned router earns its keep; when it does not, falling back to a manually selected specialist per task is often the safer, if less automated, choice.

A policy broad enough to handle anything is, by that same budget, too diffuse to handle any one thing perfectly: breadth and precision are not free to coexist inside a fixed parameter count. Figure 58.2A makes this tradeoff concrete: a generalist holds roughly 62 percent average success across the task suite while any single specialist reaches 91 percent on its one task.

The technical contract for generalist vs. specialist policies becomes a usable mental model in three moves: define the object of study, connect it to the agent loop, then test it with a compact implementation.

The four questions every policy contract answers

The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions? Figure 58.2B traces how those questions map onto a running system: an observation enters a router, which dispatches either the generalist or the specialist, and both resolve to a single action command.

Theory

The generalist-specialist tradeoff has a concrete physical basis. A vision-language-action model such as RT-2 or OpenVLA encodes observations through a large vision transformer (ViT-B or ViT-L, standard image-encoder sizes; "B" and "L" denote Base and Large parameter counts) shared across all tasks, then decodes actions as token sequences. The shared encoder spreads representational capacity across the full distribution of training scenes, so it cannot concentrate that capacity on any one workspace region or contact geometry. A specialist policy trained only on Franka Panda insertion tasks works differently. It dedicates its entire network capacity to the sub-centimeter pose variations that peg-in-hole demands. A striking consequence typically follows, in reported peg-in-hole benchmarks,: a specialist MLP trained on roughly 300 peg-in-hole demonstrations can match the insertion success rate of a generalist VLA that consumed 50,000 diverse trajectories, because the specialist never wastes a weight on recognizing soup cans or parsing language. It learns a tight, calibrated mapping from wrist-camera depth to fingertip delta-pose, where a delta-pose is a small incremental change in position and orientation applied to the end-effector each control step, typically at 200 Hz with under 2 ms inference latency on a single GPU in these small-MLP settings; exact figures vary with sensor rate and hardware. The generalist runs at 10-30 Hz because it tokenizes image patches and runs an autoregressive decoder (it generates each output action token conditioned on the tokens it already produced, one step at a time, rather than emitting the whole action in one shot). The specialist can be a compact residual MLP that reads directly from a calibrated force-torque sensor. These are not arbitrary implementation choices; they follow directly from where each model spends its representational budget.

Worked Example

That budget argument stays abstract until it is pinned to a single contact, so the rest of the section follows one rollout where the millimeters are visible. Keep one concrete rollout in view: a Franka Panda inserting a 6 mm peg into a 6.2 mm hole. The wrist-camera depth reading becomes a peg-tip pose estimate, that estimate constrains the next delta-pose command, the command drives the peg 2 mm toward the hole, and the next frame plus the force-torque reading confirm whether the peg seated or jammed against the chamfer (the angled lead-in edge machined around the hole that either guides a slightly misaligned peg into place or catches it and stops the insertion). A generalist VLA like OpenVLA and a specialist residual MLP both run this exact loop; they differ only in where the 4 mm of estimation error goes. The section's idea is useful only if it tightens that loop at the contact step.

Consider a specific case. RT-2 (a generalist VLA fine-tuned on ~130k robot trajectories, as reported in Brohan et al., 2023) achieves roughly 62% success on a 17-task manipulation suite, while a specialist policy trained only on a single pick-and-place task reaches 91% on that one task. This is the precision-coverage cliff, and it appears in every comparison between broad and narrow policies. The generalist wins on coverage, since it handles novel object categories without retraining, but loses by ~29 percentage points on the narrow task where the specialist has dense supervision. Now add a routing rule that sends pick-and-place requests to the specialist and all other queries to the generalist. It yields ~85% aggregate success with a 12 ms routing overhead per query. That tradeoff pays off only because the specialist task is frequent enough to justify maintaining a second deployment artifact. If the specialist task represents fewer than 10% of queries, the routing complexity rarely pays off.

# Simulate generalist vs. specialist policy routing and measure aggregate success
import numpy as np

rng = np.random.default_rng(42)

N_TASKS = 5_000          # total evaluation rollouts
SPECIALIST_TASK_FRAC = 0.15   # fraction of queries the specialist handles
GENERALIST_ACC = 0.62         # generalist accuracy across all tasks
SPECIALIST_ACC = 0.91         # specialist accuracy on its target task
MISROUTE_RATE = 0.06          # fraction of queries misrouted by the router

# Task labels: 1 = specialist task, 0 = generalist task
task_labels = rng.binomial(1, SPECIALIST_TASK_FRAC, size=N_TASKS)

# Outcome under plain generalist (no routing)
gen_outcomes = rng.binomial(1, GENERALIST_ACC, size=N_TASKS)

# Outcome under hybrid router
routed_outcomes = np.empty(N_TASKS, dtype=int)
for i, label in enumerate(task_labels):
    # Decide which policy the router selects
    is_misrouted = rng.random() < MISROUTE_RATE
    router_choice = label if not is_misrouted else (1 - label)
    if router_choice == 1:
        routed_outcomes[i] = rng.binomial(1, SPECIALIST_ACC)
    else:
        routed_outcomes[i] = rng.binomial(1, GENERALIST_ACC)

gen_success = gen_outcomes.mean()
hybrid_success = routed_outcomes.mean()
router_pays_off = hybrid_success > gen_success

print(f"Tasks evaluated : {N_TASKS}")
print(f"Specialist-task fraction : {SPECIALIST_TASK_FRAC:.0%}")
print(f"Generalist-only success : {gen_success:.3f}")
print(f"Hybrid-router success : {hybrid_success:.3f}")
print(f"Delta (hybrid - gen) : {hybrid_success - gen_success:+.3f}")
print(f"Router earns its cost : {router_pays_off}")

Tasks evaluated : 5000
Specialist-task fraction : 15%
Generalist-only success : 0.620
Hybrid-router success : 0.668
Delta (hybrid - gen) : +0.048
Router earns its cost : True

On a physical robot, misrouting carries immediate consequences beyond a dropped success rate. If the generalist policy takes control during a peg-in-hole insertion step that requires 1 mm tolerance, its smoothed, high-latency output can drive the end-effector into the fixture wall before the router corrects the error. Damage, wedged parts, or a safety stop result. This is why router latency and authority-transfer timing belong in the deployment contract alongside accuracy metrics: a misroute that resolves in 50 ms is qualitatively different from one that persists for a full 500 ms control cycle.

Because those physical consequences hinge on how fast the routing decision resolves, the router's own design is dominated by one constraint: it must be cheap enough to commit before any damage window opens. A lightweight router is typically a small classifier, often a linear probe or shallow MLP, trained on the same observation embedding the generalist uses. It reads the current task descriptor and a short observation window, then outputs a policy index. Crucially, the router must commit before the generalist begins its autoregressive decode, so the latency budget for routing is bounded by the generalist's first token time, usually 30 to 80 ms on a single GPU. Routers that exceed that budget force a stale-policy step, negating the precision gain.

Before reading on, guess: if your router misclassifies just 6% of queries and your specialist only outperforms the generalist by 5 percentage points, is the hybrid system better or worse than using the generalist alone?

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.