Section 12.5: Navigation and social: Habitat 3.0, AI2-THOR / ProcTHOR

"Getting to the goal is one metric. How much space you took from others on the way there is a second metric most navigation papers still omit."

A Social Navigation Critic

What This Section Builds

Two robots reach the same kitchen one second apart, yet only the dashboard tells you that the faster one clipped a humanoid's shoulder on the way and the slower one yielded the doorway: this section is about why navigation benchmarks that report only arrival time hide exactly the number that decides whether a robot is welcome in a shared hallway. It distinguishes pure navigation success from efficient navigation, rearrangement progress, and social behavior such as following a humanoid while maintaining safe distance, as Figure 12.5A previews: a navigation score should not hide the route taken, the generated scene seed, or the cost paid in social safety.

The goal is to keep path metrics, scene splits, generated-house seeds, and human-interaction rules attached to the result. Without those fields, a leaderboard row can hide whether a policy learned navigation, memorized layouts, or benefited from easier generated homes.

By the end of this section you should be able to do four concrete things: compute success weighted by path length (SPL) by hand from a small episode table; explain why non-overlapping seed ranges are what let ProcTHOR test generalization rather than memorization; read the audit-fields table to decide which platform (Habitat, Habitat 3.0, AI2-THOR, or ProcTHOR) fits a given claim; and, given two navigation results, name the protocol fields that must match before the numbers can be compared on a single leaderboard row.

Theory

This section builds the SPL formula first, then the social-distance constraint, then the seed-split mechanism that makes ProcTHOR results trustworthy; each later piece leans on the one before it, so read them in order rather than jumping to the worked example.

Navigation benchmarks often report success and path efficiency together. If the shortest path length is \(L\) and the agent path length is \(P\), success weighted by path length is \(S \cdot L / \max(P, L)\), where \(S\) is 1 for success and 0 for failure. This prevents a policy from getting full credit for eventually reaching the goal through an inefficient route.

Social navigation adds another constraint: the robot should achieve its goal without crowding, blocking, or colliding with the humanoid (a simulated human-shaped avatar that walks the scene alongside the robot, either scripted or driven by a learned policy) or human collaborator. Habitat 3.0 makes this explicit through collaborative tasks such as social navigation and social rearrangement, while AI2-THOR and ProcTHOR emphasize interactive indoor environments and procedurally generated houses. ProcTHOR's scale illustrates what generalization through procedural generation buys in practice. Train a policy on 10,000 generated houses and test it on a held-out generated panel, and it typically succeeds on unseen layouts far more often than the same policy trained on the 120 fixed AI2-THOR apartments (as of the 2022 ProcTHOR release; Deitke et al., 2022). With only 120 scenes, the agent tends to memorize structural regularities instead of learning to navigate. The gap shows up in training cost too, in the numbers the ProcTHOR authors report: the fixed AI2-THOR apartment set demands roughly 50 million steps to reach competent object-navigation success, whereas ProcTHOR's generated pool crosses the same threshold in around 8 million steps, a difference attributed to its variety forcing genuine layout reasoning rather than route recall.

Figure 12.5B places each platform in the social-safety versus scene-generalization space. Habitat 3.0 adds the social-constraint layer, while ProcTHOR extends generalization through procedural scale. Procedural generation matters for physical deployment: a real building the robot has never seen shares no memorized layout with its training environments. A policy that overfit to 120 fixed floor plans will stall at an unexpected corridor junction or misjudge a room's exit. The robot then spins, times out, or blocks doorways, all dangerous and costly to recover from.

ProcTHOR generates houses by sampling room counts, sizes, and connectivity from a grammar, then placing furniture and objects from a curated asset library according to room-type rules. The generator is seeded and versioned, so a held-out eval panel can be reconstructed exactly. Training and evaluation panels are drawn from non-overlapping seed ranges, which is what prevents the policy from memorizing specific floor plans.

Worked Example

Code Fragment 1 computes success weighted by path length for three episodes. The same pass also keeps collisions visible, because a socially unsafe route should not be celebrated as an efficient route.

# Compute navigation efficiency from the same episode table as safety.
# Success weighted by path length rewards reaching the goal efficiently,
# while collision counts keep unsafe routes visible.
episodes = [
    {"success": 1, "shortest": 6.0, "actual": 7.5, "collisions": 0},
    {"success": 1, "shortest": 5.0, "actual": 12.0, "collisions": 2},
    {"success": 0, "shortest": 8.0, "actual": 10.0, "collisions": 1},
]

spl_values = [
    ep["success"] * ep["shortest"] / max(ep["actual"], ep["shortest"])
    for ep in episodes
]
mean_spl = sum(spl_values) / len(spl_values)
total_collisions = sum(ep["collisions"] for ep in episodes)

print(f"mean_spl={mean_spl:.2f}, total_collisions={total_collisions}")

Practical Recipe

1. Choose the construct: point navigation, object navigation, rearrangement, social navigation, or social rearrangement.
2. Freeze scene split, generated-house seed list, target sampling, sensor suite, action horizon, and path-length normalization.
3. Report success, path efficiency, collisions, timeout rate, and social-distance violations from the same evaluation pass.
4. For ProcTHOR-style generation, save generator version and house seeds so the panel can be reconstructed.
5. For Habitat 3.0-style social tasks, record humanoid policy, interaction mode, and safety threshold.

Navigation and social benchmarks become useful when they preserve both route quality and interaction quality. A Habitat 3.0 social-navigation result should not report only whether the robot found and followed the person. It should also report whether the robot maintained safe distance, avoided blocking, and completed the task under the same humanoid behavior model as the baseline.

The graduate-level habit is to separate map competence from social competence. AI2-THOR and ProcTHOR stress generalization across interactive scenes and generated homes; Habitat 3.0 stresses collaboration with humanoid or human behavior. A paper-facing comparison must name which competence it measures and which scene or behavior distribution it held out.

Those audit fields describe what a careful evaluation should record; the published record shows how rarely the social half of them actually appears.

Before reading on, guess: of the policies that appear in a top-10 navigation leaderboard slot, what fraction also report social-violation rate from the same episode panel? Across published Habitat and AI2-THOR leaderboard entries through 2024, fewer than 1 in 10 rows include both SPL and social-violation rate co-computed in a single pass, which means the social safety of most "high-performing" navigation policies is simply unknown.

Building the Episode Panel

A robust navigation evaluation starts with the episode panel. The panel should list each scene or generated house, start and target, shortest path length, seed, humanoid behavior if present, and the metric rule. Every method should consume that panel unchanged.

1. Write an episode panel with scene ID, generated-house seed, start, target, and shortest path length.
2. Freeze sensors, action space, success radius, collision rule, horizon, and social-distance threshold.
3. Evaluate every method through one script that computes success, efficiency, collisions, and social violations.
4. Save per-episode replays and stratify results by seen versus unseen scenes or generated houses.
5. For social tasks, report both task completion and human-aware safety metrics.

Code Fragment 2 records the navigation protocol fields that make a result replayable. The same schema works for a static navigation panel or a ProcTHOR-generated panel.

# Record a navigation result with path and social metrics together.
# Co-computing these fields prevents a method from optimizing route
# efficiency while hiding collisions or personal-space violations.
from dataclasses import dataclass, asdict

@dataclass
class NavigationResult:
    suite: str
    scene_split: str
    generated_seed_panel: str
    success_rate: float
    mean_spl: float
    social_violations_per_episode: float

    def as_row(self) -> dict[str, object]:
        return asdict(self)

result = NavigationResult(
    suite="Habitat 3.0",
    scene_split="unseen_homes",
    generated_seed_panel="proc_panel_v2",
    success_rate=0.74,
    mean_spl=0.51,
    social_violations_per_episode=0.18,
)
print(result.as_row())

Expected output: the printed result should expose scene split, generated-seed panel, success, path efficiency, and social violations. If one field changes between methods, the comparison should stay in diagnostics rather than the paper table.

Recording those fields is only half the discipline; the other half is knowing what to do with them once a run comes back wrong.

When a navigation or social experiment fails, replay the path and tag the first failure mode: wrong target, inefficient route, collision, timeout, blocked humanoid, social-distance violation, object-state miss, or generated-scene mismatch. The tag tells you whether to fix mapping, planning, interaction policy, or the evaluation panel.