Section 59.2: Language-guided navigation with replanning

"Turn left at the vague instruction, then replan before dignity runs out."

A Navigation Policy In A Blocked Hallway

Your robot is forty seconds into "go to the kitchen and avoid the wet floor near the sink" when a cleaning cart rolls across the only corridor and the plan dies on the spot: does the agent recover the user's intent, or does it quietly forget the sink and barrel toward it? That single moment is what this section builds a system to survive. The path runs from a defined object of study, to its place in the agent loop, to a compact implementation that tests it. Figure 59.2A sketches the full loop you will build: instruction parsing into waypoints, segment execution under local obstacle avoidance, and an LLM replanner that fires only when the agent detects it is lost or blocked.

The pipeline diagram above traces the four stages this contract must specify: the LLM parser turns the instruction into (waypoint, constraint) pairs, the metric planner executes them against the occupancy grid, the replan trigger fires when path cost exceeds 1.5x the best alternative (in other words, when the current route has become at least 50% more expensive than the cheapest known alternative), and the LLM replanner emits revised symbolic constraints that feed back into the planner without ever touching the grid directly. The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

For language-guided navigation with replanning, pin the interface between the language model and the costmap planner (where a costmap is a grid whose cells carry a traversal cost, high near obstacles and low in open space, that a planner minimizes over) before any optimization begins. The LLM outputs a list of (waypoint, constraint) pairs in map-frame coordinates. The Nav2 NavfnPlanner (the ROS 2 Navigation stack's default metric path planner, chosen here because it ships with Nav2 and needs no extra dependency) receives those pairs together with the current OccupancyGrid (the ROS 2 message type holding the 2-D grid of traversal costs described above as a costmap). Every replanning call must log four things: the timestamp, the LiDAR scan that triggered the check, the old and new path costs in meters, and whether each soft constraint held or relaxed. On a TurtleBot 4 running Nav2 Humble (ROS 2, 2022 LTS), the parse-to-costmap-update round trip takes roughly 80 ms at 5 Hz LiDAR. A spike above 200 ms typically signals that the language-grounding step, not the planner, is the bottleneck. The replan trigger referenced throughout this section, a path-cost ratio compared against a fixed multiple, is defined formally later in this section under Topic-Native Deepening; for now, treat it as "replan when the current route gets much more expensive than the best alternative."

Worked Example

The silent-ungrounding failure the mechanism warns about is easiest to see when you watch a single episode unfold, so the next step is to trace one concrete rollout end to end.

Keep one concrete rollout in view. A sensor reading becomes an estimate, the estimate constrains an action, the action changes the world, and the next observation confirms or contradicts the assumption. The section's idea is useful only if it improves that loop.

Tracing one rollout in scene "17DRP5sb8fy"

Consider a specific case: an agent in the Habitat MP3D scene "17DRP5sb8fy" receives the instruction "Go to the kitchen, avoid the wet-floor area near the sink, and stop by the blue fridge." The planner converts this into three waypoints: (1) navigate to kitchen centroid at grid cell (14, 22), (2) route around a 1.2 m radius exclusion zone at (16, 19), and (3) halt within 0.5 m of the object detector's "refrigerator-blue" bounding box.

At step 47 of the episode, a newly spawned obstacle blocks the corridor between the living room and kitchen. The replanner triggers because the next-waypoint distance spikes from 2.1 m to 8.4 m via the only remaining open path. The LLM replanner receives the current map, the blocked-node annotation, and the original instruction string. It then outputs a revised waypoint sequence that routes through the dining room and preserves the sink-avoidance constraint. Total replans: 1; path inflation: 34%; constraint violations: 0. This single rollout gives you a concrete artifact to inspect: which node caused the trigger, which phrase in the instruction forced the constraint into the new route, and what the inflation cost was.

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

Step two of that recipe, the simple debuggable baseline, only stays debuggable if you assign each component the right job, and the most common way teams get that wrong is by overloading the language model.

A language model that cannot see the occupancy grid is not a path planner; it is a constraint translator, and the moment you confuse the two roles, the robot walks into walls.

Grounding ambiguity matters in embodied AI because a robot cannot pause indefinitely while the language layer resolves a vague term. A delivery robot that drops "avoid the busy area" may enter the crowded corridor the constraint was meant to prevent, causing collisions or mission failure with no log entry marking the instruction as unresolvable. Unlike a software query that returns an error, a physical agent keeps moving under the degraded plan, so the cost surfaces as collisions, blocked paths, and lost user trust rather than exception messages.

The grounding mechanism works in two stages. First, a semantic map associates names and descriptions with regions: a vocabulary encoder (such as CLIP or a sentence encoder) computes a similarity score between the instruction phrase and candidate region labels stored at planning time. The region with the highest score above a confidence threshold is selected and its bounding box is converted to a set of costmap cells. Second, those cells receive an inflated cost weight, turning the spatial constraint into a number the planner can optimize against. If no region score clears the threshold, the constraint has no cell assignment and is silently absent from the costmap. Logging the threshold check explicitly at each replan is the only reliable way to detect this failure before it propagates into the trajectory.

def validate_card(payload: dict[str, object]) -> dict[str, object]:
    assert payload, "payload must not be empty"
    return payload

# Instruction-aware navigation project card.
card = {
    "instruction": "Go to the kitchen, avoid the wet floor, then stop near the blue fridge",
    "blocked_corridor": True,
    "metrics": ["success", "replans", "constraint_violations", "path_inflation"],
}
print(validate_card(card))

{'instruction': 'Go to the kitchen, avoid the wet floor, then stop near the blue fridge', 'blocked_corridor': True, 'metrics': ['success', 'replans', 'constraint_violations', 'path_inflation']}