Section 12.4: Household and long-horizon: BEHAVIOR-1K / OmniGibson

"Picking up a cup is a skill. Clearing the table, loading the dishwasher, and wiping the counter are a task. The household benchmark only cares about the task."

A Long-Horizon Scene Designer

What This Section Builds

Ask a robot to make breakfast and it must cross a kitchen, open a refrigerator, crack eggs, find a pan, and set a table, twenty-odd actions where dropping a single mug at step nineteen erases everything a binary scoreboard would record. This section makes household and long-horizon benchmarks operational for exactly that case, focusing on tasks whose success depends on object states, room layouts, navigation, manipulation, and ordered subgoals rather than one isolated grasp. Throughout, a predicate means a testable boolean condition over simulator state, such as OnTop(mug, shelf) or Open(cabinet); a long-horizon task is satisfied when an ordered sequence of these predicates all become true, and predicate progress is simply the fraction that are satisfied. As Figure 12.4A illustrates, a single unfinished step can block final task success even when most predicates are already met.

The goal is to avoid a misleading binary score. A household policy can fail the final task while making meaningful predicate progress, or pass a task through a shortcut that ignores the intended behavior. The evaluation artifact must record both final success and subgoal evidence.

Theory

A long-horizon household benchmark is a predicate sequence over a simulated home. The policy must move through rooms, manipulate objects, change states, and satisfy a goal condition: objects placed, cleaned, opened, or arranged. The aggregate score should therefore include final success, predicate completion, path or action efficiency, and failure point. On BEHAVIOR-1K, the best published end-to-end policy at the 2023 release cleared fewer than 10 percent of tasks to full completion, yet predicate progress reached roughly 40 percent. The same robot that looks like a near-zero performer under binary scoring has actually satisfied four out of ten subgoals. To feel the difference, consider the evaluation budget. As a rough illustration rather than a measured figure, a researcher using binary success might need on the order of tens of thousands of episodes to detect a reliable improvement between two policies at that performance level, while predicate progress, which carries more signal per episode, could plausibly cut that requirement by roughly two orders of magnitude. The exact ratio depends on the variance of the underlying policies and the statistical test used, but the direction of the effect, fewer episodes needed when scoring partial credit, holds robustly. This gap is what makes predicate-level partial credit indispensable for household benchmarks, as Figure 12.4B shows directly below.

Partial credit only means something, however, if the predicates are evaluated in homes the policy has never optimized against, which makes the train-test split as load-bearing as the scoring rule itself. The split must protect scene and object generalization. If train and test share the same layouts, object placements, or task templates, a policy may learn household shortcuts rather than robust behavior. If the same simulator seed appears in tuning and testing, the result is no longer a clean held-out household evaluation.

Scene splits matter for real robots because home interiors vary along dimensions a simulation cannot fully enumerate. Cabinet heights, floor friction, lighting, and clutter density all shift the sensorimotor demands. A policy that memorizes a training kitchen's layout navigates confidently to a refrigerator in the same corner every episode, then fails immediately in a real home where the refrigerator sits across the room. Without a held-out scene split, benchmark scores measure recall of a fixed floor plan, not transferable household behavior.

Mechanically, a scene split in OmniGibson assigns each of the 50 furnished homes to a train, validation, or test partition before any policy training begins. Object placements, room connectivity, and initial states are then sampled only from the partition appropriate to each phase. Test scenes are never loaded during training or hyperparameter search, so a policy evaluated on the test partition encounters floor plans and object arrangements it has not seen, isolating spatial generalization from task-level learning.

Before reading the algorithm, think through this: if a robot completes 9 out of 10 subgoals but knocks the last object off the shelf at step 9, how should the score change compared to a robot that never leaves the starting position? Your intuition here will tell you whether a binary score or a predicate trace gives the more useful signal.

Worked Example

Code Fragment 1 turns long-horizon evaluation into predicate accounting. The key idea is that final success and partial progress should be co-computed from the same replay, not reconstructed from separate logs.

# Score household progress from a single ordered predicate trace.
# Final success and partial progress come from the same episode,
# which prevents mixing incompatible logs in one result table.
required_predicates = ("find_mug", "open_cabinet", "place_mug", "close_cabinet")
completed_predicates = ("find_mug", "open_cabinet", "place_mug")

completed = set(completed_predicates)
progress = sum(predicate in completed for predicate in required_predicates) / len(required_predicates)
final_success = all(predicate in completed for predicate in required_predicates)

print(f"progress={progress:.2f}, final_success={final_success}")

Practical Recipe

1. Define the household construct: navigation plus manipulation, object-state reasoning, task planning, recovery, or full long-horizon completion.
2. Freeze scene layouts, object instances, task templates, initial-state seeds, action horizon, and predicate definitions.
3. Report final success together with predicate progress, failure step, action count, and recovery events.
4. Stratify results by task length and scene novelty so short tasks do not hide long-horizon failures.
5. Save replays that show the object-state predicates changing over time.

Preserving the Causal Story

Household and long-horizon benchmarks become useful when they preserve the causal story of the episode. The policy may find the object, move it, change its state, and still fail because a cabinet remains open or a target condition is not satisfied. A single final success number loses that structure.

The graduate-level habit is to co-compute final success, progress, and failure labels from one replay. A method that improves progress on hard scenes has a meaningful result even when final completion remains hard. A method that improves final success by exploiting easy templates needs a narrower claim.

A robust household evaluation starts by writing the predicate schema. The schema should list the subgoals, the state variables that make each predicate true, and the time step at which each predicate is evaluated. That schema becomes the bridge between long-horizon behavior and a reproducible metric.

1. Write the predicate schema and final success rule before running policies.
2. Freeze scene, object, task-template, and initial-state splits.
3. Run every method with the same horizon, action interface, and predicate evaluator.
4. Save per-predicate completion, final success, failure step, action count, and replay path.
5. Aggregate by task length and scene novelty, not only by a global mean.

Code Fragment 2 records a household evaluation result with both progress and final success. This lets the paper say exactly what improved.

# Record household evidence with scene split and predicate progress.
# Long-horizon results need the failure step and replay path because
# final success alone does not explain where the episode broke.
from dataclasses import dataclass, asdict

@dataclass
class HouseholdResult:
    suite: str
    scene_split: str
    task_template: str
    progress: float
    final_success: bool
    failure_step: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

result = HouseholdResult(
    suite="BEHAVIOR-1K",
    scene_split="unseen_homes",
    task_template="put_away_tableware",
    progress=0.75,
    final_success=False,
    failure_step="close_cabinet",
)
print(result.as_row())

Expected output: the printed result should expose scene split, task template, progress, final success, and failure step. Without those fields, a household benchmark hides the structure of the episode.

When a household experiment fails, inspect the first unsatisfied predicate. It points to the subsystem at fault: a never-found object implicates perception and navigation, a found-but-unchanged object implicates manipulation and physics, and a changed state that still misses the goal implicates the evaluator and task definition.