Section 26.2: The options framework

"An option is an action that knows when it has finished. That self-knowledge is what makes planning across time possible."

Section 26.2

Concretely, by the end of this section you will be able to do three things with the options framework rather than merely recognize its name: (1) write the triple \((I, \pi, \beta)\) for a candidate robot skill and check each field against the audit checklist below; (2) compute the SMDP Q-value of an option by hand, as the worked step-through does for a three-step grasp; and (3) diagnose a composition failure where one option's postcondition does not satisfy the next option's initiation set, as the Common Pitfall callout demonstrates.

Why Hierarchy Matters

Ask a flat planner to fetch a mug from the kitchen and it must commit to thousands of individual motor twitches up front; ask a planner built on options and it commits to four words, "go," "open," "grasp," "return," letting each skill privately negotiate its own timing, contact, and recovery while the planner reasons only about the seams between them.

Options are the mathematical bridge between reinforcement learning and robot skills. They let a high-level policy choose temporally extended actions while the low-level policy handles many primitive steps before returning control.

A skill that cannot say when it is finished is not a skill; it is an open-ended hope delegated to the planner.

To make that interface precise enough to plan with, the next step is to pin each of those promises down as a formal object the high-level policy can reason about.

Formal Contract

Figure 26.2A above shows the three parts of an option side by side: the initiation set that gates where it may start, the policy that runs while it executes, and the termination condition that ends it and composes the whole into a semi-MDP over skills.

An option \(\omega\) is a triple

$$\omega = (I,\, \pi,\, \beta),$$

where \(I \subseteq \mathcal{S}\) is the initiation set (the states from which \(\omega\) may be selected), \(\pi: \mathcal{S} \to \Delta(\mathcal{A})\) is the intra-option policy (the primitive-action distribution the skill follows while running), and \(\beta: \mathcal{S} \to [0,1]\) is the termination function (the probability of handing control back to the high-level policy in each state). The option executes for a random duration \(\tau\) determined by \(\beta\): at each step, it terminates with probability \(\beta(s_t)\) and continues otherwise. The resulting process is a semi-MDP (SMDP) at the high level, because transitions take variable time.

Why variable duration breaks the plain MDP

The variable duration matters for physical robots. Actuator cycles, sensor latency, and contact dynamics mean that a "grasp" option may take 0.3 seconds on a clean approach or 2.1 seconds after a slip recovery. A standard Markov Decision Process (MDP) assumes every action takes exactly one time step. It therefore mis-discounts rewards from slow skills and mis-values fast ones, and the planner then prefers fragile shortcuts over robust options. The semi-MDP resolves this: it treats each option as a single abstract transition whose reward and discount integrate over however many real steps the option consumed.

Mechanically, the SMDP pauses the high-level clock during option execution. The intra-option policy \(\pi\) runs at the primitive rate while the high-level agent waits and accumulates discounted reward \(\sum_{k=0}^{\tau-1}\gamma^k r_{t+k}\). When \(\beta(s_t)\) fires, that return becomes a single temporal-difference target (the observed value used to correct the current estimate of \(Q_\Omega\) toward, one update at a time) for \(Q_\Omega\), and the planner selects the next option, restarting its clock from the termination state.

The SMDP Q-function over options values starting state \(s\) and selected option \(\omega\):

$$Q_\Omega(s,\omega)=\mathbb{E}\left[\sum_{k=0}^{\tau-1}\gamma^k r_{t+k}+\gamma^\tau V_\Omega(s_{t+\tau})\mid s_t=s,\omega_t=\omega\right].$$

Here \(\tau\) is the (random) option duration, the discounted sum accumulates primitive rewards over the option's execution, and \(V_\Omega(s_{t+\tau})\) is the value at the state where the option terminates. The high-level policy selects from \(\Omega\) (the set of all options) at every option boundary, not at every primitive time step. Use this tuple as an audit checklist when designing a skill: if any field of \((I, \pi, \beta)\) is implicit or missing, the option will fail silently at task boundaries.

Consider a concrete case: the Option-Critic architecture (Bacon et al., 2017), a method that learns the initiation set, policy, and termination function of each option jointly with gradient descent instead of hand-specifying them, applied to a four-room navigation task. Each room transition becomes a natural option, with \(\beta(s) \approx 1\) at doorways and \(\beta(s) \approx 0\) in open floor space. With 8 options and a discount of \(\gamma = 0.99\), the high-level policy makes roughly 4 decisions to cross a 40-step trajectory instead of 40 primitive choices. In this kind of four-room setting, option-based learners typically converge within a few hundred thousand environment steps, while a flat Q-learner on primitive actions often needs an order of magnitude more steps to match the same policy quality; the exact ratio depends on room size, option count, and reward shaping. The SMDP Q-function correctly discounts the 10-step room-crossing sub-trajectory with \(\gamma^{10} \approx 0.90\), so the planner still prefers shorter options when two paths reach the same goal. Options provide temporal compression of decisions: they reduce planning cost from one choice per primitive step to one choice per meaningful sub-goal. The framework pays off when primitive time steps are cheap but planning decisions are expensive, and when natural sub-goals (doorways, grasp contacts, lane-change completions) define clean termination states.

The triple and its semi-MDP return are abstract until you watch them collapse a burst of control ticks into a single outcome, so the next fragment runs exactly that compression in code.

Worked Implementation

Code Fragment 1 for The options framework should expose initiation, progress, termination, verification, and failure reporting before connecting the skill to ROS 2, BehaviorTree.CPP, Drake, or a learned policy.

# Simulate an option that moves through several primitive control ticks.
# The high-level controller sees one option outcome, not every inner action.
def execute_option(start_position, goal_position, max_steps=5):
    position = start_position
    trace = []
    for step in range(max_steps):
        delta = min(0.4, goal_position - position)
        position += delta
        trace.append(round(position, 2))
        if abs(goal_position - position) < 0.05:
            return "terminated", trace
    return "timeout", trace

status, trace = execute_option(0.0, 1.0)
print(status, trace)

The expected output trace shows why the option is useful to a high-level planner: three internal control updates compress into one symbolic result. The final position reaches the goal within tolerance, so the option terminates instead of exposing each intermediate motor step.

That verification loop only pays off if the skills feeding into it are named, bounded, and composed with discipline, which is what the following recipe lays out step by step.

Practical Recipe

1. Name each skill with a verb and object: navigate_to_station, grasp_handle, dock_drone, or change_lane.
2. Write preconditions, postconditions, safety guards, timeout, and recovery behavior before training a policy.
3. Represent sequencing as a finite-state graph, behavior tree, or task-and-motion plan so failures have explicit routes.
4. Use language as a planner only after commands are grounded into a typed skill library with affordance checks.
5. Evaluate composition, not only individual success. Many failures occur when two correct skills meet at a bad boundary.