Section 31.6: Human-agent interaction

"The robot that cannot be interrupted is not a collaborator. It is a hazard with good intentions."

A Human Factors Engineer, Mid-Incident Review

Picture a robot carrying a full pot of soup across a kitchen when a person says "not that counter, the other one": the moment between hearing those words and committing the next step is where human-agent interaction lives or dies. As Figure 31.6A shows, the human here is not a one-shot commander who walks away but a continuous partner in the control loop, and reading that loop means tracing the language input, grounding evidence, action representation, safety gate, and logged result before you trust any agent behavior described below.

Embodied agents must communicate uncertainty, accept corrections, and trade autonomy against user oversight during ongoing tasks.

The practical question is how much authority to give the robot before it must surface uncertainty or defer to the human.

Theory

Let \(u_t\) denote a human input at time \(t\), such as a command, correction, or approval. A shared-control policy can be written as $$a_t \sim \pi(a_t \mid h_t, x, u_{0:t}),$$ where the interaction history updates both task intent and trust calibration (the running adjustment of how much autonomy the agent is allowed to claim, based on its recent success and correction record; the algorithm below defines this precisely as the trust delta \(\delta_t\)). The agent should not treat all human inputs equally: a correction signal often carries more control value than a new high-level command.

Interaction quality depends on observability in both directions. The robot must observe the user's intent, but the user must also observe enough of the robot's internal state to predict what it will do next. Explanations, preview actions, and confidence signals therefore become part of the control interface, not just user-interface decoration. How those signals are designed is explored in depth in trust calibration.

Legibility as a control signal

Why bidirectional observability matters physically. A robot arm mid-swing gives no warning before contact. If the operator cannot read the robot's next intended waypoint, any correction arrives too late to prevent a collision or dropped load. Physical irreversibility is the key asymmetry: a misread email can be re-read, but a shattered object or pinched hand cannot be undone. In one household manipulation study, agents that surfaced their next waypoint 300 ms before moving needed an average of 2 operator corrections per session. Visually opaque agents running the same policy needed 17, because operators had no window in which to intervene before contact. Bidirectional observability converts a one-sided command channel into a genuine shared-control loop where the human can intervene before, not after, the cost is paid.

How it works in practice. The agent projects its next target pose onto a display or LED indicator before joint motion begins. Simultaneously, it emits a confidence score and a risk estimate so the operator can gauge whether confirmation is needed. The human's verbal or gestural response is parsed through the same intent encoder used for commands, meaning corrections update the trajectory planner's prior rather than bypassing it, keeping the physical motion smooth and within joint limits.

Worked Example

The proposal-gated algorithm above hinges on one decision step, the choice to execute or confirm, so it helps to see that step isolated as runnable code before trusting the full loop.

Code Fragment 1 shows a simple interaction gate that chooses between direct execution and confirmation. The policy uses both uncertainty and action risk, because even a confident proposal may deserve review if the consequence is expensive.

# Ask for confirmation when uncertainty or action risk is high.
# Human interaction is a control channel, not just a cosmetic interface.
# The gate should consider both confidence and consequence.
proposal_confidence = 0.58
action_risk = 0.72

need_confirmation = proposal_confidence < 0.7 or action_risk > 0.6
decision = "confirm" if need_confirmation else "execute"

print({"confidence": proposal_confidence, "risk": action_risk, "decision": decision})

Practical Recipe

1. Expose the next Cartesian waypoint or joint-space target on a shared display or wrist-mounted LED ring (e.g., the Franka Panda's joint torque indicators) before the arm begins moving, so the operator can sanity-check the trajectory without reading log files.
2. Wire a hardware interrupt to the robot's safety controller, not only to the planning stack. On a Boston Dynamics Spot, this means the E-stop signal must reach the motor firmware within the 50 ms guaranteed by the Spot API's safety layer; a software-only "abort" that takes 300 ms can still allow an arm to collide at full velocity.
3. Treat corrections as structured sensor readings: a verbal "go left" during a pick task on a mobile manipulator updates the base pose prior in the Extended Kalman Filter (EKF), not just the final goal. Replay the correction through the same state estimator you would use for a LIDAR scan.
4. Measure intervention rate per task-minute rather than per episode, because a 90-minute household session (as in the YAY Robot study) reveals confirmation fatigue that is invisible in five-minute demos. Track separately for high-risk actions (near fragile objects, above 0.5 m) versus routine motions.
5. Tune confidence and risk thresholds on the specific kinematic envelope of the deployment robot. A threshold of 0.6 risk on a Franka Panda with 3 kg payload will not transfer without re-calibration to a UR10e at 10 kg payload, because irreversible contact damage scales with payload mass, not policy confidence.

Autonomy calibration itself is not a one-time setting but the running average of recent trust deltas: each time the agent executes without correction, \(\tau_c\) can drift slightly lower (claiming more autonomy), and each correction nudges it back up, so the threshold the gate uses today reflects how reliably the agent has performed over the last several interactions, not a fixed design-time choice.

Answering that self-check requires tracking how a plan evolves under correction, not just whether it ended in success. That points to a broader lesson about measurement. Human-agent interaction shows why embodied AI cannot be judged by single-episode reward alone. The human is part of the loop, so the system should optimize for correction cost, legibility, and trust calibration over task success alongside nominal completion. The difference is measurable: in the YAY Robot study, adding mid-task verbal correction cut operator interventions by 50% across 90-minute sessions. A baseline that ignored corrections and optimized only for episode reward showed no improvement in operator effort at all.

This reframes demonstrations too. A correction is not a bare 'wrong' label; it reveals where the human expected the robot's internal state to differ. Strong systems keep that signal for later planning and personalization, a practice examined under human correction as data.

Code Fragment 2 stores a minimal interaction event with proposal, human response, and final execution decision. That event is the unit you need for studying trust, intervention rate, and correction efficiency.

1. Log the robot proposal before the user responds.
2. Record whether the user approved, corrected, or overrode the action.
3. Update the task state or policy threshold after the interaction, not only after episode end.
4. Track intervention frequency together with success rate and completion time.
5. Replay interaction traces to measure whether the same misunderstanding recurs across tasks.

When interaction fails, separate perception or planning errors from interface design errors. A system may have good low-level control yet still be unusable because correction is too slow, too opaque, or too expensive for the human. A robot that cannot be corrected mid-task is not autonomous: it is simply unsupervised.