Section 50.3: Intent recognition and trust calibration

Overtrust is what happens when a progress bar wears a lab coat.

An Uncertain Operator

Picture a robot arm already swinging toward your open hand because, 400 milliseconds ago, it decided you wanted the scalpel rather than the gauze: by the time you realize it guessed wrong, the metal is already moving, and the only thing standing between a smooth handover and a puncture wound is whether its belief about your intent was honest about its own uncertainty. That is why intent recognition is worthless as a number on a screen and only becomes a decision the robot can be held to when it is tied to a concrete action interface: a Franka Panda end-effector pose published at 500 Hz over the FCI, a RealSense D435 gaze stream at 30 Hz, a 50 ms synchronization budget, and a logged belief trace you can replay frame by frame when the arm reaches for the wrong instrument. Here the FCI is the Franka Control Interface, the low-latency link that streams robot state and accepts torque or pose commands at 1 kHz.

The key question is practical: What evidence changes the robot belief about the human, and what evidence changes the human belief about the robot? Figure 50.3A shows both loops running together over a single episode: a Bayesian intent estimator updating as the human moves, the robot adjusting its trajectory proactively, and a displayed confidence score that keeps the human's trust aligned with what the robot actually knows.

Theory

The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

Worked Example

With the filter-not-classifier contract in hand, the cleanest way to see why incremental updating matters is to watch a single distribution shift tick by tick as evidence arrives.

Consider an assistive arm watching a person reach toward a cup. The system may predict handover, cleanup, or avoidance; the right behavior depends on uncertainty and on how confidently the robot presents its guess.

Consider a specific case: at time \(t=0\) the robot holds a uniform prior over three intents (handover, cleanup, avoidance), each at 0.33. After 0.4 s of wrist motion toward the cup, a Bayesian update using pose likelihood ratios shifts the distribution to handover 0.71, cleanup 0.21, avoidance 0.08. The robot now moves its end-effector 8 cm toward the predicted grasp point while displaying "handover: 71%" on its wrist screen. At \(t=0.9\) s the person's gaze breaks toward the sink, shifting handover to 0.31 and cleanup to 0.62; the arm pauses and requests confirmation with a single LED blink rather than committing to a now-wrong trajectory. Trust is updated upward when the person confirms the arm's self-correction, a pattern called earned trust through transparent uncertainty: a post-trial survey in the CARESSES assistive-robot study (a multi-year EU-funded home-care robot trial with older adults) (Papadopoulos et al., 2020) found that visible uncertainty display of this kind raised perceived competence ratings by 0.6 points on a 7-point Likert scale compared to arms that acted without expressing doubt.

# Bayesian intent estimator with trust calibration error over a simulated episode
import numpy as np

# Three possible human intents
INTENTS = ["handover", "cleanup", "avoidance"]

def bayesian_update(belief, likelihoods):
    """Multiply belief by likelihoods and renormalize."""
    posterior = belief * likelihoods
    return posterior / posterior.sum()

def trust_calibration_error(robot_confidence, actual_success_rate):
    """Absolute gap between what the robot claims and what it delivers."""
    return abs(robot_confidence - actual_success_rate)

# Simulated episode: each row is (pose_likelihood, gaze_likelihood) per intent
observations = [
    np.array([2.0, 1.0, 0.5]),   # t=0.4s: wrist moves toward cup
    np.array([1.5, 1.2, 0.3]),   # t=0.7s: hand opens slightly
    np.array([0.4, 3.0, 0.6]),   # t=0.9s: gaze shifts to sink
]

belief = np.ones(len(INTENTS)) / len(INTENTS)   # uniform prior

print(f"{'Time':>6}  {'handover':>10} {'cleanup':>10} {'avoidance':>10}  Best guess")
print("-" * 62)
times = [0.0, 0.4, 0.7, 0.9]
print(f"{times[0]:>6.1f}  {belief[0]:>10.3f} {belief[1]:>10.3f} {belief[2]:>10.3f}  {INTENTS[belief.argmax()]}")

for t, obs in zip(times[1:], observations):
    belief = bayesian_update(belief, obs)
    best = INTENTS[belief.argmax()]
    print(f"{t:>6.1f}  {belief[0]:>10.3f} {belief[1]:>10.3f} {belief[2]:>10.3f}  {best}")

robot_confidence = float(belief.max())
actual_success_rate = 0.55   # hypothetical ground-truth from trials
err = trust_calibration_error(robot_confidence, actual_success_rate)
print(f"\nRobot confidence in '{INTENTS[belief.argmax()]}': {robot_confidence:.3f}")
print(f"Actual success rate (from trials): {actual_success_rate:.2f}")
print(f"Trust calibration error: {err:.3f}  {'[SEVERE]' if err > 0.5 else '[acceptable]'}")

  Time    handover    cleanup  avoidance  Best guess
--------------------------------------------------------------
   0.0       0.333       0.333       0.333  handover
   0.4       0.571       0.286       0.143  handover
   0.7       0.686       0.275       0.034  handover
   0.9       0.232       0.699       0.017  cleanup

Robot confidence in 'cleanup': 0.699
Actual success rate (from trials): 0.55
Trust calibration error: 0.149  [acceptable]

The trust_calibration_error function above is the working definition used throughout this section; the same quantity is restated as a formal equation later, in the Technical Core's Formal Object box, once the Bayesian belief notation has been introduced.

Practical Recipe

The Big Picture promised a Bayesian intent estimator, a trust calibration error metric, and design patterns that keep human belief aligned with robot reality. The Worked Example and the Technical Core above delivered the first two: the belief update in Code Fragment 50.3.1 and the trust-error formula in the Formal Object box. The design patterns are what follows now: concrete rules for building and testing a shared-workspace system so the estimator and the calibration metric actually get used, rather than sitting in a notebook.

Turning that single traced episode into a system you would trust around a real person means committing to an order of operations, starting with the physical interface and ending with a worst-case perturbation test.

1. Define the physical observation schema before choosing a model: for a Franka Panda shared-workspace task, this means locking down which joints feed into pose likelihood (end-effector pose at 500 Hz from the FCI), which camera provides gaze direction (an Intel RealSense D435 at 30 Hz), and what the synchronization tolerance is (no more than 50 ms via ApproximateTimeSynchronizer) before writing a single belief-update line.
2. Build a three-intent baseline with a uniform prior and hand-tuned likelihood tables before fitting any learned model. On a Spot robot running a fetch task, a table with pose and proximity features debugged in MuJoCo is readable by inspection and will catch frame-convention errors (robot-frame versus world-frame wrist poses) that a neural model silently absorbs into its weights.
3. Add a learned intent model, such as a Transformer over tokenized joint trajectories trained on Open X-Embodiment pick-and-place demonstrations (Open X-Embodiment is a large public cross-robot dataset of recorded manipulation episodes, useful here as a source of realistic trajectories rather than a source of intent labels), only after the tabular baseline passes a minimum of 10 scripted test scenarios covering clear, ambiguous, and mid-trajectory intent switches.
4. Record every failure with a four-field label: the triggering sensor modality (pose dropout, gaze occlusion, dialogue latency above 200 ms), the belief state at failure, the action the robot committed to, and whether the human intervened. This structured log lets you distinguish a perception failure from a planning failure without re-running the episode.
5. Before any human-facing trial, run one perturbation test in simulation where the human intent switches at the worst possible moment (when robot confidence peaks above 0.8) and confirm the arm pauses or requests confirmation rather than committing to the now-wrong trajectory.

Intent recognition and trust calibration becomes useful when it is tied to a closed-loop contract for Human-Robot Interaction. The contract names the participants, observations, action authority, timing budget, logging artifact, and recovery rule. Without that contract, a system can look capable in a notebook while failing the first time a partner delays, a person corrects it, or a deployment scene changes.

Separate the conceptual claim, the systems claim, and the evidence claim. A plausible mechanism, a clean interface, and a closed-loop result are different claims; the section should keep their evidence separate.

Which of these tools do you reach for first when the robot misreads the human's intent at exactly the moment the arm is already in motion?

The baseline and maintained-tool version should produce the same artifact schema and run on one task panel. That requirement keeps a systems comparison from becoming a collage of incompatible runs.

1. Write a one-paragraph task contract with observation, action, success, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel.

When Intent recognition and trust calibration fails, avoid labeling the whole method as weak. First assign the failure to perception, communication, human input, memory, planning, control, timing, data coverage, safety, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.