Section 53.2: Model uncertainty and calibration

A robust robot is not the one that never sees surprise, it is the one that notices surprise early enough to act differently.

A Runtime Monitoring Engineer

Figure 53.2.1 sets the visual theme for this section: a confidence number is trustworthy only when it matches how often the prediction is actually right.

Why This Matters

A self-driving car's perception stack reports 98% confidence that the road ahead is clear. The bus full of passengers behind it trusts that number. The one time in fifty that the number is wrong is the only time that matters. Calibration is the discipline of making a 98% claim right 98 times out of 100, so that a brake can be wired to a confidence score. Model uncertainty and calibration is useful only when it distinguishes disturbance sources and ties them to specific corrective actions. Robustness is not one scalar, it is a map from perturbation class to degraded behavior, detection delay, and residual risk.

To see why calibration matters in practice: imagine a robot arm that assigns 0.92 confidence to every candidate grasp, but only 68% of those grasps actually succeed. A runtime gate set at a 0.85 threshold will never trigger, because the raw confidence is always above it regardless of true difficulty. The robot proceeds confidently into failure after failure. The issue is not that the model is wrong on average; it is that the confidence scale has decoupled from the empirical success rate, so no threshold can separate good grasps from bad ones. Calibration is the correction that re-couples the scale to reality, and this decoupling is called the confidence-accuracy gap.

A confidence score that cannot be trusted as a number is not a safety signal; it is a decoration on a threshold that will never mean what it says.

In embodied AI this gap carries physical consequences. A robot arm acting on miscalibrated scores cannot self-correct through repeated attempts: each failed grasp risks damaging fragile parts, displacing the target object, or exhausting a narrow time window. On legged platforms, a miscalibrated terrain-safety score can commit the robot to a foothold from which it cannot recover. The physical world does not allow confidence values to correct themselves through backpropagation.

The gap arises because training pushes neural networks to maximize class separation, not probability accuracy. A sigmoid or softmax output that ranks options reliably can still saturate near 0 or 1, because cross-entropy loss rewards correct orderings, not correct magnitudes. Larger, deeper networks amplify this effect (Guo et al., 2017). Temperature scaling divides the logits (the raw pre-softmax scores a network emits for each class, where the softmax is the function that turns those scores into a probability distribution summing to one) by a learned scalar before the softmax. This compresses overconfident outputs back toward the empirical success rate without changing the ranking. A second post-hoc method, isotonic regression (a technique that fits the best-fitting non-decreasing curve through a set of points, so confidence values can only be remapped, never reordered), fits a free-form monotonic mapping from raw confidence to calibrated probability. It can therefore correct arbitrary distortions in the confidence scale rather than just a single global temperature, at the cost of needing more held-out data to fit reliably. Figure 53.2.2 contrasts a raw overconfident model against its calibrated counterpart and traces the pipeline that either map slots into.

A standard calibration statistic is the expected calibration error (ECE) $$\mathrm{ECE} = \sum_{b=1}^{B} \frac{|S_b|}{n}\, |\mathrm{acc}(S_b) - \mathrm{conf}(S_b)|,$$ where \(S_b\) is the set of predictions whose confidence falls in bin \(b\). In embodied systems, calibration should typically be evaluated on action-relevant predictions, not only class labels, since a class label with no control effect can stay miscalibrated without changing robot behavior.

Before applying either fix, note what each depends on: temperature scaling needs only a single scalar fit on held-out data, while isotonic regression needs a larger held-out set to fit a free-form monotonic curve reliably. ECE is the diagnostic that tells you whether either correction is needed at all, so in practice teams compute ECE first, then choose a calibration map, then re-measure ECE on a separate shifted panel to confirm the fix held.

Worked Example

A manipulation policy may be good at ranking candidate grasps yet still overclaim 0.95 confidence on cases that succeed only 0.70 of the time. A runtime gate built on that confidence will intervene too late.

predictions = [
    {"confidence": 0.9, "correct": 1},
    {"confidence": 0.8, "correct": 1},
    {"confidence": 0.8, "correct": 0},
    {"confidence": 0.6, "correct": 1},
    {"confidence": 0.55, "correct": 0},
    {"confidence": 0.95, "correct": 1},
]

def binned_ece(preds, n_bins=10):
    """Proper binned ECE: sum over bins of (n_b / N) * |accuracy_b - confidence_b|."""
    N = len(preds)
    edges = [i / n_bins for i in range(n_bins + 1)]
    ece = 0.0
    for i in range(n_bins):
        lo, hi = edges[i], edges[i + 1]
        in_bin = [p for p in preds if lo < p["confidence"] <= hi or (i == 0 and p["confidence"] == 0.0)]
        if not in_bin:
            continue
        n_b = len(in_bin)
        acc_b = sum(p["correct"] for p in in_bin) / n_b
        conf_b = sum(p["confidence"] for p in in_bin) / n_b
        ece += (n_b / N) * abs(acc_b - conf_b)
    return ece

ece = binned_ece(predictions)
print({"n_predictions": len(predictions), "binned_ece": round(ece, 4)})

{'n_predictions': 6, 'binned_ece': 0.15}

Computing ECE tells you the scale is trustworthy on average, but it does not by itself pick a threshold. Once the confidence map is calibrated, choose the runtime gate threshold by walking the reliability bins and selecting the lowest calibrated confidence value whose bin accuracy still meets the task's required success rate; for example, if the safety budget for a grasp attempt requires at least 90% empirical success, scan the calibrated bins from high to low confidence and take the first bin boundary where accuracy drops below 0.90 as the gate threshold. Because the confidence is now calibrated, that threshold value means what it says: candidates below it fail at least as often as the budget allows, so the gate is not an arbitrary cutoff but a direct readout of an acceptable-risk boundary.

Expected output: The small gap here suggests the average scale is close, but a real evaluation would still inspect bins because cancellation can hide local miscalibration. Calibration is about the full reliability shape, not only the mean.

These tool anchors are not hypothetical conveniences; the same calibration gaps they measure have already surfaced in production fleets. Waymo's public safety reports (2020 onward) suggest raw perception confidence is typically overconfident on rare classes such as cyclists in low light, prompting engineers to recalibrate against field outcomes before gating disengagement. RT-2 (Brohan et al., 2023) reported a similar pattern in tabletop manipulation: it predicted 0.9+ success where the new environment delivered closer to 0.55, a gap large enough to invalidate a threshold-based safety monitor unless teams fit isotonic regression on a held-out set from the target domain. Both cases share one structure: the model ranks options correctly, but its absolute confidence scale is wrong, so any fixed threshold either fires constantly or never fires.

In embodied systems, calibration must be tied to consequences. A poorly calibrated collision predictor and a poorly calibrated object classifier are not equally serious if only one gates a safety-critical maneuver. In practice, teams often compare temperature scaling, isotonic regression, and conformal intervals on the exact signal that drives a stop, reroute, or human-review threshold.

One signal, end to end

The implementation contract is simple: the confidence value in the PyTorch or JAX tensor must be the same signal logged in the replay artifact, plotted in Weights & Biases or TensorBoard, and consumed by the ROS 2 gate. A calibration curve that is detached from the deployed threshold is analysis theater, not safety evidence.

Keeping that single signal consistent across the pipeline is necessary but not sufficient, because where you fit the calibration map matters as much as which signal you fit it on. A frequent failure is to calibrate on clean validation data and deploy on shifted scenes. The confidence scale then looks disciplined in the notebook and collapses in the field.

Cross-References

This section connects to Section 53.3 on out-of-distribution (OOD) detection and Section 54.4 on shielded policies, where calibrated thresholds become actual intervention logic.