Section 55.4: Logging, monitoring, model updates

"If you cannot replay the second before a failure, you did not observe it; you only survived it."

A Careful Control Loop

Problem First

Picture the 2 a.m. incident review from the opening: the robot stopped, a worker stepped back, and now five engineers stare at a dashboard that shows a single red latency spike and nothing else, with no way to tell whether perception, planning, control, timing, model drift, or operator context produced it.

In short: which artifact proves the claim afterward? For the full framing of that question, see section 55.2. Figure 55.4B traces the evidence contract that answers it: observation, state estimate, action, and monitor state all flow into a single result artifact on a shared time axis.

Theory

Before reading on, guess: if a robot arm causes unexpected contact at 2 a.m. and your team has no per-step log, how many replays will it take before you can identify whether a model weight change or a latency spike caused the incident? Most teams discover the answer is never: without the right fields in the log, the causal chain is permanently unrecoverable.

A robot whose failure cannot be reconstructed from its logs did not fail in a recoverable way; it failed twice.

Observability is the difference between a repeatable incident review and a post hoc story. Figure 55.4A shows the continuous stack this section builds: the robot writes timestamped observation-action pairs to a ring buffer, where a ring buffer is a fixed-size log that overwrites its oldest entries once full, so the most recent window of steps is always retained without unbounded storage growth, anomaly detectors flag unexpected reward or latency spikes, and a model-update pipeline retrains when the drift metric crosses a threshold. This property is called the closed-loop evidence chain, and it requires that every decision in the loop leaves a timestamped, version-tagged record. The deployment trace should align observations, estimated state, chosen action, model version, latency, monitor transitions, and operator interventions on a shared time axis.

The Observability Tuple

A useful observability tuple is

$$o_i = (t_i, y_i, \hat s_i, a_i, v_i, q_i, m_i, u_i),$$

where \(v_i\) is the version id, \(q_i\) is queue and latency telemetry, \(m_i\) is the monitor state, and \(u_i\) is any human intervention. Model updates should be promoted only if a canary or shadow evaluation, where shadow evaluation runs the candidate model on the live observation stream but discards its actions so its predictions are scored without ever driving the robot, shows gain without violating retained-skill or safety thresholds.

The "drift metric" referenced above (Figure 55.4A) needs a concrete definition to be actionable: one common choice is the population stability index (PSI) between the deployment-time distribution of a chosen observation feature (or, for high-dimensional inputs, a pretrained embedding of that feature) and the training-time distribution of the same feature, computed over a rolling window of logged \(y_i\) or \(\hat s_i\) values. A typical operating convention treats PSI below 0.1 as no meaningful shift, between 0.1 and 0.25 as moderate shift worth a shadow evaluation, and above 0.25 as a shift large enough to block promotion and trigger a retraining review; teams adjust these cutoffs to their own risk tolerance rather than treating them as universal constants. The threshold crossing is what the model-update pipeline in Figure 55.4A checks before it retrains: PSI (or an equivalent embedding-distance statistic) is computed each evaluation interval from the logged tuple stream, compared against the chosen cutoff, and only a crossing opens the shadow-then-canary path described next.

Consider a specific case. Waymo's robotaxi fleet logs over 20 telemetry streams per vehicle, including perception confidence, planner latency, and safety arbiter state. All streams share a microsecond clock. When a disengagement occurs, engineers replay the ROS-style bag against the exact model version and monitor configuration active at that moment. Industry analyses of autonomous vehicle disengagement data (as of 2024) trace many disengagements to a mismatch between the sensor input distribution at deployment time and the training distribution, not to model errors on in-distribution inputs. That is why \(v_i\) and \(m_i\) must appear in the same log row. The version tag lets engineers isolate the rollout cohort and diagnose the shift. A disengagement (an event where the autonomous system hands control back to a human safety driver, typically because the policy encountered a situation it was not confident handling) is exactly the kind of incident this logging discipline is meant to make replayable. The cost of missing those fields follows a consistent pattern across the deployments we have reviewed, though the exact numbers vary by fleet and tooling: in practice, teams with complete tuple logging have typically resolved similar regressions in a handful of replays within a few hours, while teams missing the version and monitor fields have reported spending weeks without reaching a causal conclusion.

Worked Example

To see why each tuple field is load-bearing rather than decorative, walk through a concrete update that the abstract schema above must be able to diagnose.

Suppose a grasp update improves carton picks but causes unexpected failures on reflective packaging. A deployment trace must answer whether the change came from model weights, input distribution drift, calibration drift, or monitor-threshold edits.

required_fields = {
    "timestamp", "obs_hash", "state_estimate", "action", "model_version",
    "queue_age_ms", "monitor_state", "operator_event"
}
logged_fields = {
    "timestamp", "obs_hash", "state_estimate", "action", "model_version",
    "monitor_state",
}

# Coverage: unique logged fields that match required fields, divided by total required fields.
coverage = len(required_fields & logged_fields) / len(required_fields)
update_gate = {
    "candidate_version": "grasp_v12",
    "shadow_panel_passed": True,
    "rollback_ready": True,
    "diagnostic_coverage": coverage,
}
print(update_gate)

{'candidate_version': 'grasp_v12', 'shadow_panel_passed': True, 'rollback_ready': True, 'diagnostic_coverage': 0.75}

The expected output is not just a green status. The key interpretation is that promotion is unjustified unless the update is both measurable and reversible. A candidate with improved task score but incomplete diagnostic coverage should still be rejected.

The algorithm below promotes a candidate to a "canary slice" as its next-to-last step; that term is defined here, before the algorithm, so the promotion step is unambiguous when you reach it. A canary slice is a small, designated subset of the physical fleet, typically one to three robots in a controlled zone, that runs the candidate model under full monitoring while the rest of the fleet remains on the production version. In embodied AI this boundary is not optional: a bad update that reaches every robot simultaneously cannot be recalled mid-motion, and physical damage to equipment or bystanders is irreversible within the failure window.

Practical Recipe

With the tuple defined and the canary gate in place, the remaining work is procedural: turn those ideas into a fixed order of operations that a team can follow on every rollout.

1. Write the observation, action, monitor, metric, and artifact fields before selecting a model.
2. Run a deterministic smoke test and one named perturbation from the panel.
3. Log success, safety events, latency, energy or resource use, and recovery status in the same row group.
4. Compare only methods evaluated by the same script on the same panel and seed plan.
5. Attach a short postmortem to each failed rollout so the artifact remains useful after the plot is forgotten.

Logging, monitoring, model updates becomes operational when the metric is tied to a runtime interface. The interface names the sensor stream, state estimate, action representation, timing budget, safety or robustness monitor, and deployment artifact.

As with other sections in this module, each claim must stay separate: conceptual, systems, and evidence. For the full framing, see section 55.2. For logging and monitoring specifically, the evidence claim centers on whether logs capture enough state to reconstruct failures and whether monitors fire before a silent degradation reaches safety-critical behavior.

Cross-References

Connect benchmark design, sim-to-real transfer, uncertainty, and safety barriers through the deployment artifact that will be checked before release.

When an update misbehaves, assign the failure to data drift, instrumentation gap, threshold drift, model regression, or release-process error. Then replay one canonical incident against both the previous and current version with identical telemetry collection.