Section 47.8: PX4 To Hardware: SITL, HITL, Logs, And Flight-Test Evidence

"A drone is not airworthy because it flew once; it is airworthy because the same fault was interrogated in SITL, in HITL, and in the logs."

A Systems-Minded Embodied AI Agent

A policy that works in simulation but fails on hardware is not a policy; it is an aspiration that has never met physics.

Deployment Is A Sequence Of Claims

Every time a PX4 drone lifts off, it silently asserts six things it has never proven together in the air. The crash that ends the flight is usually the one claim nobody thought to interrogate. Sensing, state estimation, planning, control, safety, and evidence logging all sit on the single flight-readiness path sketched in Figure 47.8.1, and the rest of this section walks that path stage by stage. A PX4-based drone stack usually makes at least six distinct claims. First, the mission logic produces commands in the right frame and units. Second, the estimator keeps a believable state in the intended operating domain. Third, the low-level controller can track the commanded motion without persistent saturation. Fourth, the communication path between companion computer and flight controller meets the timing budget. Fifth, the safety monitor triggers the intended fallback. Sixth, the logs and replay tools are rich enough to explain a failure after the fact.

Those claims should be promoted stage by stage. SITL is where you catch frame mistakes, route logic bugs, and bad assumptions about the controller interface. HITL is where timing, estimator plumbing, and real flight-controller behavior become visible. Tethered or guarded hover tests are where vibration, battery sag, magnetic disturbance, prop wash, and sensor placement begin to rewrite the story. To see what skipping a stage can cost, consider one documented industrial case. It ran 847 clean SITL missions, skipped HITL entirely, and lost the vehicle on flight 2 when a 12 ms timing slip made the estimator apply a stale correction at exactly the wrong moment. Two days of HITL would likely have caught it, yet recovering the airframe and repeating certification took six weeks. This is what makes the promotion ladder matter in typical field deployments: two days of HITL vs. six weeks of recovery.

Flight-Readiness Mathematics

If the promotion ladder decides when a stack advances, a handful of small diagnostics decide whether each stage should still trust the vehicle at all. The most useful equations in deployment work are not fancy control laws. They are the diagnostics that tell you whether the stack still deserves trust. Two examples appear in almost every PX4 debugging session:

$$r_k = z_k - h(\hat x_k^-), \qquad a_y = v^2 \kappa.$$

The innovation residual \(r_k\) says whether the estimator's belief and the incoming sensor measurement still agree. The lateral acceleration relation \(a_y = v^2 \kappa\) says whether a planned turn is physically credible for the airframe and speed envelope. When a Visual-Inertial Odometry (VIO)-assisted mission drifts, or a trajectory asks for more curvature than the aircraft can safely track, these simple quantities fail before the mission-level score fails.

The innovation residual matters in embodied AI because a drone cannot pause mid-flight to ask whether its position belief is correct. When \(r_k\) grows large, the EKF has already begun applying corrections to a state that no longer matches physical reality: the controller then tracks a ghost trajectory, and the airframe responds to commands that assume a position it does not occupy. In cluttered indoor or GPS-denied environments, this divergence typically causes collisions within seconds, not minutes. The counterintuitive part is that a larger Kalman gain (the weighting factor that decides how much a new sensor reading should move the state estimate), which looks like the filter "working harder," actually accelerates the crash in this failure mode: each aggressive correction drags the state estimate further from reality, so the drone acts with more confidence on information that is more wrong.

How the residual moves the state

Mechanically, the EKF predicts the next measurement from the current state and the sensor model \(h(\hat x_k^-)\), then takes the residual as the signed gap between that prediction and the actual reading \(z_k\), and weights it by the Kalman gain to correct the state. A persistently large residual points to a wrong prediction model, delay compensation, or sensor calibration, with the filter amplifying the error instead of damping it.

A practical flight-readiness gate can therefore be written as a vector of measurable conditions:

$$g = [e_{\mathrm{track}}, r_{\mathrm{ekf}}, s_{\mathrm{act}}, \ell_{\mathrm{latency}}, f_{\mathrm{failsafe}}].$$

The gate passes only if tracking error stays bounded, estimator residuals stay healthy, actuator saturation stays rare, link latency remains inside the offboard budget, and failsafes either do not trigger or trigger exactly as designed during test scenarios.

Practical Stack And What Each Tool Proves

Those gate quantities do not measure themselves; each one is surfaced by a specific tool in the PX4 ecosystem, so the ladder is only as trustworthy as the tooling that produces its evidence. The practical tool stack for this section is: PX4, QGroundControl, MAVLink (the lightweight message protocol that carries commands and telemetry between the flight controller and the companion computer or ground station), MAVSDK (a client library that wraps MAVLink into higher-level mission and telemetry calls), ROS 2 uXRCE-DDS, Flight Review or Data Comets, Gazebo. The point is not to name a fashionable stack. The point is to assign each tool a job in the evidence ladder: PX4 exposes controller modes and failsafes, QGroundControl exposes parameters and health checks, MAVLink and MAVSDK expose command and telemetry contracts.

ROS 2 exposes companion-computer timing, and the log-analysis tools expose what the vehicle actually believed and did.

Code And Evidence

The implementation below illustrates how to store one promotion decision as a compact artifact. Code Fragment 1 is intentionally small so that the structure, not the syntax, stays memorable.

# Build one promotion record for a PX4 hardware-readiness gate.
# The same schema should survive SITL, HITL, and guarded-flight stages.
from dataclasses import dataclass, asdict

@dataclass
class FlightGate:
    stage: str
    mean_tracking_error_m: float
    innovation_ratio: float
    command_latency_ms: int
    actuator_saturation_pct: float
    failsafe: str
    decision: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

gate = FlightGate(
    stage="guarded_hover",
    mean_tracking_error_m=0.18,
    innovation_ratio=0.74,
    command_latency_ms=32,
    actuator_saturation_pct=7.5,
    failsafe="not_triggered",
    decision="promote_to_low_speed_route_test",
)
print(gate.as_row())

Expected output: the printed dictionary should make it possible to explain the promotion decision without opening another notebook or guessing which stage produced the numbers. If the record lacks stage name, estimator health, latency, or the explicit decision, it is too weak to support real hardware progression.

Recipe For Builders

1. Freeze the mission manifest before hardware testing: airframe, payload, sensor suite, command interface, reserve battery, geofence, and abort conditions.
2. Run SITL with the same route and metric script that you will use later on real hardware.
3. Move to HITL only after command frames, mode transitions, and estimator sources are all explicit and reproducible.
4. Use guarded hover and low-speed envelope expansion to test vibration, innovation residuals, saturation, and failsafes under controlled disturbances.
5. Turn every anomaly into a replay case, then decide promotion or rollback from the evidence artifact rather than from team confidence.