Section 17.6: Throughput, wall-clock, and cost engineering

"A million frames per second means nothing if they all arrive after the budget is gone."

A GPU Cluster Invoice

Two engineers stand at a whiteboard: one quotes 2,000,000 environment steps per second, the other quotes 400,000, and the slower system is the one that shipped a deployable policy under budget. That inversion is the whole problem of this section, and it only makes sense once GPU RL is treated as a cost-accounting exercise where simulator fidelity, PPO rollout semantics, reward terms, and reset distribution (the rule that decides where and how each parallel environment respawns after an episode ends) are versioned in the same training artifact.

This section develops the cost contract for massively parallel RL, where thousands of simulated environments run in lockstep on a single GPU to generate rollouts (the trajectories of states, actions, and rewards a policy produces while acting in the environment). We define the denominator for every claim: environment steps per second, policy updates per minute, wall-clock to target success, GPU memory at peak, evaluation time, and dollars per successful checkpoint.

The key question is practical: did the optimization make learning cheaper, or did it only move time from rollout collection into compilation, synchronization, logging, or evaluation? A run reporting 2,000,000 steps per second can still cost more than a run reporting 400,000 steps per second: if the faster system spends 40 percent of wall-clock on evaluation and another 15 percent on logging, the slower system reaches the 90 percent held-out threshold first (held-out meaning measured on evaluation seeds withheld from training, not merely re-run on the same conditions used to update the policy) and at lower dollar cost. A throughput number without a success threshold is a speedometer reading on a road with no destination. Figure 17.6A frames the tension this section resolves: raw speed on its own says nothing about whether a deployable policy was reached within budget. Figure 17.6B makes the accounting concrete, tracing how each wall-clock phase feeds the single cost ledger introduced below.

Theory

Let a run collect \(S\) environment steps in \(W\) wall-clock seconds. The raw throughput is \(S/W\), but the cost metric should include training, evaluation, checkpointing, and failed runs. If the target is a 90 percent held-out success rate, the relevant question is how many dollars and minutes were spent before the first checkpoint reached that target.

In contact-rich manipulation and legged locomotion, the throughput-versus-sample-efficiency tradeoff has a physical edge that pure RL benchmarks hide. Isaac Lab locomotion runs with 4,096 parallel Anymal environments and a PPO rollout horizon of 24 steps typically saturate (in practice, on this class of hardware; exact numbers vary with driver version and environment complexity) an A100 at around 200,000 environment steps per second (as of 2024, on Isaac Lab with RSL-RL). Doubling the batch to 8,192 raises device utilization but stretches policy lag (the real-time delay between when the environment state was sampled and when the resulting action, computed from that stale state, actually reaches the robot) to roughly 48 real-time milliseconds per update cycle. For a quadruped recovering from a push disturbance, that lag can decide between a recoverable stumble and a fall, because the contact phase that needs the corrective torque command typically lasts only 60 to 80 ms for this class of push-recovery task. Smaller batches keep the policy fresher but leave the GPU at 40 to 50 percent utilization, which wastes the hardware budget. The RSL-RL locomotion stack fixes this by tuning rollout horizon rather than batch size: fix the environment count at what fills GPU memory, then shorten the horizon until the update rate matches the task's fastest transient dynamics. The horizon matters more than raw step count. Halving it from 24 to 12 steps on 4,096 environments cuts the policy lag from 48 ms to 24 ms while collecting the same 98,304 transitions per update, because the robot receives a fresher policy twice as often within the same wall-clock window.

Worked Example

Code Fragment 17.6.1 computes the metrics that should appear together in a cost report. The same artifact contains throughput, wall-clock to target, evaluation overhead, and compute spend.

# Compute throughput and cost from one RL training run.
# The target-success checkpoint, not peak steps per second, drives the decision.
env_steps = 1_200_000_000
train_minutes = 42.0
eval_minutes = 6.0
gpu_dollars_per_hour = 2.20
heldout_success = 0.92
target_success = 0.90

total_minutes = train_minutes + eval_minutes
steps_per_second = env_steps / (train_minutes * 60)
total_cost = (total_minutes / 60) * gpu_dollars_per_hour
cost_per_billion_steps = total_cost / (env_steps / 1_000_000_000)

print(f"train throughput: {steps_per_second:,.0f} env steps/s")
print(f"total wall-clock: {total_minutes:.1f} min")
print(f"held-out success: {heldout_success:.2f}")
print(f"target reached: {heldout_success >= target_success}")
print(f"total compute cost: ${total_cost:.2f}")
print(f"cost per billion steps: ${cost_per_billion_steps:.2f}")

Expected output: the trace should include both systems and learning metrics. A run that reports 476,190 steps per second but omits held-out success has not shown that those steps bought a deployable policy.

Practical Recipe

1. Define the success target before training, such as 90 percent held-out success across 256 evaluation seeds.
2. Measure rollout, learning, evaluation, logging, and checkpoint time separately.
3. Track peak GPU memory, utilization, host-device transfer time, and compilation time where relevant.
4. Report dollars per target-reaching checkpoint, not only dollars per hour.
5. Repeat the run across seed panels when the result will support a paper table or hardware decision.

The Same-Denominator Rule

Every number needs a denominator, a principle called the same-denominator rule. Steps per second uses training seconds. Wall-clock to target uses training plus evaluation seconds. Cost uses all billable time. Paper tables and engineering decisions should not mix these denominators. Concretely: reporting Run A's steps per second (measured over rollout seconds only) next to Run B's dollars per success (measured over rollout, learner, evaluation, sync, and log seconds combined) in the same table row implies a direct comparison that the two numbers cannot support, because they were integrated over different windows of the same run.

In embodied AI this matters because teams decide robot hardware purchases and deployment budgets by comparing training runs. A locomotion controller that appears twice as cheap may reflect only training seconds while its competitor counted evaluation too. It may also exclude three failed runs that consumed half the real budget. A team that buys cheaper GPUs from a mismatched comparison discovers the error only after the hardware arrives.

The rule requires a single accounting window: one script, one run, one artifact. Every phase timer starts and stops inside that window, so all fractions sum to total billable time. Two runs compare only when both ledgers share the held-out seed panel and the success definition, which surfaces denominator drift at merge time instead of at paper review.

Report the bottleneck breakdown. A low throughput run may be simulation-bound, learner-bound, memory-bound, evaluation-bound, logging-bound, or compilation-bound. Each bottleneck implies a different fix.

Each tool in that table contributes a column to one shared record, and the structure that binds those columns together is the cost ledger itself. A robust implementation starts with a cost ledger. The ledger makes it impossible to claim a throughput win from one run and a success win from another run.

1. Start the ledger before launch with instance type, price, batch size, target metric, and evaluation cadence.
2. Update the ledger at each checkpoint with train seconds, eval seconds, peak memory, success, and reset reasons.
3. Mark failed runs explicitly so cost estimates include search and debugging, not only the winning run.
4. Save profiler summaries with the same run ID as the policy checkpoint.
5. Compare cost only when success is computed on the same held-out panel.

# Record one cost ledger row for a GPU RL checkpoint.
# Keep systems metrics and held-out behavior in the same artifact.
from dataclasses import dataclass, asdict

@dataclass
class CostLedgerRow:
    checkpoint: int
    train_minutes: float
    eval_minutes: float
    peak_gpu_gb: float
    heldout_success: float
    billable_cost_usd: float

    def as_row(self) -> dict[str, object]:
        return asdict(self)

row = CostLedgerRow(
    checkpoint=320,
    train_minutes=42.0,
    eval_minutes=6.0,
    peak_gpu_gb=18.4,
    heldout_success=0.92,
    billable_cost_usd=1.76,
)
print(row.as_row())

Once such a row is recorded, its real value shows up the moment the cost it reports comes back higher than expected. When a cost result disappoints, avoid changing the algorithm first. Identify whether the budget was lost to simulator stepping, learner updates, GPU memory pressure, host-device transfers, compilation, evaluation, logging, or failed hyperparameter searches. Each cause has a different repair.