Section 23.6: The LeRobotDataset format and pipeline

"The folder looked organized until the policy asked which camera was the wrist camera."

A Dataset Loader

Format Contract

Before a single gradient step runs, a robot dataset must survive one brutal interrogation. What is the observation? What is the action? Which episode does this frame belong to, and what metadata defines the body and the task? Get any one answer wrong and the policy learns confidently from a lie. Figure 23.6A sketches this shared contract: metadata, video, sensorimotor signals, and episode boundaries all agreeing on one explicit set of names and units. LeRobotDataset v3 organizes these answers around standardized feature names, Parquet tables (a columnar binary file format that stores each field contiguously on disk, so a loader can read just the action column without scanning every camera reference), videos or images, and metadata files. Getting those names right depends on the same concerns about data quality, diversity, and labeling that apply to the collection stage.

Code Fragment 1 validates a tiny feature schema before conversion. This catches the most common mistake: an action or timestamp field that exists in prose but not in the actual files.

# Validate a LeRobotDataset-style feature schema before conversion
# Demonstrates the minimum required fields and unit-annotation checks

import numpy as np

# Minimal required fields for a LeRobotDataset-compatible episode table
REQUIRED_FIELDS = {
    "timestamp",           # float, seconds since episode start
    "episode_index",       # int, episode identifier
    "frame_index",         # int, frame position within episode
    "observation.state",   # float array, joint angles in radians
    "action",              # float array, joint targets in radians
    "observation.images.front",  # str or bytes, camera frame reference
}

# Simulated feature schema from a conversion script (what the converter claims)
candidate_schema = {
    "timestamp":                  {"dtype": "float32", "unit": "seconds"},
    "episode_index":              {"dtype": "int64",   "unit": None},
    "frame_index":                {"dtype": "int64",   "unit": None},
    "observation.state":          {"dtype": "float32", "unit": "radians", "shape": (14,)},
    "action":                     {"dtype": "float32", "unit": "radians", "shape": (14,)},
    "observation.images.front":   {"dtype": "video",   "unit": None},
}

missing = REQUIRED_FIELDS - set(candidate_schema.keys())
schema_ok = len(missing) == 0

print(f"schema ok: {schema_ok}")
print(f"missing fields: {sorted(missing)}")

# Verify action unit is declared (not left ambiguous)
action_meta = candidate_schema.get("action", {})
unit = action_meta.get("unit")
unit_declared = unit is not None and unit != ""
print(f"action unit declared: {unit_declared} ({unit})")

# Simulate a shape smoke-test on a random synthetic episode
rng = np.random.default_rng(0)
n_frames = 10
synthetic_state  = rng.uniform(-1.0, 1.0, (n_frames, 14)).astype(np.float32)
synthetic_action = rng.uniform(-1.0, 1.0, (n_frames, 14)).astype(np.float32)
mean_abs_action = float(np.mean(np.abs(synthetic_action)))
all_zero_action = mean_abs_action < 1e-6
print(f"mean |action|: {mean_abs_action:.4f}  (near-zero signals missing follower data: {all_zero_action})")

schema ok: True
missing fields: []
action unit declared: True (radians)
mean |action|: 0.5032  (near-zero signals missing follower data: False)

The expected output is intentionally boring: schema ok: True and an empty missing-field list. That boring result is valuable because it means every later training script can assume episode identity, temporal order, timestamp alignment, observation image, robot state, and action are present. If observation.state or timestamp is missing, do not patch the trainer; repair the conversion pipeline so every downstream policy receives the same scientific object.

Code Fragment 23.6.1 above validates a schema; it does not perform a conversion. The LeRobotDataset.create() API (from the lerobot package) is the actual conversion entry point: it takes the validated schema plus per-frame arrays for state, action, and images, and writes the Parquet and video files that satisfy the contract. The Pipeline Recipe below walks through that conversion step by step; the Lab exercise later in this section gives a runnable version of the full create-then-verify cycle.

Pipeline Recipe

With the schema contract pinned down, the remaining work is the disciplined sequence that turns a live recording into a dataset that honors it, step by step from the robot's clock to the published license.

The five steps below produce three distinct layers of data (raw, normalized, training); that three-layer separation is only named here in passing and is defined in full, with a diagram, in the "Conversion And Verification" section just after this recipe.

1. Collect raw logs with hardware timestamps from the robot's onboard clock, not the laptop OS clock. On a Franka Panda running at 1 kHz, a 10 ms timestamp jitter between the arm state and wrist camera frame can shift the action label by one full control cycle, causing the policy to learn to react to contact forces that have already dissipated.
2. Normalize feature names and units: joint angles must be declared as radians (the LeRobotDataset contract) even when vendor SDKs such as HEBI or Dynamax (robot-arm actuator manufacturers whose motor-control APIs report joint position in degrees by default) return degrees, and gripper width must be declared in meters even when the raw field is a 0-to-1 normalized float. Store the per-field unit in the schema, not in a README that may diverge from the data.
3. Convert frames and states into a standardized directory: one Parquet file per episode for sensorimotor rows, one MP4 per camera stream per episode for images, and a single info.json at the root with fps, robot type, and feature shapes. Embedding video frames directly in Parquet typically inflates file size by roughly 30x (the exact factor depends on video codec and resolution) and breaks random-access replay on episode sets above roughly 50 GB.
4. Run a loader smoke test that renders camera frames alongside aligned robot state and action for episode 0, a middle episode, and the final episode. A near-zero mean absolute action across all joints signals that the follower arm stream was never recorded and the leader stream was written to the action field by mistake, a bug that costs roughly 40 GPU-hours to discover at training time.
5. Publish metadata, dataset card, split manifest, and license before reporting results. Policies trained on ALOHA (A Low-cost Open-source HArdware system for bimanual teleoperation) data collected at 50 Hz are not comparable to policies trained on data collected at 30 Hz unless the fps field and the action-chunk horizon (the number of future action steps the policy predicts per inference call, introduced in section 22.3) are both reported; without these fields, cross-lab comparisons conflate architecture differences with data-rate differences.

A 10 ms timestamp jitter and a 40 GPU-hour discovery cost sound abstract until you have felt them; both numbers are worked through concretely in the Step-Through and Practical Example below, so if the reasoning in this recipe feels rushed, that walkthrough is the place to see it applied frame by frame.

Conversion And Verification

Before reading on, consider: if a second lab downloads your dataset six months from now and the only documentation is a README you wrote the night before submission, which field will silently corrupt their training run first?

A robust conversion pipeline keeps three layers separate. The raw layer preserves the original robot logs, including vendor-specific messages and leader-device signals. The normalized layer exposes canonical features such as observation.images.front, observation.state, and action. The training layer may add cached tensors, resized videos, or model-specific transforms, but those derived artifacts should be reproducible from the normalized layer. This separation is what enables reuse without re-collection: a second lab can rebuild only the training layer from your normalized layer, adapting to a new resolution or action representation without touching the raw logs. A dataset without a declared schema is not a scientific artifact; it is a time bomb left for the next researcher to defuse. Figure 23.6B shows how the three layers connect.

Why Parquet, Not CSV

That layered contract only pays off if the normalized layer is also cheap to read at training time, and the reason it is comes down to how the rows are physically stored. Training loops read robot datasets column-by-column, not row-by-row. A loop fetches all action values to compute statistics, or all episode_index values to build a split manifest, without touching camera references. Parquet stores each column contiguously on disk and compresses it independently, so those column scans skip unrelated data entirely. On a cold SSD, a 100,000-row episode table with 14-dimensional state and action fields typically loads in under a second as Parquet, versus several seconds for the equivalent CSV (the exact gap depends on disk speed and compression settings). Float columns typically compress 3-to-5x in Parquet, and the read head never seeks across interleaved row data. That gap compounds at dataset scale: a project that would fit in roughly 8 GB as Parquet can balloon to roughly 30 GB as CSV, often the difference between a dataset that fits on a single NVMe drive and one that requires a remote storage mount, adding a network round-trip to every training step.

In embodied AI this is not just a storage optimization. A policy that reads the wrong column because field order shifted between a CSV export and re-import will execute physically wrong joint targets on the robot. Parquet's schema is stored in the file footer and is checked on every open, so a unit mismatch or missing field raises an error at load time rather than producing a silently wrong trajectory on hardware.

The most important verification step is random-access replay. Sample an episode from the beginning, middle, and end; render the camera frame; print the aligned robot state; and overlay the action that follows. This catches off-by-one frame shifts, stale calibration, swapped wrist cameras, and action-unit mistakes that a shape-only validator misses. The payoff at reuse time depends on keeping the layers apart: a second lab retraining at a different image resolution or a different action representation rebuilds only the training layer from your normalized layer, untouched raw logs and all. Collapse normalization and training transforms into one script and that reuse vanishes; the second lab has to re-collect.

Consider a specific case: a lab records 200 ALOHA bimanual episodes at 50 Hz, each roughly 10 seconds long, yielding 500 frames per episode and 100,000 total rows. The normalized Parquet table stores one row per frame with columns timestamp, observation.state (14 joint angles in radians, one per arm), action (14 joint targets), and episode_index. Two camera streams (front and wrist) are stored as separate MP4 files named by episode index, not embedded in Parquet. The smoke test samples episode 0, episode 99, and episode 199; renders frame 0 and frame 250 of each; and prints the mean absolute action value per episode. If any episode's mean action is near zero for all joints, the follower signal was never recorded and the raw leader stream was mistakenly written to the action field instead. In practice, catching this bug after training costs roughly 40 GPU-hours and 3 days of retraining; catching it in the smoke test costs under 30 seconds.