LeRobot data

Overview

This workshop opens a real dataset with the real lerobot package, version 0.6.1: .

The mental model: a LeRobot dataset is the 30 Hz record loop from Robot basics, saved: one row per frame, plus the camera videos.

Files

def open_dataset(**kwargs):
    # pyav decodes the videos; torchcodec, LeRobot's default, needs a PyTorch older than the one here.
    return LeRobotDataset(REPO, video_backend="pyav", **kwargs)

"""A tour of a real LeRobot dataset: 50 SO-101 episodes of putting a pink LEGO brick into a box.

Run (the lerobot environment, see README.md):
    ~/.virtualenvs/lerobot/bin/python classes/ml/labs/lerobot/data_tour.py

The first run downloads 86 MB from the Hugging Face Hub into ~/.cache/huggingface/lerobot.
"""

import numpy as np
import torch
from lerobot.datasets.lerobot_dataset import LeRobotDataset

REPO = "lerobot/svla_so101_pickplace"


def open_dataset(**kwargs):
    # pyav decodes the videos; torchcodec, LeRobot's default, needs a PyTorch older than the one here.
    return LeRobotDataset(REPO, video_backend="pyav", **kwargs)


def columns(dataset):
    """The whole recording as arrays: one row per frame, 30 frames a second."""
    table = dataset.hf_dataset
    return {key: np.array(table[key]) for key in ("episode_index", "frame_index", "timestamp", "observation.state", "action")}


def lag(columns, joint, max_shift=15):
    """Mean |state − action| when the state is read `shift` frames after the action, for shifts 0 … 14."""
    errors = []
    for shift in range(max_shift):
        per_episode = []
        for e in np.unique(columns["episode_index"]):
            rows = columns["episode_index"] == e
            state, action = columns["observation.state"][rows, joint], columns["action"][rows, joint]
            per_episode.append(np.abs(state[shift:] - action[:len(action) - shift]).mean())
        errors.append(float(np.mean(per_episode)))
    return errors


def normalise(value, mean, std, eps=1e-8):
    """What ACT's preprocessor does to every joint value and action before the network sees it."""
    return (value - mean) / (std + eps)


if __name__ == "__main__":
    dataset = open_dataset()
    print(f"{dataset.num_episodes} episodes, {len(dataset)} frames at {dataset.fps} fps; task: {dataset.meta.tasks.index[0]}")
    frame = dataset[0]
    for key, value in frame.items():
        print(f"  {key:26s} {tuple(value.shape) if hasattr(value, 'shape') else value}")

    data = columns(dataset)
    lengths = np.bincount(data["episode_index"])
    print(f"episode lengths: {lengths.min()} to {lengths.max()} frames, median {np.median(lengths)}")

    names = dataset.meta.features["action"]["names"]
    for j, name in enumerate(names):
        errors = lag(data, j)
        best = int(np.argmin(errors))
        print(f"  {name:18s} |state − action| {errors[0]:.2f} read at the same frame, {errors[best]:.2f} read {best} frames later")

    action = data["action"]
    for j, name in enumerate(names):
        print(f"  {name:18s} action exactly −100: {int((action[:, j] == -100).sum()):3d}, exactly 100: {int((action[:, j] == 100).sum()):3d}, "
              f"exactly 0: {int((action[:, j] == 0).sum()):3d}")

    holding = (action[:, 5] < 3) & (data["observation.state"][:, 5] > 5)      # told to close, but the jaws stay open
    print(f"gripper told to close (action < 3) but measured above 5 in {holding.sum()} frames, in "
          f"{len(np.unique(data['episode_index'][holding]))} episodes; measured there: median {np.median(data['observation.state'][holding, 5]):.1f}")

    stats = dataset.meta.stats["action"]
    value = action[0, 0]
    print(f"normalising shoulder_pan action {value:.2f}: ({value:.2f} − {stats['mean'][0]:.2f}) / {stats['std'][0]:.2f} "
          f"= {normalise(value, stats['mean'][0], stats['std'][0]):.3f}")

    # One training batch as ACT asks for it: each frame with the next 100 actions (a chunk of 3.3 s).
    chunked = open_dataset(delta_timestamps={"action": [i / dataset.fps for i in range(100)]})
    loader = torch.utils.data.DataLoader(chunked, batch_size=8, shuffle=True, generator=torch.Generator().manual_seed(0))
    batch = next(iter(loader))
    for key in ("observation.images.up", "observation.images.side", "observation.state", "action", "action_is_pad"):
        print(f"  batch {key:26s} {tuple(batch[key].shape)}")
    left = lengths[data["episode_index"]] - data["frame_index"]               # frames left in the episode, this one included
    print(f"frames whose 100-action chunk runs past the episode's end: {(left < 100).sum()} of {len(left)}")

LeRobotDataset downloads the files from the Hugging Face Hub once, then reads them from disk:

• The parquet file is a table with one row per frame: the joint numbers, the action, the time and the episode. All 50 episodes take .
• The two videos hold every camera frame of all 50 episodes, back to back. They are almost all of the .
• meta/ holds the description (info.json), where each episode starts, and the statistics used for normalisation.

Why video files, not one picture per frame? Video compression stores mostly what changes from one frame to the next, so the frames of each camera fit in about 43 MB. LeRobot decodes a frame only when it is asked for.

One frame

dataset[0] returns the first frame as a dictionary of tensors:

The two camera images arrive decoded, as (3, 480, 640) tensors with values from 0 to 1: channels, height, width, as in Convolutional networks.

An episode

Episode 0 lasts frames, about ten seconds. Step through it: the cameras, and one joint’s action and measured state.

Action and state

Robot basics showed, in simulation, that the follower trails its targets: the strong follower reached the leader’s angle 4 frames later. Is a real SO-101 the same?

For every joint, compare the state with the action a few frames earlier, averaged over all 50 episodes:

Every joint matches best when the state is read 4 frames after the action: 133 ms, the same lag as the simulated follower.

The gripper shows the other effect from Robot basics, the gap that remains. While a brick is held, the action says “close” (median ), but the measured jaws stay at .

That happens in frames, in every one of the episodes. The brick is in the way, and the servo keeps pushing.

So a policy that learns the action column learns what to command, which is not the same as where the arm will be.

Units

LeRobot 0.6.1 reports SO-101 joints in degrees by default. This dataset was recorded on LeRobot’s other scale, and its numbers are not degrees. The data says so itself:

A continuous reading lands exactly on −100.0 or 100.0 only if something cut it off there. In LeRobot’s other scale, RANGE_M100_100, every joint is cut to its calibrated range and mapped to −100 … 100, and the gripper to 0 … 100:

The degree scale, from Robot basics’ Calibration, has no such cut-off. So these numbers are percentages of each joint’s calibrated range.

The dataset also names its robot so100_follower: the SO-100 and SO-101 share one LeRobot driver.

Why it matters: recordings you make with LeRobot 0.6.1 will be in degrees by default. A policy trained on this dataset expects −100 … 100 numbers, and mixing the two scales would quietly mean different poses.

Units

what a number means physically. Check them in the data before trusting a column’s name.

Normalisation

Before any number reaches ACT, its preprocessor rescales it. For joints and actions:

def normalise(value, mean, std, eps=1e-8):
    """What ACT's preprocessor does to every joint value and action before the network sees it."""
    return (value - mean) / (std + eps)

The mean and standard deviation of every column are in meta/stats.json:

The reason is the one from Optimizers: numbers of very different sizes make a hard loss surface. After rescaling, every joint is centred near 0 with a spread near 1.

Images are handled differently. LeRobot’s training script defaults to use_imagenet_stats=True: each colour channel is rescaled with ImageNet’s means and standard deviations, the ones ResNet-18 was pretrained with, not this dataset’s own:

Normalisation

rescaling inputs and outputs to mean 0 and spread 1 before training, and back afterwards.

One batch

ACT predicts a chunk of the next 100 actions, not one. LeRobot builds those chunks when you ask for them by time offsets:

chunked = open_dataset(delta_timestamps={"action": [i / dataset.fps for i in range(100)]})
loader = torch.utils.data.DataLoader(chunked, batch_size=8, shuffle=True, generator=torch.Generator().manual_seed(0))
batch = next(iter(loader))

• delta_timestamps asks, for every frame, for the actions at 0, 1/30, 2/30 … 99/30 seconds later: 3.3 s of future.
• batch_size=8 is LeRobot’s default training batch size.

Near the end of an episode there are fewer than 100 actions left. LeRobot repeats the last one and marks the copies in action_is_pad.

That affects of the frames, the last 99 of every episode. ACT’s L1 loss leaves the marked actions out.

This is ACT’s whole input for one training step: two camera images, six joint numbers, and the 100 actions it should have produced.

Recap

• LeRobotDataset(REPO, video_backend="pyav") downloads 86 MB once: a parquet table, two videos and meta/.
• dataset[i] returns one frame: two decoded images, observation.state, action and its place in time.
• lag(columns, joint) finds that the measured state trails the action by 4 frames on every joint.
• Exact −100, 100 and 0 values show the joints are in LeRobot’s −100 … 100 scale, not degrees.
• normalise(value, mean, std) is what ACT does to joints and actions; images use ImageNet’s statistics.
• delta_timestamps turns each frame into a training example with a chunk of 100 future actions and their padding flags.

Next, ACT itself: what it does with this batch, how it trains, and how well it works once it drives a robot.