Robot basics
A LeRobot recording of the SO-101 arm saves two lists of six numbers, 30 times a second. One is called observation.state, the other action.
They look almost the same. What is each one, and why do they differ?
Overview
The SO-101 is a chain of six small motors. Each motor turns one joint and reports the angle it is at.
The mental model for this lesson: the command is a target angle, and the measured angle chases it.
The running example is the two-link arm from the robot-arm workshops: two 10 cm links. Here it gets mass, gravity and a motor in each joint, simulated in MuJoCo.
Joints
A robot arm is rigid pieces connected by joints. Each SO-101 joint has one motor, so the arm has six numbers to measure and six to command.
- Link
- a rigid piece of the arm, such as the upper arm.
- Joint
- a connection that lets one link turn relative to the one before it.
- Joint position
- how far a joint is turned: an angle in degrees.
The six joints:
- shoulder_pan turns the whole arm left and right.
- shoulder_lift raises and lowers the upper arm.
- elbow_flex bends the elbow.
- wrist_flex tips the hand up and down.
- wrist_roll rotates the hand around its own axis.
- gripper opens and closes the jaws.
shoulder_lift and elbow_flex move the arm within one upright plane. Our MuJoCo arm keeps just those two, with the same 10 cm links as the workshops.
LeRobot records these joint angles, not where the gripper is, because angles are what the motors measure and accept.
Position control
The SO-101’s motors are servos. A servo is given a target angle, and a small controller inside it keeps pushing the joint toward that target.
The push depends on two things:
- The gap, target minus angle: the farther the joint is from its target, the harder the push. A gain \(k_p\) sets how hard.
- The speed: the faster the joint moves, the more the push is reduced, like a brake. A gain \(k_v\) sets how much.
The push is a torque, in N·m. So \(k_p\) is in N·m per radian of gap, and \(k_v\) in N·m·s per radian: \(k_p = 6\) pushes with 6 N·m when the joint is one radian (57°) from its target.
In MuJoCo that controller is one line per joint in the model:
<actuator>
<position name="shoulder" joint="shoulder" kp="{kp}" kv="{kv}"/>
<position name="elbow" joint="elbow" kp="{kp}" kv="{kv}"/>
</actuator>Full file · robot_basics.py · 162 lines
"""Robot basics: a two-link arm with position-controlled joints, simulated in MuJoCo.
Run: python3 site/ml/foundations/examples/robot_basics.py
The arm moves in a vertical plane, like the SO-101's shoulder_lift and elbow_flex joints. Each joint has a
motor that is given a target angle; its controller pushes toward that target. Angles on the page are in
degrees, as in a LeRobot SO-101 dataset; MuJoCo itself works in radians.
"""
import math
import mujoco
import numpy as np
L1, L2 = 0.10, 0.10 # link lengths in metres (the same as the robot-arm workshops)
TIME_STEP = 0.001 # seconds per physics step
FPS = 30 # LeRobot records and commands at 30 frames per second
GAINS = { # kp: N·m per radian of error; kv: N·m per rad/s of speed
"weak P + D": (1.5, 0.3),
"strong P + D": (6.0, 0.4),
"strong P, no D": (6.0, 0.0),
}
def make_model(kp, kv):
xml = f"""
<mujoco model="two-link arm">
<option timestep="{TIME_STEP}" gravity="0 0 -9.81"/>
<worldbody>
<body name="upper_arm">
<joint name="shoulder" type="hinge" axis="0 -1 0" armature="0.02"/>
<geom type="capsule" fromto="0 0 0 {L1} 0 0" size="0.012" mass="0.20" contype="0" conaffinity="0"/>
<body name="forearm" pos="{L1} 0 0">
<joint name="elbow" type="hinge" axis="0 -1 0" armature="0.02"/>
<geom type="capsule" fromto="0 0 0 {L2} 0 0" size="0.010" mass="0.15" contype="0" conaffinity="0"/>
<site name="hand" pos="{L2} 0 0"/>
</body>
</body>
</worldbody>
<actuator>
<position name="shoulder" joint="shoulder" kp="{kp}" kv="{kv}"/>
<position name="elbow" joint="elbow" kp="{kp}" kv="{kv}"/>
</actuator>
</mujoco>
"""
return mujoco.MjModel.from_xml_string(xml)
def angles(data):
return np.degrees(data.qpos).copy() # measured joint angles, in degrees
def push(data):
"""The shoulder motor's torque now: kp × (target − angle) − kv × speed, in N·m."""
return data.actuator_force[0]
def step_response(kp, kv, start=(90.0, 0.0), target=(30.0, 0.0), seconds=2.0):
"""Hold the arm at `start`, then command `target` at time 0. Record the angles every physics step."""
model = make_model(kp, kv)
data = mujoco.MjData(model)
data.qpos[:] = np.radians(start)
data.ctrl[:] = np.radians(start)
for _ in range(3000): # hold the start pose (straight up: gravity has no lever)
mujoco.mj_step(model, data)
data.qvel[:] = 0.0
data.time = 0.0
data.ctrl[:] = np.radians(target) # the command: a target angle for each joint
mujoco.mj_forward(model, data) # the motors react to the new targets
rows = [(data.time, *angles(data), push(data))]
while data.time < seconds - 1e-9:
mujoco.mj_step(model, data) # the position controller pushes toward the target
mujoco.mj_forward(model, data) # refresh the motor torque for the new angles
rows.append((data.time, *angles(data), push(data)))
return rows
def leader_angles(t):
"""A person moving the leader arm: reach out and up in 1 s, hold, come back in 1 s, hold."""
home, reach = np.array([30.0, -100.0]), np.array([80.0, -40.0])
def smooth(s): # 0 → 1 with zero speed at both ends
s = min(max(s, 0.0), 1.0)
return 10 * s**3 - 15 * s**4 + 6 * s**5
if t < 1.5:
return home + (reach - home) * smooth((t - 0.5) / 1.0)
return reach + (home - reach) * smooth((t - 2.0) / 1.0)
def teleoperate(kp, kv, seconds=3.5):
"""The LeRobot record loop at 30 Hz: read the follower, read the leader, send the leader's angles, save both."""
model = make_model(kp, kv)
data = mujoco.MjData(model)
data.qpos[:] = np.radians(leader_angles(0.0))
data.ctrl[:] = np.radians(leader_angles(0.0))
for _ in range(3000): # settle at the start pose before recording
mujoco.mj_step(model, data)
data.qvel[:] = 0.0
data.time = 0.0
frames, trace = [], []
for i in range(round(seconds * FPS)):
t = i / FPS
observation_state = angles(data) # robot.get_observation(): the follower's measured angles
action = leader_angles(t) # teleop.get_action(): the leader's angles
data.ctrl[:] = np.radians(action) # robot.send_action(): they become the follower's targets
frames.append({"frame_index": i, "timestamp": t,
"observation.state": observation_state, "action": action})
while data.time < (i + 1) / FPS - 1e-9: # physics runs on between two frames
mujoco.mj_step(model, data)
trace.append((data.time, *angles(data)))
return frames, trace
def raw_to_degrees(raw, range_min, range_max, resolution=4096):
"""LeRobot's degree reading for an STS3215 servo: 0° is the middle of the calibrated range."""
mid = (range_min + range_max) / 2
return (raw - mid) * 360 / (resolution - 1)
def summary(rows, target):
"""Start, furthest point, final angle and remaining gap of the shoulder, plus the time to cover 90% of the move."""
t = np.array([r[0] for r in rows])
shoulder = np.array([r[1] for r in rows])
start, final = shoulder[0], shoulder[-1]
moved = (shoulder - start) / (target - start) # 0 at the start, 1 at the target
furthest = shoulder[np.argmax(moved)]
return {"start": start, "furthest": furthest, "final": final, "gap": target - final,
"rise_time": t[np.argmax(moved >= 0.9)]}
def gravity_gap(kp, pose_deg):
"""At rest the motor's push kp × gap balances gravity, so gap = gravity torque / kp (radians)."""
model = make_model(kp, 0.0)
data = mujoco.MjData(model)
data.qpos[:] = np.radians(pose_deg)
mujoco.mj_forward(model, data)
torque = data.qfrc_bias[0] # with the arm at rest, this is gravity's torque on the shoulder
return torque, torque / kp
if __name__ == "__main__":
print("Step response, shoulder 90° → 30° (elbow target stays 0°):")
for name, (kp, kv) in GAINS.items():
rows = step_response(kp, kv)
s = summary(rows, 30.0)
print(f" {name:15s} kp {kp:3.1f} kv {kv:4.2f}: first push {rows[0][3]:+.4f} N·m, furthest {s['furthest']:.1f}°, after 2 s {s['final']:.1f}° "
f"(gap {s['gap']:.1f}°, elbow {rows[-1][2]:.1f}°), 90% of the way after {s['rise_time']:.3f} s")
if kv > 0:
torque, gap = gravity_gap(kp, rows[-1][1:3])
print(f" by hand: gravity torque {torque:.4f} N·m / kp {kp} = {gap:.3g} rad = {math.degrees(gap):.1f}°")
print("\nTeleoperation at 30 Hz (strong P + D), shoulder column, degrees:")
frames, _ = teleoperate(*GAINS["strong P + D"])
for f in frames[15:66:5]:
print(f" frame {f['frame_index']:3d} t {f['timestamp']:.3f} s observation.state {f['observation.state'][0]:6.1f}"
f" action {f['action'][0]:6.1f}")
print("\nCalibration: raw counts to degrees (range 1010 … 3070)")
for raw in (1010, 2040, 2500, 3070):
print(f" raw {raw} → {raw_to_degrees(raw, 1010, 3070):.2f}°")
A program then commands the arm by writing target angles, in radians, into data.ctrl. It never sets a torque itself.
Our test: the arm points straight up, then at time 0 the shoulder’s target jumps to 30°. Before pressing play, predict: will the arm stop exactly at 30°?
Step response lab
How to read the lab:
- The blue arm and blue curve are the measured angle.
- The orange dashed arm and line are the target.
- The faint arm is the start, pointing straight up.
The first push, worked by hand for the strong controller: the gap is \(30^\circ - 90^\circ = -60^\circ = -1.0472\) rad, and the speed is zero.
\[6 \times (-1.0472) - 0.4 \times 0 = -6.2832\ \text{N·m}\]Minus means downward. The weak controller, \(k_p = 1.5\), starts at a quarter of that push: N·m.
The three runs, measured:
- Without D, nothing brakes the arm. It swings past the target to −34° and keeps swinging.
- With D, the brake takes out the swing. The strong controller dips past the target once and settles.
- Weak P + D is slower and stops ° short of the target.
Why does the arm stop short of 30°? At rest the speed is zero, so the push is \(k_p \times \text{gap}\). It must hold up the arm against gravity, so a gap must remain:
\[\text{gap} = \frac{\text{gravity torque}}{k_p}\]For the weak controller at its resting pose, MuJoCo reports a gravity torque of N·m on the shoulder. Then the gap is \(0.3035 / 1.5 = 0.202\) rad \(= 11.6^\circ\): exactly the measured gap. The strong controller’s is \(0.2837 / 6 = 0.0473\) rad \(= 2.7^\circ\).
- Position control
- commanding a joint by its target angle; a controller turns the gap into a push.
- PD controller
- a controller whose push is P (proportional) times the gap minus D (derivative) times the speed. \(k_p\) and \(k_v\) are its gains.
- Step response
- how a joint moves after its target jumps: how fast, how far past, and where it settles.
- Steady-state error
- the gap left once the joint stops moving.
The SO-101’s STS3215 servos run the same kind of controller. LeRobot sets their P gain to 16, their D gain to 32 and their I gain to 0, in the servo’s own units. That is why a real arm under load also stops a little short of its target.
Why not a huge \(k_p\), to make the gap vanish? The push on anything in the way grows with \(k_p\) too, so a stiff arm that hits the table presses hard into it. A gain is a trade between accuracy, overshoot and gentleness.
Calibration
A servo does not measure degrees. Its sensor counts 4,096 steps per full turn, from 0 to 4095.
Which count means which pose depends on how the motor was mounted. Two identical arms, in the same pose, report different counts.
That matters for the next section, where one arm copies another: “go to count 2500” would put the two arms in different poses. Calibration gives every arm the same scale. LeRobot does it in two steps:
- Middle: you hold the arm in the middle of its range. Each servo stores an offset so that this pose reads 2047, half a turn.
- Range: you move each joint from one end of its travel to the other. LeRobot records the smallest and largest count.
Every later reading becomes degrees from the middle of that range:
\[\text{middle} = \frac{\text{min} + \text{max}}{2}\] \[\text{degrees} = (\text{count} - \text{middle}) \times \frac{360}{4095}\]For a reading of 2500: \((2500 - 2040) \times 360 / 4095 = 40.44^\circ\). One count is \(360/4095 = 0.088^\circ\), the finest step the arm can measure.
So on the SO-101, 0° means “the middle of this joint’s range”, not “horizontal”. LeRobot reports the five turning joints in degrees and the gripper from 0 (closed) to 100 (open).
- Calibration
- recording each joint’s middle and range, so raw counts become angles that mean the same on every arm.
Teleoperation
To teach the arm a task, a person first does it. LeRobot uses two SO-101 arms for this:
- The leader arm has its motors switched off. The person moves it by hand, and its servos only measure.
- The follower arm has its motors on. Its targets are the leader’s angles.
- Teleoperation
- a person controlling a robot from a distance, here by moving the leader arm.
A loop runs this 30 times a second. Each pass reads the follower, reads the leader, sends the leader’s angles to the follower, and saves both:
for i in range(round(seconds * FPS)):
t = i / FPS
observation_state = angles(data) # robot.get_observation(): the follower's measured angles
action = leader_angles(t) # teleop.get_action(): the leader's angles
data.ctrl[:] = np.radians(action) # robot.send_action(): they become the follower's targets
frames.append({"frame_index": i, "timestamp": t,
"observation.state": observation_state, "action": action})
while data.time < (i + 1) / FPS - 1e-9: # physics runs on between two frames
mujoco.mj_step(model, data)
trace.append((data.time, *angles(data)))Full file: robot_basics.py, in its first card above ↑
- The comments name the LeRobot calls that do the same on the real arms, in the same order.
- Between two frames, 33 ms pass. MuJoCo takes 33 or 34 physics steps of 1 ms, and the controller pushes at every one of them toward the last target.
leader_angles(t)stands in for the person: a reach out and up in one second, a pause, and back.
Predict: when the leader is moving fast, how far behind will the follower be?
Teleoperation lab
How to read the lab:
- Orange steps are the action: the leader’s angle, held for one frame each.
- Blue dots are
observation.state: the follower’s angle at the start of each frame. The thin blue line is the follower between frames. - The table shows the saved rows around the selected frame, in degrees.
At frame 30 the leader’s shoulder is at °, and the strong follower at °. The follower first reaches ° at frame , four frames (133 ms) later.
With the weak follower, the gap grows to ° and the same angle is reached only at frame . While the leader holds still, the gravity gap from Position control stays: ° for the strong follower, ° for the weak one.
So: the action is where the arm was told to go; the observation is where it was. During a fast move they differ by several degrees, and they never quite meet under gravity.
Dataset
A real SO-101 recording has the same two columns, with six numbers each:
| Column | Holds | Units |
|---|---|---|
observation.state | the follower’s six measured joint positions | degrees; gripper 0–100 |
action | the six targets sent to the follower: the leader’s positions, same names | the same |
observation.images.* | one camera frame per camera | pixels |
timestamp, frame_index | when the row was saved | seconds; 1/30 s apart |
The six names, in order: shoulder_pan.pos, shoulder_lift.pos, elbow_flex.pos, wrist_flex.pos, wrist_roll.pos, gripper.pos.
This is what makes the dataset useful for learning. The observation columns are what a robot can see at that moment. The action column is what the person decided to do.
- Policy
- the rule that picks an action from an observation, as in the reinforcement-learning lessons. Here it will be a network.
A network trained to produce the action column from the observation columns can then replace the person and the leader arm. That is imitation learning, the next workshop: Robot arm · imitation learning.
Why learn the action and not the next observation? The action is the only thing the robot can send. The motors and gravity then decide where the arm ends up, as the lab showed.
Recap
make_model(kp, kv)builds the two-link arm: two hinge joints, 10 cm links with mass, gravity, and apositionmotor on each joint with gainskpandkv.data.ctrl[:] = np.radians(target)is the command: target angles. The controller turns each gap into a push at every 1 ms physics step.step_response(kp, kv)shows the result of one command: overshoot without D, a gravity gap of \(\text{gravity torque} / k_p\) at rest.raw_to_degrees(raw, range_min, range_max)is LeRobot’s calibration: counts from the middle of the recorded range, times \(360/4095\).teleoperate(kp, kv)runs the 30 Hz record loop. Each frame savesobservation.state(the follower, measured) andaction(the leader, the follower’s target).
The SO-101 measures six joint angles and accepts six target angles. Its dataset saves both, and a policy learns to produce the targets.
Key concepts
- Link · joint
- A rigid piece of the arm · a connection that lets the next link turn.
- Joint position
- How far a joint is turned, in degrees.
- Servo
- A motor with an angle sensor and a controller built in, commanded by a target angle.
- Position control
- Commanding a joint by its target angle.
- PD controller
- Push = \(k_p\) × gap − \(k_v\) × speed.
- Step response
- How a joint moves after its target jumps.
- Steady-state error
- The gap left at rest; under gravity, gravity torque / \(k_p\).
- Calibration
- Recording each joint’s middle and range, so counts become comparable angles.
- Teleoperation
- A person controlling the robot, here through a leader arm.
- Leader · follower
- The arm moved by hand, motors off · the arm that copies it, motors on.
- Control frequency
- How often a new command is sent: 30 times a second in LeRobot recordings.
- observation.state · action
- The follower’s measured joint positions · the targets sent to it.