Robot arm · Q-table

Overview

Here the robot must reach a cube with a two-joint arm, and a barrier makes the direct route fail.

In Catch, every drop ended after seven moves, and all seven simply shared the final score. The arm may need hundreds of commands before it touches the cube, so one shared score says little about which commands helped.

Instead, the arm learns how much each command is worth from each pose, one step at a time. That is a second way to learn from rewards, and it starts with a table.

World

Each command turns one joint by ten degrees. Watch the arm and the blue dot together:

• shoulder changes move the dot across the map;
• elbow changes move it up and down.

You can also select an allowed pose on the map.

State

the two joint angles.

Action

one of the four joint commands.

The arm itself:

• two links, each 10 cm long;
• the shoulder ranges from 0° to 180°;
• the elbow from −180° to 0°, relative to the first link.

Nineteen settings per joint make 361 pose slots.

The map’s colours:

• blue squares are allowed poses;
• orange squares reach the target;
• grey squares are forbidden poses: a link crosses the table or the barrier.

A command into a forbidden pose is rejected, and the arm stays where it is.

after, reward, done = step(state, action)

"""The existing hg-robot arm, made into a small reusable environment.

Angles are on a 10-degree grid; links are 10 cm. A forbidden destination
rejects the command. This checks poses, not the continuous motion between them.
No dynamics or physical contact simulation is performed.
"""
import numpy as np

ANGLES = np.arange(0, 181, 10)
ELBOW = np.arange(-180, 1, 10)
ACTIONS = np.array([[1, 0], [-1, 0], [0, 1], [0, -1]])
ACTION_NAMES = ["Shoulder +10°", "Shoulder −10°", "Elbow +10°", "Elbow −10°"]
HOME = (12, 6)
TARGET = np.array([14.0, 4.0])
GAMMA = 0.9


def forward(shoulder, elbow):
    a, b = np.deg2rad([shoulder, shoulder + elbow])
    joint = 10 * np.array([np.cos(a), np.sin(a)])
    return joint, joint + 10 * np.array([np.cos(b), np.sin(b)])


def forbidden(shoulder, elbow):
    joint, hand = forward(shoulder, elbow)
    fraction = np.linspace(0, 1, 60)[:, None]
    points = np.concatenate([fraction * joint, joint + fraction * (hand - joint)])
    below = (points[:, 1] < -1e-9).any()
    barrier = ((points[:, 0] >= 9) & (points[:, 0] <= 10.5)
               & (points[:, 1] >= 0) & (points[:, 1] <= 5)).any()
    return bool(below or barrier)


BLOCKED = np.array([[forbidden(a, b) for b in ELBOW] for a in ANGLES])
DISTANCE = np.array([[np.linalg.norm(forward(a, b)[1] - TARGET) for b in ELBOW] for a in ANGLES])
GOAL = (DISTANCE < 1.5) & ~BLOCKED
STARTS = [tuple(map(int, x)) for x in np.argwhere(~BLOCKED & ~GOAL)]


def step(state, action):
    i, j = np.array(state) + ACTIONS[action]
    if not (0 <= i < 19 and 0 <= j < 19) or BLOCKED[i, j]:
        return tuple(state), 0.0, False
    return (int(i), int(j)), float(GOAL[i, j]), bool(GOAL[i, j])


def encode(states):
    return np.asarray(states, dtype=np.float32) / 9 - 1


def rollout(values, start=HOME, limit=60):
    state = tuple(start)
    path = [state]
    rejected = 0
    for _ in range(limit):
        action = int(np.argmax(values[state]))
        after, reward, done = step(state, action)
        rejected += after == state
        path.append(after)
        state = after
        if done:
            return {"path": path, "success": True, "moves": len(path) - 1, "rejected": rejected}
    return {"path": path, "success": False, "moves": limit, "rejected": rejected}


def evaluate(values):
    trials = [rollout(values, s) for s in STARTS]
    return {"trials": len(trials), "successes": sum(t["success"] for t in trials),
            "home": rollout(values), "starts": [t["success"] for t in trials]}


def reference():
    """Full-model value iteration: an evaluation baseline, never training labels."""
    values = np.zeros((19, 19))
    for _ in range(100):
        q = np.zeros((19, 19, 4))
        for state in STARTS:
            for action in range(4):
                after, reward, done = step(state, action)
                q[state][action] = reward + (0 if done else GAMMA * values[after])
        new = q.max(axis=2)
        if np.max(np.abs(new - values)) < 1e-10:
            break
        values = new
    return q


if __name__ == "__main__":
    q = reference()
    print(f"{len(STARTS)} nonterminal starts; {int(GOAL.sum())} goal poses")
    print("Reference from home:", rollout(q))
    print("Home value:", q[HOME].max())

The dashed circle shows the reaching zone. An allowed pose within 1.5 cm of the target at (14, 4) cm gives reward = 1 and ends the episode; every other command gives zero.

The target and the barrier stay fixed throughout these two workshops.

Baseline

First try a simple policy: take whichever move brings the hand closest to the cube.

It starts well, then gets stuck above the barrier: every available improvement is blocked.

Reaching needs one move that temporarily takes the hand farther away.

The reference route takes nine moves. We compute it using the whole world to grade the learner; the learner trains from the outcomes of its own commands.

Q-table

Each pose gets four numbers, one per command.

Action value (Q)

an estimate of the reward still to come if the arm makes that command now and chooses well afterwards.

Discount factor

how much a later reward counts: each move of delay multiplies it by 0.9.

So the nine-move reference route from home is worth \(0.9^8 \approx 0.430\): eight zero rewards, then one.

q = np.zeros((19, 19, 4))
values = q[state]
action = int(np.argmax(values))

"""Learn four action values per pose from the arm's own transitions.

Run: python3 classes/ml/labs/robot-arm/q_table.py
Seed 7, 300 episodes, at most 400 moves each. Evaluation freezes Q.
"""
import numpy as np
from world import HOME, GAMMA, step, evaluate


def train(seed=7, episodes=300, observe=None):
    rng = np.random.default_rng(seed)
    q = np.zeros((19, 19, 4))
    lengths, first = [], None
    records = []
    for episode in range(episodes):
        state = HOME
        for move in range(400):
            action = int(rng.integers(4)) if rng.random() < 0.2 else int(np.argmax(q[state] + 1e-9 * rng.random(4)))
            after, reward, done = step(state, action)
            target = reward + (0 if done else GAMMA * q[after].max())
            if reward and first is None:
                first = {"episode": episode + 1, "record": [*state, action, reward, *after, done],
                         "previous": records[-1] if records else None}
            q[state][action] += 0.5 * (target - q[state][action])
            records.append([*state, action, reward, *after, done])
            state = after
            if done:
                break
        lengths.append(move + 1)
        if observe:
            observe(episode + 1, q)
    return {"seed": seed, "episodes": episodes, "lengths": lengths, "first": first,
            "evaluation": evaluate(q), "q": q}


if __name__ == "__main__":
    result = train()
    print("First success episode:", result["first"]["episode"])
    print("Training moves:", sum(result["lengths"]))
    print("Evaluation:", {k: v for k, v in result["evaluation"].items() if k != "starts"})

• q[state] reads the four values for a pose.
• argmax returns the index of the largest one.
• The array stores 1,444 estimates, starting at zero.

Greedy policy

always choose the action with the largest value.

ε-greedy policy

choose a random action with probability ε, and the largest-value action otherwise. Here \(\epsilon=0.2\): 20% random during training.

Equal-value ties are broken randomly.

One update

Here is the actual move that first reached the cube in our recorded run.

Take the move, then update its estimate. Predict which number changes.

Work it through:

• The successful move earns one and ends the episode, so its target is one. Halfway from zero toward one gives 0.5.
• Repeat it: 0.5 moves halfway toward one again, giving 0.75.
• Choose “One move earlier” after the first update: reward zero plus \(0.9\times0.5\) gives a target of 0.45, so its zero estimate becomes 0.225.

after, reward, done = step(state, action)
target = reward + (0 if done else GAMMA * q[after].max())
q[state][action] += 0.5 * (target - q[state][action])

Only the entry for the recorded pose and action changes.

Each update moves one estimate halfway toward the reward plus the discounted best estimate at the next pose. An ending transition never adds a future value.

Q-learning

this way of learning action values: move each estimate toward the reward plus the discounted best next estimate.

Training

Now repeat the process. Each episode starts at home, explores, and stops on success or after 400 commands.

The seed fixes the random choices, so you can reproduce this run.

This player shows saved Python runs:

• on the map, brighter blue means a higher estimate for the best action at a pose;
• the arm shows the whole route chosen by the frozen greedy policy;
• the curve counts the commands taken during training, exploration included.

Random exploration can make a training episode longer than the learned route.

The first success came in episode two. Across 300 episodes the arm took 7,849 commands.

Its greedy policy now reaches from home in nine moves. Try other starts in the coverage map below.

Test

Freeze the table and turn exploration off: no estimates change during this test.

Start from home, then try another pose, or select one on the map.

The table passes 59 of 251 starts; the full-model reference passes all 251.

• Blue squares pass the 60-command test; red crosses fail it.
• Click a square to see its route.

Training always started at home, so the map shows where the learned choices also work away from that start.

Recap

• step(state, action) is the supplied world; its geometry and rewards stay fixed.
• q = np.zeros((19, 19, 4)) creates the estimates once.
• action comes from the table or a random exploratory choice.
• target comes from observed reward plus estimated continuation.
• q[state][action] moves halfway toward that target after each command.
• evaluate(q) freezes the table and tests greedy choices from every allowed nonterminal start.

The arm learned a route around the barrier without receiving a route as training labels. Next we replace these separate table entries with a neural network that calculates the same four estimates.