Generalization and evaluation

Overview

In the several-inputs lesson the network learned the chart from 80 customers, then sized 190 of 200 new customers correctly. That went well for three reasons:

• Clean labels: every answer came straight from the chart.
• A small network: 4 hidden neurons cannot memorise single customers.
• Familiar customers: the new ones came from the same height and weight range.

Real data rarely grants all three. This lesson breaks them one at a time: messy labels (Records), a network big enough to memorise them (Overfitting), and customers from outside the usual range (Distribution shift).

The mental model: a model is only graded fairly on data that played no part in building it, and only for the kind of data that data represents.

Records

In the several-inputs lesson the network learned from the shop’s chart, which gives one right size for every height and weight. Now the shop learns from its sales records: the size each customer actually kept.

Most kept their chart size. But kept a neighbouring size, because they like it looser or tighter.

def sales_records(people, generator):
    """The size each customer kept: the chart's size, or for 15% of them one size up or down."""
    chart = torch.tensor([size_chart(h, w) for h, w in people.tolist()])
    other = torch.rand(len(chart), generator=generator) < KEPT_OTHER
    step = torch.where(torch.rand(len(chart), generator=generator) < 0.5, -1, 1)
    return torch.where(other, (chart + step).clamp(0, 2), chart)

"""Will the T-shirt network work on customers it has not seen? Train, validation and test sets, overfitting,
early stopping and distribution shift, on the shop from Neural network: several inputs.

Run: python3 site/ml/foundations/examples/pytorch_generalization.py

The shop now learns from its sales records instead of its chart: what each customer actually kept.
Most kept their chart size, but 15% kept a neighbouring size (they like it looser or tighter).
"""

import copy

import torch
from torch import nn

from pytorch_choices import size_chart, make_people, scale

KEPT_OTHER = 0.15     # share of customers who kept a neighbouring size


def sales_records(people, generator):
    """The size each customer kept: the chart's size, or for 15% of them one size up or down."""
    chart = torch.tensor([size_chart(h, w) for h, w in people.tolist()])
    other = torch.rand(len(chart), generator=generator) < KEPT_OTHER
    step = torch.where(torch.rand(len(chart), generator=generator) < 0.5, -1, 1)
    return torch.where(other, (chart + step).clamp(0, 2), chart)


def customers(count, generator, heights=(150, 195), weights=(45, 105)):
    """`count` customers in a height and weight range, with their records. The default is the shop's usual range."""
    if (heights, weights) == ((150, 195), (45, 105)):
        people, _ = make_people(count, generator)
    else:
        h = heights[0] + (heights[1] - heights[0]) * torch.rand(count, generator=generator)
        w = weights[0] + (weights[1] - weights[0]) * torch.rand(count, generator=generator)
        people = torch.stack([h, w], dim=1)
    return people, sales_records(people, generator)


def split(seed):
    """40 customers to train on, 40 for validation, and a fixed test set of 1,000 used once at the end."""
    generator = torch.Generator().manual_seed(seed)
    train_set = customers(40, generator)
    validation_set = customers(40, generator)
    test_set = customers(1000, torch.Generator().manual_seed(1000))
    return train_set, validation_set, test_set


def make_network(hidden=64):
    """Far more parameters (4,547 with 64) than the 40 customers it learns from."""
    return nn.Sequential(nn.Linear(2, hidden), nn.ReLU(), nn.Linear(hidden, hidden), nn.ReLU(), nn.Linear(hidden, 3))


def accuracy(network, data):
    people, sizes = data
    with torch.no_grad():
        return (network(scale(people)).argmax(dim=1) == sizes).float().mean().item()


def train(seed=0, steps=3000, lr=0.05, hidden=64, observe=None):
    """Plain SGD on all 40 training customers at once. Early stopping keeps the network from the first step
    with the highest validation accuracy."""
    torch.set_num_threads(1)
    torch.manual_seed(seed)
    train_set, validation_set, _ = split(seed)
    network = make_network(hidden)
    optimizer = torch.optim.SGD(network.parameters(), lr=lr)
    loss_fn = nn.CrossEntropyLoss()
    inputs, answers = scale(train_set[0]), train_set[1]
    best_accuracy, best_step, best = -1.0, 0, None
    for step in range(steps + 1):
        validation_accuracy = accuracy(network, validation_set)
        if validation_accuracy > best_accuracy:              # early stopping: remember the best network so far
            best_accuracy, best_step, best = validation_accuracy, step, copy.deepcopy(network)
        if observe:
            observe(step, network)
        if step == steps:
            break
        loss = loss_fn(network(inputs), answers)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    return network, best, best_step


SHIFTS = {"taller and heavier": ((195, 215), (105, 130)), "smaller": ((125, 150), (25, 45))}


def report(seed):
    train_set, validation_set, test_set = split(seed)
    final, best, best_step = train(seed)
    chart = lambda data: (torch.tensor([size_chart(h, w) for h, w in data[0].tolist()]) == data[1]).float().mean().item()
    print(f"seed {seed}: the chart itself matches {chart(test_set):.3f} of the 1,000 test records")
    print(f"  after 3000 steps: training accuracy {accuracy(final, train_set):.3f}, validation {accuracy(final, validation_set):.3f}")
    print(f"  early stopping: highest validation accuracy first at step {best_step}; training {accuracy(best, train_set):.3f}, "
          f"validation {accuracy(best, validation_set):.3f}")
    print(f"  test, used once: early-stopped {accuracy(best, test_set):.3f}, after 3000 steps {accuracy(final, test_set):.3f}")
    for name, (heights, weights) in SHIFTS.items():
        shifted = customers(500, torch.Generator().manual_seed(2000), heights, weights)
        print(f"  {name} customers ({heights[0]}-{heights[1]} cm, {weights[0]}-{weights[1]} kg): "
              f"early-stopped {accuracy(best, shifted):.3f}, chart {chart(shifted):.3f}")


if __name__ == "__main__":
    report(6)        # the seed the overfitting lab shows
    report(0)        # the seed the shift lab opens on: the best test score, and the smaller customers mostly wrong

• size_chart is the several-inputs lesson’s chart, unchanged.
• other marks the customers who kept a different size; step says one size down or up.
• clamp(0, 2) keeps the size between S and L.

No model can predict a customer’s taste from height and weight. So even the chart itself matches only of the test records: that is the best score to hope for.

Three sets

The shop has records of 80 customers to learn from. It splits them before training starts, and draws a third, separate set:

Training set

the 40 customers the network learns from.

Validation set

40 other customers, used to make choices: when to stop training, which network size.

Test set

1,000 customers, used once, at the very end, to grade the final choice.

def split(seed):
    """40 customers to train on, 40 for validation, and a fixed test set of 1,000 used once at the end."""
    generator = torch.Generator().manual_seed(seed)
    train_set = customers(40, generator)
    validation_set = customers(40, generator)
    test_set = customers(1000, torch.Generator().manual_seed(1000))
    return train_set, validation_set, test_set

Why two held-back sets, not one? Every choice made by looking at a set fits the model a little to that set. After many choices, its score there is too optimistic, so a final grade needs data nobody has looked at.

Overfitting

The network is much bigger than the several-inputs lesson’s: two hidden layers of 64 ReLU neurons, weights and biases for 40 customers. It trains with plain SGD, rate 0.05, on all 40 at once, for 3,000 steps.

Predict: as training goes on, will the validation customers be sized better, worse, or first better then worse?

Early stopping

The validation curve gives the cure. Keep a copy of the network from the step where the validation customers were sized best, and use that copy, not the last one.

for step in range(steps + 1):
    validation_accuracy = accuracy(network, validation_set)
    if validation_accuracy > best_accuracy:              # early stopping: remember the best network so far
        best_accuracy, best_step, best = validation_accuracy, step, copy.deepcopy(network)

• accuracy(network, validation_set) sizes the 40 validation customers with the current weights; nothing is trained on them.
• copy.deepcopy(network) saves the whole network whenever validation accuracy beats its best so far. A tie keeps the earlier step.

We stop on accuracy because sizing customers right is the shop’s goal; the loss also rewards being sure. For the seed shown above, the best step is ; the lab marks it.

Early stopping

keeping the model from the training step with the best validation score.

Test

Now, once, the 1,000 test customers. For every seed, the early-stopped network against the one after all 3,000 steps:

• Early stopping scores higher in of 10 seeds: median against .
• Training accuracy said 100% for most final networks; the test says about three in four.
• Seeds differ by several points, so one run is not a result. Every seed is in the table.

Both stay well below the chart’s : 40 customers are simply too few for a network to find the chart’s straight edges.

And once the test score is read, it must not be used to choose again. Picking the seed with the best test score would turn the test set into a second validation set, and the score into an optimistic one.

Distribution shift

The shop now sells to two new groups, outside the range of every customer so far (150–195 cm, 45–105 kg):

• Taller and heavier: 195–215 cm, 105–130 kg. The chart says L.
• Smaller: 125–150 cm, 25–45 kg. The chart says S.

Predict: will a network that scores well on the test set also size these customers well?

Recap

• sales_records(people, generator) gives realistic labels: the chart’s size, except for the 15% who kept another.
• split(seed) makes the three sets: 40 to train on, 40 to choose with, 1,000 to grade once.
• make_network(64) is deliberately large, with 4,547 parameters for 40 customers.
• train(seed) runs plain SGD and keeps best, the copy from the step with the highest validation accuracy.
• accuracy(best, test_set) is the grade, read once, for every seed.
• customers(500, ..., heights, weights) builds the shifted groups; their scores hold the warning that no in-range number gives.

Hold data back, watch training and validation part ways, stop where validation is best, test once, report every seed. Then remember that the grade only covers the kind of data it was measured on.

You will use these ideas again:

• The bouncing-ball workshop grades its network on 100 throws it never trained on. It also compares its designs on those same throws, so by this lesson’s names they serve as a validation set.
• The robot-arm and Catch lessons find that a low training loss does not mean the task succeeds.
• The imitation workshop holds back demonstration episodes to choose its network size, and meets distribution shift on a robot arm.