Neural network: several inputs

Overview

Here is the network this lesson explains:

return nn.Sequential(nn.Linear(2, 4), nn.ReLU(), nn.Linear(4, 3))

"""A network that chooses: T-shirt size S, M or L from height and weight.

Run: python3 site/ml/foundations/examples/pytorch_choices.py
Every person comes with the right size, so this is the PyTorch basics loop again,
with two inputs instead of one and three choices instead of one number.
"""

import torch
from torch import nn

SIZES = ["S", "M", "L"]


def size_chart(height, weight):
    """The shop's chart: the right answers we train on (made up for this lesson)."""
    if height > 180 or weight > 85:
        return 2                                   # L
    if height < 165 and weight < 60:
        return 0                                   # S
    return 1                                       # M


def make_people(count, generator):
    heights = 150 + 45 * torch.rand(count, generator=generator)    # 150-195 cm
    weights = 45 + 60 * torch.rand(count, generator=generator)     # 45-105 kg
    sizes = torch.tensor([size_chart(h, w) for h, w in zip(heights.tolist(), weights.tolist())])
    return torch.stack([heights, weights], dim=1), sizes


def scale(people):
    """Centre both inputs near 0 and give them a similar spread, so one learning rate suits both."""
    return (people - torch.tensor([170.0, 75.0])) / torch.tensor([10.0, 15.0])


def make_network():
    return nn.Sequential(nn.Linear(2, 4), nn.ReLU(), nn.Linear(4, 3))


def train(seed=0, steps=3000, lr=0.5, observe=None):
    torch.set_num_threads(1)                                       # same numbers on every machine
    torch.manual_seed(seed)
    generator = torch.Generator().manual_seed(seed)
    people, answers = make_people(80, generator)
    inputs = scale(people)
    network = make_network()
    optimizer = torch.optim.SGD(network.parameters(), lr=lr)
    for step in range(steps):
        scores = network(inputs)                                   # 80 × 3: one score per size
        log_probabilities = torch.log_softmax(scores, dim=1)       # softmax, then log
        right = log_probabilities.gather(1, answers.unsqueeze(1))  # log p of the right size
        loss = -right.mean()
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        if observe:
            observe(step + 1, network, loss.item())
    return network, people, answers


if __name__ == "__main__":
    def report(step, network, loss):
        if step in (1, 100, 500, 3000):
            print(f"step {step:4d}: loss {loss:.3f}")

    network, people, answers = train(observe=report)
    with torch.no_grad():
        # nn.CrossEntropyLoss is the usual one-call name for the same loss.
        scores = network(scale(people))
        print("same loss from nn.CrossEntropyLoss:", round(nn.CrossEntropyLoss()(scores, answers).item(), 3))
        picked = scores.argmax(dim=1)
        print(f"training people sized correctly: {(picked == answers).sum().item()} of {len(answers)}")
        new_people, new_answers = make_people(200, torch.Generator().manual_seed(99))
        new_picked = network(scale(new_people)).argmax(dim=1)
        print(f"200 new people sized correctly: {(new_picked == new_answers).sum().item()}")
        person = torch.tensor([[178.0, 70.0]])
        probabilities = torch.softmax(network(scale(person)), dim=1)[0]
        print("178 cm, 70 kg:", ", ".join(f"{s} {p:.2f}" for s, p in zip(SIZES, probabilities.tolist())))

The training loop is the one from PyTorch basics. Two things are new: neurons that read more than one input, and an output that is a choice instead of a number.

Data

The shop has a size chart:

• L if you are taller than 180 cm or heavier than 85 kg;
• S if you are shorter than 165 cm and lighter than 60 kg;
• M otherwise.

We made up 80 customers and sized each one with the chart.

The network never sees the chart itself. It only gets the 80 examples, each a customer with the right size, and has to find a rule that gives the same answers.

def size_chart(height, weight):
    """The shop's chart: the right answers we train on (made up for this lesson)."""
    if height > 180 or weight > 85:
        return 2                                   # L
    if height < 165 and weight < 60:
        return 0                                   # S
    return 1                                       # M

The sizes are stored as numbers: 0 for S, 1 for M, 2 for L.

Notice the corners in the chart. A single straight line cannot separate M from S and L, so the network will need its hidden layer.

Supervised learning

learning from examples that come with the right answers. PyTorch basics, the one-input lesson and this lesson are all supervised learning.

Classification

predicting which of a few categories something belongs to, like a size. Predicting a number, as in the one-input lesson, is called regression.

Two inputs

Before the network sees them, both inputs are rescaled to small numbers around zero:

return (people - torch.tensor([170.0, 75.0])) / torch.tensor([10.0, 15.0])

A height of 178 cm becomes \((178-170)/10 = 0.8\) and a weight of 70 kg becomes \((70-75)/15 = -0.33\). Without this, height (around 170) would swamp weight (around 75) in every \(z\), and one learning rate could not suit both.

In the one-input lesson every hidden neuron had one input, so it held one weight and one bias.

Here each hidden neuron reads both inputs, so it holds one weight per input: two weights and one bias.

Call the two rescaled inputs \(x_1\) (height) and \(x_2\) (weight), and a neuron's numbers \(w_1\), \(w_2\) and \(b\). The neuron computes:

\(z\) is the weighted sum of the inputs plus the bias; \(h\) is what the neuron passes on. Here is hidden neuron 1 of the trained network, working on that 178 cm, 70 kg customer:

Neuron 4 does the same with its own \(w_1\), \(w_2\) and \(b\), gets a negative \(z\), and ReLU turns it into 0: for this customer, that neuron is switched off.

Weighted sum (z)

each input times its weight, added up, plus the bias: \(w_1 x_1 + w_2 x_2 + b\). It is what a neuron computes before ReLU.

Each neuron owns the lines coming into it, plus its bias:

• Hidden layer, nn.Linear(2, 4): 4 neurons, each with 2 weights and 1 bias. 12 numbers.
• Output layer, nn.Linear(4, 3): 3 neurons, one per size, each with 4 weights (one per hidden neuron) and 1 bias. 15 numbers.
• Total: 27 numbers to learn.

What does one hidden neuron do now? In the one-input lesson, a neuron with one input bent at one point: where its \(z\) crossed zero.

With two inputs, \(z\) is zero along a line across the height–weight map. On one side the neuron passes \(z\) on; on the other, ReLU makes it 0.

The Training section draws all four lines on the map.

Softmax

Why three outputs, and not one output that says 0, 1 or 2? One number would treat the sizes as points on a line:

• M would sit halfway between S and L;
• “S for an L customer” would count as twice as wrong as “M”;
• the network could answer 1.5, which is not a size at all.

For categories with no order, like cat, dog and bird, one number makes no sense at all.

So the network gets one output per answer, and each answer gets its own score.

The output layer gives three numbers, one per size. They are scores: any value, positive or negative, and they do not add up to anything.

To pick a size, we want probabilities. Click anywhere on the map to place a customer and watch their scores turn into probabilities.

Softmax does two steps:

• Make every score positive with \(e^{\text{score}}\). A bigger score gives a much bigger number.
• Divide each by the total, so the three shares add up to 1.

For our 178 cm, 70 kg customer, the untrained network gives:

Move the three scores yourself and watch both steps:

Three things to try:

• Raise one score by 1. Its \(e^{\text{score}}\) grows about 2.7 times, so its share grows and the others shrink. The shares always add up to 1.
• Push one score far above the others. Its probability approaches 1.
• Set all three equal. Each gets ⅓, whatever the value.

As a formula, for size \(k\) with score \(s_k\):

Score

a raw output number, one per choice. Bigger means “more likely”, but it is not a probability yet.

Softmax

turns a list of scores into probabilities that add up to 1; bigger score, bigger share.

The untrained network leans towards L for this customer, but the right answer is M. How wrong is that, as one number?

Loss for choices

In the one-input lesson the loss was the squared distance between the guess and the right number. For a choice there is no distance, but there is a probability: how much did the network give to the right answer? The loss is minus the log of that probability:

The untrained network gave M a probability of , so the loss is \(-\log\) \(=\) .

After training it gives M , and the loss is .

A probability of 1 for the right answer would give a loss of 0. The less it gives the right answer, the faster the loss grows. So lowering the loss means raising the probability of the right answer.

scores = network(inputs)                                   # 80 × 3: one score per size
log_probabilities = torch.log_softmax(scores, dim=1)       # softmax, then log
right = log_probabilities.gather(1, answers.unsqueeze(1))  # log p of the right size
loss = -right.mean()

• scores = network(inputs) gives a table with one row per customer and one score per size: 80 × 3.
• torch.log_softmax(scores, dim=1) applies softmax across each row (dim=1 means across the three sizes), then takes the log. PyTorch does both in one step because it is more accurate than torch.log(torch.softmax(...)).
• answers.unsqueeze(1) turns the 80 right answers into an 80 × 1 column, the shape gather needs.
• gather(1, ...) picks, in each row, the log-probability in the right answer's column: the right size for each customer.
• loss = -right.mean() averages the 80 values of \(-\log p_{\text{right}}\) into one number.

This loss has a standard name, and PyTorch has a one-call version: nn.CrossEntropyLoss()(scores, answers) computes exactly the same number. The example file prints both to show they match.

Cross-entropy loss

the average of \(-\log p_{\text{right}}\): small when the network gives the right answer a high probability.

Training

The rest is the PyTorch basics loop, line for line: SGD at learning rate 0.5, all 80 customers in every step, 3,000 steps.

network = make_network()
optimizer = torch.optim.SGD(network.parameters(), lr=lr)
for step in range(steps):
    scores = network(inputs)                                   # 80 × 3: one score per size
    log_probabilities = torch.log_softmax(scores, dim=1)       # softmax, then log
    right = log_probabilities.gather(1, answers.unsqueeze(1))  # log p of the right size
    loss = -right.mean()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Scrub through the run:

• the background shows the size the network would pick at every height and weight;
• the numbered lines are the four hidden neurons.

Predict: where will the boundaries end up?

At the start, the four lines sit wherever their random weights put them. The network picks the same size almost everywhere.

As training runs, the lines move and the coloured regions follow the chart's corners. Nobody placed the lines; training did, by lowering the loss.

Each neuron's job is its line: which side of it a customer is on, and how far past it. The output layer then weighs those four answers to score each size.

That is why the coloured edges are not the neuron lines themselves. An edge falls wherever two sizes end up with equal scores, a mix of all four neurons.

The trained network sizes all of the 80 customers correctly.

On 200 new customers it never saw, made up the same way, it gets right.

Nine of its ten misses are M customers weighing between 80 and 85 kg, just under the chart's 85 kg line. The network put its boundary between M and L a few kilograms too low there.

Every number

The whole network is 27 numbers, and nothing else is hidden. Where 27 comes from:

The output layer is easy to forget: S, M and L are neurons too. Each reads the four hidden outputs \(h_1 \dots h_4\), so it has four weights, written \(v_1 \dots v_4\) as in the one-input lesson, and a bias \(c\):

Here are all 27, with what each neuron computes for one customer. Click the map to choose a customer, and switch between before and after training.

How to read the two tables:

• Hidden neurons: each row computes \(z = w_1 x_1 + w_2 x_2 + b\) from the customer's two rescaled inputs. ReLU turns a negative \(z\) into 0, giving \(h\).
• Output neurons: each size computes its score from the four \(h\) values, with its \(v_1 \dots v_4\) and \(c\). There is no ReLU here, so a score can be negative.
• Probability: softmax turns the three scores into the last column.

Training changed these 27 numbers and nothing else. Switch to before training to see where they started.

Recap

• scale(people) rescales height and weight to small numbers around zero.
• nn.Linear(2, 4) is the hidden layer: 4 neurons, each with 2 weights (one per input) and a bias.
• nn.ReLU() switches a neuron off on one side of its line.
• nn.Linear(4, 3) is the output layer: one score per size.
• torch.log_softmax(scores, dim=1) turns scores into log-probabilities.
• gather(1, answers.unsqueeze(1)) picks the right answer's log-probability for each customer.
• loss = -right.mean() is the cross-entropy loss; zero_grad(), backward() and step() are unchanged.

In one line each:

• a neuron with many inputs holds one weight per input;
• an output that is a choice gives scores;
• softmax makes them probabilities;
• the loss is \(-\log\) of the right answer's probability.

The next lesson, Gradients and backpropagation, shows how backward() finds the direction for every one of those weights.