Gradients and backpropagation

Overview

We keep the model from PyTorch basics. Input \(x = 2\), wanted answer \(5\), weight \(w = 1\), bias \(b = 0\):

\[ \text{guess} = w \cdot x + b = 1 \cdot 2 + 0 = 2, \qquad \text{loss} = (\text{guess} - 5)^2 = 9 \]

The guess is too low. But we cannot change the guess directly; we can only change \(w\) and \(b\).

So the real questions are:

• which of them should change;
• should each go up or down;
• and by how much?

Parameters

There is no single correction for the model. Every parameter, here \(w\) and \(b\), in a network every weight and every bias, gets its own: its own direction and its own size. Each one is moved by the same rule, the update from PyTorch basics:

\[ w \leftarrow w - \text{lr} \cdot \frac{\partial\, \text{loss}}{\partial w}, \qquad b \leftarrow b - \text{lr} \cdot \frac{\partial\, \text{loss}}{\partial b} \]

\(\frac{\partial\, \text{loss}}{\partial w}\) is read “the partial derivative of the loss with respect to \(w\)”.

Partial derivative

how fast the loss changes when one parameter alone changes a little.

Gradient

the partial derivatives for all parameters, together.

The rest of this lesson is about finding those numbers.

Nudging

The simplest way to find a parameter’s direction is to ask it one question: “if I nudge you up a little, does the loss go up or down?” Try it with \(w\) and with \(b\):

Both nudges lower the loss, so both parameters should go up. The rate (loss change divided by the nudge) is the partial derivative.

It tells us two things:

• Its sign is the direction: negative means “raising me lowers the loss”.
• Its size is how strongly this parameter matters. Here \(w\) matters twice as much, because its change is multiplied by the input 2.

Each parameter has its own loss curve: change only that parameter, keep the other fixed, and plot the loss.

The partial derivative is the slope of that curve at the current point. The update walks downhill on each curve separately.

Change the input, the answer or the starting values and watch both slopes:

Now set the input to \(x = -2\). The guess is \(-2\), still too low, and \(b\) should still go up.

But \(w\) should go down: raising \(w\) would make \(w \cdot (-2)\) even more negative.

So the right direction for a weight depends on the input it multiplies. You cannot read it off the miss alone; you have to ask each parameter.

Chain rule

Nudging works, but it is slow and only approximate: one extra forward pass per parameter, for every single update.

Instead we use the fact that every step of the calculation is simple math whose rate we know exactly.

The loss depends on the guess, and the guess depends on \(w\) and \(b\):

\[ \frac{\partial\, \text{loss}}{\partial\, \text{guess}} = 2(\text{guess} - \text{answer}), \qquad \frac{\partial\, \text{guess}}{\partial w} = x, \qquad \frac{\partial\, \text{guess}}{\partial b} = 1 \]

The chain rule says: to get the rate along a chain, multiply the rates of its links. If raising the guess lowers the loss 6 times as fast, and raising \(w\) raises the guess 2 times as fast, then raising \(w\) lowers the loss \(6 \times 2 = 12\) times as fast:

\[ \frac{\partial\, \text{loss}}{\partial w} = \frac{\partial\, \text{loss}}{\partial\, \text{guess}} \cdot \frac{\partial\, \text{guess}}{\partial w} = (-6) \cdot 2 = -12, \qquad \frac{\partial\, \text{loss}}{\partial b} = (-6) \cdot 1 = -6 \]

Exactly the nudge results, without any nudging.

A useful way to picture it is passing blame backwards. The loss tells the guess “you are responsible at rate \(-6\)”. The guess passes that blame on to its inputs, multiplied by how strongly each one affects it.

That is all backpropagation is, repeated through a whole network.

Backpropagation

Here is the real training iteration. There is no step that picks out a “key” parameter, and nothing is nudged:

Let’s watch it on the kind of network from the previous lesson, made small enough to show every number: one input, one hidden ReLU neuron, one linear output neuron. Four parameters, \(w_1, b_1\) in the hidden neuron and \(v, c\) in the output:

\[ z = w_1 x + b_1, \qquad h = \mathrm{ReLU}(z), \qquad y = v\,h + c, \qquad \text{loss} = (y - 5)^2 \]

Read it as two neurons in a row:

• The hidden neuron makes the bend: \(z = w_1 x + b_1\), then \(h = \mathrm{ReLU}(z)\).
• The output neuron is the output layer: \(y = v\,h + c\), a weighted sum with no activation. \(y\) is the network’s output.

Why not stop at \(y = h\)? Because \(h\) can never be negative and has no scale or shift of its own. The output layer adds both; in a wider network it is also what adds the hidden ramps together.

It has no ReLU, so the answer can take any value, positive or negative (Neural network: one input explains why).

Press Next. The forward pass fills in values from left to right; the backward pass then hands blame from right to left, one link at a time.

Each link multiplies the blame arriving from its right by one local fact, usually a value it remembered from the forward pass.

At the end, all four parameters update together.

Three things to notice:

• Every step needs only local information: the blame from its right, and a value it already had.
• Blame computed once is reused. \(\partial L / \partial y = -6\) served \(c\), \(v\) and \(h\). In a big network, a hidden neuron’s blame is computed once and serves every weight feeding into it.
• A switched-off neuron passes no blame. Switch to neuron off: \(z = -1\), so ReLU outputs 0 and passes back \(\times 0\). No blame reaches \(w_1\), \(b_1\) or \(v\) (because \(h = 0\)); only \(c\) moves.

That is the dead neuron from Neural network: one input, seen from the inside.

Is one sweep really the same as nudging every parameter? Here are both, side by side:

They agree; the sweep is exact, the nudges are approximate, and the sweep cost one pass instead of four.

Training loop

One iteration moved all four parameters a little and cut the loss from 9 to 5.59. Training is that iteration, repeated: forward with the updated parameters, a new loss, a new backward sweep, a new update.

The parameters with large gradients move most in the early iterations.

As the guess approaches 5, every gradient shrinks, so every move shrinks, and the loop settles by itself. Nobody chose which parameter to change: each one followed its own gradient every time.

In code, one iteration is five lines:

guesses = model(batch_inputs)            # forward: PyTorch records every operation
loss = loss_fn(guesses, batch_answers)   # one number for the whole batch
optimizer.zero_grad()                    # clear the previous iteration's gradients
loss.backward()                          # one sweep, loss → inputs: a gradient for every parameter
optimizer.step()                         # update every parameter at once

backward() knows the chain because of the forward pass.

Every operation PyTorch applies (multiply, add, ReLU, the squared loss) comes with its local derivative built in, and PyTorch records which operation was applied to which values.

Computational graph

that record of operations and values: exactly the diagram in the stepper above.

Autograd

the part of PyTorch that walks the computational graph backwards.

Speed

Compare the cost of one update:

• Nudging needs one extra forward pass for every parameter.
• Backpropagation needs one forward pass and one backward pass. A backward pass costs roughly as much as two forward passes, whatever the number of parameters.

Slide through network sizes; the time assumes one forward pass takes one millisecond:

The gap grows with the network: for a billion parameters, nudging would take about 12 days for one update; backpropagation takes milliseconds and gives exact gradients. That difference is why neural networks with millions of parameters can be trained at all.

Batches

So far there was one example. With a batch, every example gets its own guess, its own miss, and its own partial derivatives: its vote for each parameter.

The batch loss is the average of the examples’ losses, and the rate of an average is the average of the rates.

So the batch’s gradient is simply the average of the votes:

\[ \frac{\partial\, \text{loss}_{\text{batch}}}{\partial w} = \frac{1}{n} \sum_{i=1}^{n} \frac{\partial\, \text{loss}_i}{\partial w}, \qquad \frac{\partial\, \text{loss}_i}{\partial w} = 2(\text{guess}_i - \text{answer}_i)\, x_i \]

Here are the three examples from PyTorch basics, inputs 1, 2, 3, with \(w = 1\) and \(b = 0\). Drag an answer and watch its vote change; push one answer below its guess and that example votes the other way:

At the defaults:

• the votes for \(w\) are \(-4, -12, -24\), averaging \(-13.33\);
• the votes for \(b\) are \(-4, -6, -8\), averaging \(-6\).

One update with learning rate 0.05 gives \(w = 1 - 0.05 \cdot (-13.33) = 1.667\) and \(b = 0.3\): exactly the first step of the training loop in PyTorch basics.

When examples disagree, the average settles it, weighted by how strongly each one votes. That is why a batch gives a steadier direction than any single example.

This is also where the name of PyTorch’s optimizer comes from.

Stochastic gradient descent (SGD)

shuffle the examples, cut them into random batches, and step every parameter against each batch’s averaged gradient.

The name in two parts:

• Gradient descent: every parameter steps downhill against its own gradient.
• Stochastic (random): each step’s direction comes from a random sample of the data, so it is a slightly noisy estimate of the direction for the whole dataset.

The batch size gives the variants their names:

• batches of one example: the original SGD;
• batches of a few hundred: mini-batch gradient descent, the usual choice;
• the whole dataset at once: full-batch gradient descent.

One detail: torch.optim.SGD only does the step, \(p \leftarrow p - \text{lr} \cdot \partial\,\text{loss}/\partial p\), with whatever gradient is in .grad. The shuffling and batching, the “stochastic” part, live in your training loop.

In PyTorch

PyTorch does all of this for you.

• loss.backward() runs the chain rule backwards from the loss to every parameter, and writes each partial derivative into that parameter’s .grad.
• For a batch, it computes every example’s contribution and averages them, all at once with tensors, with no Python loop.

The example file checks it against the numbers on this page:

for i in range(3):
    model.zero_grad()
    loss_fn(model(inputs[i:i + 1]), answers[i:i + 1]).backward()
    print(model.weight.grad.item(), model.bias.grad.item())   # -4 -4, then -12 -6, then -24 -8

model.zero_grad()
loss_fn(model(inputs), answers).backward()                     # the mean loss over all three rows
print(model.weight.grad.item(), model.bias.grad.item())       # -13.33 -6.0: the average of the votes

"""Which way to correct each parameter: nudges, the chain rule, PyTorch's backward(), and a batch.

Run: python3 examples/pytorch_gradients.py
The model is the one from PyTorch basics: guess = weight * input + bias, loss = squared miss.
"""

import torch
from torch import nn


def loss_at(w, b, x, answer):
    return (w * x + b - answer) ** 2


# 1. Nudge each parameter a little and watch the loss.
x, answer, w, b = 2.0, 5.0, 1.0, 0.0
base = loss_at(w, b, x, answer)
print(f"guess {w * x + b:.2f}, loss {base:.4f}")
for name, nudged in [("w", loss_at(w + 0.01, b, x, answer)), ("b", loss_at(w, b + 0.01, x, answer))]:
    print(f"  nudge {name} up by 0.01: loss {nudged:.4f}, change {nudged - base:+.4f}, rate {(nudged - base) / 0.01:+.2f}")

# 2. The chain rule by hand: how the loss reacts to the guess, times how the guess reacts to each parameter.
guess = w * x + b
dloss_dguess = 2 * (guess - answer)
print(f"chain: dloss/dguess = {dloss_dguess:+.0f}; dloss/dw = {dloss_dguess:+.0f} x {x:.0f} = {dloss_dguess * x:+.0f}; "
      f"dloss/db = {dloss_dguess:+.0f} x 1 = {dloss_dguess:+.0f}")

# 3. PyTorch's backward() finds the same numbers, for any input.
loss_fn = nn.MSELoss()
for x in (2.0, -2.0):
    model = nn.Linear(1, 1)
    with torch.no_grad():
        model.weight.fill_(1.0)
        model.bias.fill_(0.0)
    loss = loss_fn(model(torch.tensor([[x]])), torch.tensor([[5.0]]))
    loss.backward()
    print(f"backward() with input {x:+.0f}: weight gradient {model.weight.grad.item():+.0f}, "
          f"bias gradient {model.bias.grad.item():+.0f}")

# 4. A batch: each example's own gradient (its vote), and backward() on the batch's mean loss.
inputs = torch.tensor([[1.0], [2.0], [3.0]])
answers = torch.tensor([[3.0], [5.0], [7.0]])
model = nn.Linear(1, 1)
with torch.no_grad():
    model.weight.fill_(1.0)
    model.bias.fill_(0.0)
for i in range(3):
    model.zero_grad()
    loss_fn(model(inputs[i:i + 1]), answers[i:i + 1]).backward()
    print(f"example {i + 1}: vote for weight {model.weight.grad.item():+.2f}, for bias {model.bias.grad.item():+.2f}")
model.zero_grad()
loss_fn(model(inputs), answers).backward()      # the mean loss over all three rows
print(f"whole batch:  weight {model.weight.grad.item():+.2f}, bias {model.bias.grad.item():+.2f}  (the average of the votes)")

# 5. Backpropagation in a tiny network: x -> z = w1*x + b1 -> h = ReLU(z) -> y = v*h + c -> loss.
def tiny(w1, b1, v, c):
    net = nn.Sequential(nn.Linear(1, 1), nn.ReLU(), nn.Linear(1, 1))
    with torch.no_grad():
        net[0].weight.fill_(w1); net[0].bias.fill_(b1)
        net[2].weight.fill_(v); net[2].bias.fill_(c)
    return net


def grads(net):
    return [p.grad.item() for p in (net[0].weight, net[0].bias, net[2].weight, net[2].bias)]


x, answer = torch.tensor([[2.0]]), torch.tensor([[5.0]])
for label, start in [("neuron on ", (1.0, 0.0, 1.0, 0.0)), ("neuron off", (1.0, -3.0, 1.0, 0.0))]:
    net = tiny(*start)
    loss = loss_fn(net(x), answer)                  # forward: PyTorch records every operation
    loss.backward()                                 # one sweep: a gradient for every parameter
    print(f"{label}: loss {loss.item():.2f}, gradients w1 b1 v c = " + " ".join(f"{g:+.0f}" for g in grads(net)))

# Check the sweep against nudging each parameter (neuron on), then update all four at once.
net = tiny(1.0, 0.0, 1.0, 0.0)
base = loss_fn(net(x), answer).item()
nudged = []
for p in net.parameters():
    with torch.no_grad():
        p += 0.01
        nudged.append((loss_fn(net(x), answer).item() - base) / 0.01)
        p -= 0.01
print("nudge estimates w1 b1 v c = " + " ".join(f"{g:+.2f}" for g in nudged))

optimizer = torch.optim.SGD(net.parameters(), lr=0.01)
losses = []
for iteration in range(20):
    loss = loss_fn(net(x), answer)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    losses.append(loss.item())
    if iteration == 0:
        print("after one update: w1 b1 v c = " + " ".join(f"{p.item():.2f}" for p in net.parameters()),
              f"-> new loss {loss_fn(net(x), answer).item():.2f}")
print("loss over 20 iterations:", " ".join(f"{v:.3f}" for v in losses))

The first loop asks each example separately; the second call asks the whole batch at once. The answers agree, which is the whole point: a batch is not a different kind of learning, just every example’s correction averaged into one step.

Recap

One training iteration, in the order it happens:

• guesses = model(batch_inputs) is the forward pass: input → network → output. PyTorch records every operation and remembers every intermediate value: the computational graph.
• loss = loss_fn(guesses, batch_answers) turns the whole batch into one number: how wrong the model is right now.
• optimizer.zero_grad() clears the previous iteration’s gradients, because PyTorch adds new gradients to old ones.
• loss.backward() is backpropagation: one sweep from the loss back through the graph. Each link multiplies the blame arriving from its right by its local rate, so every weight and bias ends up with its own \(\partial\,\text{loss}/\partial p\) in .grad, averaged over the batch.
• optimizer.step() updates every parameter at once, each by its own amount: \(p \leftarrow p - \text{lr} \cdot \partial\,\text{loss}/\partial p\). Big gradient, big move; zero gradient, no move.

Then the next batch, with the updated parameters. Nobody picks a “key” parameter: the gradient decides, for every parameter, every iteration.