Layers
A wide row of neurons can already trace a shape. A second row becomes useful when it reads the first row’s answers and makes a new quantity that the raw input did not provide.
AFTER THIS LESSON By the end, you can follow a number through two layers and explain why a later layer can use an answer made by an earlier one.
Intermediate features
Bob may care less about the exact thermometer reading than about how far his room is from comfortable. We can ask two small questions first: “How far below 16°C?” and “How far above 24°C?” Their answers are never negative. Inside the comfortable interval, both are zero. On either side, one begins to grow.
These two neurons form a first layer because they read the same input side by side. Add their answers and you have a new quantity: distance from the comfortable range. This number keeps the information Bob needs for the next decision while discarding which side of comfort he was on.
Different rooms, identical middle answer
heat distance 6 + cool distance 0distance 6heat distance 0 + cool distance 6distance 6turbo = max(0, 6 − 6)0 extraFeature combinations
Suppose Bob wants extra effort only when the room is more than 6 degrees beyond the comfortable interval. One more neuron can read the distance and add a boost above that point. At 8°C and 32°C, the distance is 8, so turbo adds 2 in either case. The later neuron does not need a separate cold rule and hot rule because the earlier answers made the common feature explicit.
This is the reason for stacking layers: an answer from one stage becomes the input to another. A deep network can build complicated answers in stages, although its real internal features may be less tidy than our hand-built “distance” label. All numbers in this example are still chosen by us. The next lessons ask how a machine can learn them.
Try the layers
Cold and hot rooms can produce the same middle answer. The turbo reads that answer, not the thermometer.
At 10°C the room is 6 degrees below comfortable. The turbo sees 6 and stays off.
Predict before you reveal: what does turbo output at 32°C?
The cooler distance is 32 − 24 = 8, while the heater distance is zero. The next layer receives 8 and outputs 8 − 6 = 2 extra. Try 8°C: it also gives distance 8 and turbo 2.
WHAT CHANGED? The second layer receives a useful answer made by the first layer, instead of reading the temperature directly.
Key concepts
- Layer
- A group of neurons that reads the same set of inputs and produces a set of answers. The first layer here reads temperature.
- Hidden representation
- The intermediate numbers a network makes for itself. “Distance from comfortable” is more useful to the turbo than raw temperature.
- Depth
- The number of processing stages chained together. A later stage reads earlier answers instead of starting again from the raw input.
The compact version, once you have seen it
heat = max(0, 16 − temperature)cool = max(0, temperature − 24)distance = heat + coolturbo = max(0, distance − 6)
A layer is a set of neurons reading the same inputs. A deeper layer reads the previous layer’s outputs.