A network is a stack of simple sums
A neural network is a program built from layers of small units called neurons, which a deep neural network stacks several layers high. Knowing what each layer does turns the network from a black box into sums you can follow with a pencil.
A hire shop can feed in tomorrow's temperature, rain and day of the week. The network gives back one number: busy or quiet.
By the end of this lesson
Note
I can describe what the layers of a neural network do.
Key words
| Word | What it means | Example |
|---|---|---|
| Neuron | a small unit that turns several numbers into one | |
| Layer | a row of neurons that work side by side | |
| Weight | how much one input counts | 3 |
| Bias | a number added to shift the total | -1 |
| Activation function | a rule that bends a neuron's total | ReLU |
| Deep | having two or more hidden layers |
The cells students fill in are left blank in this preview.
A neuron adds weighted inputs, then decides
A neuron takes several numbers in and gives one number out. It multiplies each input by its weight and adds the results. Then it adds its bias.
A big positive weight means that input pushes the total up. A negative weight pushes it down. A weight of 0 means the input is ignored.
Worked example: one neuron on one day
| Input | Value | Weight | Product |
|---|---|---|---|
| Temperature in tens of degrees | 2 | 3 | 6 |
| Rain in cm | 1 | -4 | -4 |
| Weekend | 1 | 2 | 2 |
| Bias | none | none | -1 |
| Total | none | none | 3 |
The cells students fill in are left blank in this preview.
Change the rain and run it
# Inputs: temperature in tens of degrees, rain in cm, weekend (1 or 0).
inputs = [2, 1, 1]
weights = [3, -4, 2]
bias = -1
total = bias
for n in range(3):
total = total + weights[n] * inputs[n]
print("total:", total)
# Change the rain from 1 to 0 and press Run.
# The total should rise. By how much?
Layers stack neurons into a network
Neurons sit in layers. The input layer holds the raw numbers and does no sums. A hidden layer is a row of neurons between the input and the answer. The output layer gives the answer.
Every neuron reads every number from the layer before. A network with two or more hidden layers is called deep.
Every connection has a weight to learn
Each line in a diagram is a weight. Each neuron also has one bias. These numbers are the network's parameters, and training is the job of setting them.
A layer with 3 inputs and 4 neurons has 12 weights and 4 biases. The table counts a whole network.
Worked example: counting a small network
| Layer | Inputs in | Neurons | Weights | Biases |
|---|---|---|---|---|
| Hidden layer 1 | 3 | 4 | 12 | 4 |
| Hidden layer 2 | 4 | 4 | 16 | 4 |
| Output layer | 4 | 1 | 4 | 1 |
| All layers | none | none | 32 | 9 |
The cells students fill in are left blank in this preview.
Count the parameters with Python
docker exec jupyter python3 -c "print(sum(a * b + b for a, b in zip([3, 4, 4], [4, 4, 1])))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
An activation function bends the total
If every neuron only added, a whole network could be squashed into one neuron. An activation function is a rule applied to each neuron's total. It is what makes extra layers worth having.
Step gives 1 for a positive total, else 0. ReLU gives the total if it is positive, else 0. A sigmoid gives a smooth S-shaped number between 0 and 1.
Worked example: three rules on five totals
| Total | Step | ReLU | Hard sigmoid |
|---|---|---|---|
| -3 | 0 | 0 | 0 |
| -1 | 0 | 0 | 0.25 |
| 0 | 0 | 0 | 0.5 |
| 1 | 1 | 1 | 0.75 |
| 3 | 1 | 3 | 1 |
The cells students fill in are left blank in this preview.
Worked example: two plain layers equal one
| Input | Layer 1 (times 2) | Layer 2 (times 3) | One layer (times 6) |
|---|---|---|---|
| 1 | 2 | 6 | 6 |
| 2 | 4 | 12 | 12 |
| 3 | 6 | 18 | 18 |
The cells students fill in are left blank in this preview.
ReLU on five totals
docker exec jupyter python3 -c "print([max(0, x) for x in [-3, -1, 0, 1, 3]])"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think a deep network understands its data more deeply. Deep only counts hidden layers. More layers let it combine simple patterns into richer ones, but it can still be wrong. Judge a network by how it does on rows it has never seen.
Going further
In a notebook cell, build MLPClassifier(hidden_layer_sizes=(4, 3), max_iter=2000) from sklearn.neural_network and fit it on any three-column table of numbers with a 0 or 1 label. Then print [w.shape for w in net.coefs_].
Predict the three shapes before you run it, using the counting table in this lesson.