Following one day through the network
A forward pass is how a neural network turns its inputs into an answer, one layer at a time. Once you can do one by hand, you can check any small network and know what it will say before you run it.
Each layer takes the numbers from the layer before, mixes them with its weights and passes the result on.
By the end of this lesson
Note
I can work out what a small neural network outputs, layer by layer.
Key words
| Word | What it means | Example |
|---|---|---|
| Forward pass | numbers moving from the inputs to the output | |
| Hidden layer | neurons between the inputs and the output | |
| Output layer | the last layer, which gives the score | |
| ReLU | gives the total if it is positive, else 0 | |
| Score | the output number before the final decision | 5 |
The cells students fill in are left blank in this preview.
Each layer works on the layer before it
To pass a day through a layer, work one neuron at a time. Multiply each input by that neuron's weight, add them, then add its bias. Apply the activation.
The results are the inputs of the next layer. The last layer's result is the score. Our network has two hidden neurons that use ReLU, and one output neuron that gives the raw score.
Worked example: a warm weekend day
| Neuron | Sum | Total | After ReLU |
|---|---|---|---|
| Hidden 1 | 1x2 + -2x1 + 3x1 + 0 | 3 | 3 |
| Hidden 2 | -1x2 + 1x1 + 0x1 + 0 | -1 | 0 |
| Output | 2x3 + -3x0 + -1 | 5 | score 5 |
The cells students fill in are left blank in this preview.
Worked example: a cold wet weekday
| Neuron | Sum | Total | After ReLU |
|---|---|---|---|
| Hidden 1 | 1x1 + -2x3 + 3x0 + 0 | -5 | 0 |
| Hidden 2 | -1x1 + 1x3 + 0x0 + 0 | 2 | 2 |
| Output | 2x0 + -3x2 + -1 | -7 | score -7 |
The cells students fill in are left blank in this preview.
The whole network in Python
# Inputs: temperature in tens of degrees, rain in cm, weekend.
inputs = [2, 1, 1]
hidden_rows = [[1, -2, 3], [-1, 1, 0]]
hidden_bias = [0, 0]
out_row = [2, -3]
out_bias = -1
hidden = []
for r in range(2):
total = hidden_bias[r]
for n in range(3):
total = total + hidden_rows[r][n] * inputs[n]
if total < 0:
total = 0
hidden.append(total)
score = out_bias
for r in range(2):
score = score + out_row[r] * hidden[r]
print("hidden:", hidden)
print("score:", score)
# Change the weekend input from 1 to 0 and press Run.
# Does the score stay above 0?
A layer is a table of weights
A layer's weights fit in a table with one row for each neuron. Each row holds that neuron's weights, one for each input. The biases are a list with one number for each neuron.
A layer with 3 inputs and 2 neurons has 2 rows of 3 numbers. A library multiplies the table by the inputs in one step.
Worked example: the shape of each table
| Layer | Inputs | Neurons | Rows by columns | Weights |
|---|---|---|---|---|
| Hidden | 3 | 2 | 2 by 3 | 6 |
| Output | 2 | 1 | 1 by 2 | 2 |
The cells students fill in are left blank in this preview.
One layer in one step with NumPy
docker exec jupyter python3 -c "import numpy as np; print(np.dot([[1, -2, 3], [-1, 1, 0]], [2, 1, 1]).tolist())"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
The last layer gives a score
The output layer gives a score. For a yes or no question, a score above 0 means yes, here a busy day. A score of 0 or less means no.
With more than two answers, the layer has one score for each answer and the biggest wins. Many networks turn the scores into shares that add up to 1. That step is called softmax.
Worked example: three answers, three scores
| Answer | Score | Chosen |
|---|---|---|
| Quiet | 1 | no |
| Normal | 4 | yes, the biggest |
| Busy | 2 | no |
The cells students fill in are left blank in this preview.
Find the biggest score
docker exec jupyter python3 -c "scores = [1, 4, 2]; print(scores.index(max(scores)))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think the forward pass changes the weights. It does not. The weights stay fixed while numbers flow through. They change only when the network learns from its errors, which is the next lesson.
Going further
Invent a third day for the network in the code block, such as a mild dry weekday with inputs [2, 0, 0]. Work out both hidden totals and the score by hand first.
Then change the code to check you. A good check: if you add 1 to a hidden bias, which score changes, and by how much?