Convolutions and pooling explained

A convolution slides a small grid of numbers, called a filter, across an image and records how strongly each patch matches a pattern. It is how a convolutional neural network (CNN) finds edges and shapes in a photo without anyone describing them.

The same idea works on a row of numbers, so we start with one.

Take a row of numbers and a short filter. Place the filter at the start. Multiply each number under it by the matching filter weight, and add. That is one output.

Slide the filter one place along and repeat. Our row is dark then bright: 0, 0, 0, 9, 9, 9. Our filter is -1 and 1, so each output says how much brighter the next number is.

docker exec jupyter python3 -c "import numpy as np; print(np.correlate([0, 0, 0, 9, 9, 9], [-1, 1], 'valid').tolist())"

A grey photo is a grid of pixels. Each pixel is one number, here from 0 for black to 9 for bright. A 2 by 2 filter covers four pixels at a time. It moves along a row, then down to the next.

A 4 by 4 image and a 2 by 2 filter give a 3 by 3 result. The size is the image size minus the filter size plus 1.

image = [[0, 0, 9, 9], [0, 0, 9, 9], [0, 0, 9, 9], [0, 0, 9, 9]]
filt = [[-1, 1], [-1, 1]]

for r in range(3):
    row = []
    for c in range(3):
        total = 0
        for a in range(2):
            for b in range(2):
                total = total + image[r + a][c + b] * filt[a][b]
        row.append(total)
    print(row)
# Change the filter to [[-1, -1], [1, 1]] and press Run.
# This one looks for a horizontal edge. What do you see?

Max pooling keeps the biggest value in each small block. A 2 by 2 pool turns a 4 by 4 feature map into a 2 by 2 map.

It keeps the strongest signal and forgets exactly where it was. That makes the next layer smaller, and less fussy about where a pattern sits.

docker exec jupyter python3 -c "import numpy as np; print(np.array([[1, 3, 2, 0], [4, 6, 1, 1], [0, 2, 5, 7], [1, 1, 3, 2]]).reshape(2, 2, 2, 2).max(axis=(1, 3)).tolist())"

A CNN repeats convolution, ReLU and pooling, then ends with ordinary dense layers that give the answer. Early filters tend to find edges. Later ones combine edges into shapes.

A filter's weights are learned by gradient descent, like any other weight. They are shared across the whole image, which keeps the count small.

In a notebook cell, call np.correlate([1, 2, 3, 4, 5], [1, 1, 1], 'valid') after import numpy as np. Work out the three answers by hand first.

What does a filter of three ones do to a row? Then try a filter of 1, 0 and -1, and say what it measures.