How a GPU organises its threads

A GPU, or graphics processing unit, is a chip built to run thousands of simple threads at once. It groups them into blocks, and the blocks into a grid. Every thread works out which piece of the data is its own from its place in that grid.

This lab has no GPU. It shows the same arrangement in plain Python, so you can follow every thread by eye.

A CPU, the central processing unit, has a few powerful cores. Each is good at running one tricky sequence of steps quickly.

A GPU has thousands of simple cores. One of its threads is slower than a CPU's, but thousands run together. That wins when the job is the same small step on a great many independent items.

Every thread runs the same function, so it must find its own piece of the data. It does that with one line of arithmetic:

global index = block number x block size + thread number within the block

The thread knows its block number and its place in the block. The block size is the same for every block.

# Which element does each thread own? 2 blocks of 5 threads.
block_dim = 5
for block_idx in range(2):
    for thread_idx in range(block_dim):
        i = block_idx * block_dim + thread_idx
        print("block", block_idx, "thread", thread_idx, "owns element", i)
# Change block_dim to 3 and the number of blocks to 4, then run again.

The lab's kernel.py launches 3 blocks of 4 threads over 10 elements. It prints the block, the thread and the global index of each of the 12 threads.

Twelve threads cover ten elements, so the last two have no element. The script tells you which ones. The second Run works one index by hand: block 3, thread 1, block size 4, then 7 turned back into a block and a thread.

docker exec jupyter python3 steps/kernel.py

docker exec jupyter python3 -c "print(3 * 4 + 1, 7 // 4, 7 % 4)"