What is an eigenvector?
An eigenvector of a matrix is a direction that the matrix only stretches and never turns. Machine learning uses eigenvectors to find the directions in which data varies most, so that a few numbers can stand in for many.
This lesson shows how to find one by repeated multiplication, and what it is used for.
By the end of this lesson
Note
I can say what an eigenvector is and why machine learning uses them.
Key words
| Word | What it means |
|---|---|
| Eigenvector | a direction a matrix only stretches |
| Eigenvalue | how much an eigenvector is stretched |
| Power iteration | multiplying again and again to find the biggest eigenvector |
| Correlation matrix | a grid of how strongly each pair of measurements move together |
| Dimension reduction | keeping fewer numbers for each item |
The cells students fill in are left blank in this preview.
An eigenvector keeps its direction
Most vectors turn when a matrix multiplies them. A few special ones keep their direction and only get longer or shorter.
For the matrix [[2, 1], [1, 2]], the vector [1, 1] becomes [3, 3] and the vector [1, -1] stays [1, -1]. Those stretch factors, 3 and 1, are the eigenvalues.
Worked example: three vectors and one matrix
| Vector | The matrix times it | Same direction? | Stretch |
|---|---|---|---|
| [1, 1] | [3, 3] | Yes | 3 |
| [1, -1] | [1, -1] | Yes | 1 |
| [1, 0] | [2, 1] | No | Not an eigenvector |
The cells students fill in are left blank in this preview.
Multiply two special vectors
docker exec jupyter python3 -c "import numpy as np; m = np.array([[2, 1], [1, 2]]); print((m @ np.array([1, 1])).tolist(), (m @ np.array([1, -1])).tolist())"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Ask NumPy for the eigenvalues
docker exec jupyter python3 -c "import numpy as np; print(np.linalg.eigvalsh(np.array([[2, 1], [1, 2]])).round(3).tolist())"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Power iteration finds the biggest eigenvector
Start with any vector. Multiply it by the matrix, again and again. Each step leans the vector more towards the eigenvector with the biggest eigenvalue. This is power iteration.
To stop the numbers growing, divide the vector by its length after each step. The length of the product before you divide is then the eigenvalue.
Worked example: five steps from [1, 0]
| Step | The vector | First number divided by second |
|---|---|---|
| 0 | [1, 0] | |
| 1 | [2, 1] | 2 |
| 2 | [5, 4] | 1.25 |
| 3 | [14, 13] | 1.08 |
| 4 | [41, 40] | 1.03 |
The cells students fill in are left blank in this preview.
Take ten steps in one product
docker exec jupyter python3 -c "import numpy as np; m = np.array([[2, 1], [1, 2]]); w = np.linalg.matrix_power(m, 10) @ np.array([1, 0]); print(w.tolist(), round(w[0] / w[1], 4))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
The vector leans towards [1, 1]
The Run printed [29525, 29524] 1.0. After ten steps the two numbers are almost equal, so the vector points almost exactly along [1, 1].
That is the eigenvector with the biggest eigenvalue, 3. The numbers grew about threefold each step, which is the eigenvalue at work.
A common mistake
Warning
Many people think power iteration finds every eigenvector. It finds only the one with the biggest eigenvalue. Other eigenvectors need other methods, such as a library function.
Run power iteration in Python
# Power iteration on [[2, 1], [1, 2]], starting from [1, 0].
m = [[2, 1], [1, 2]]
v = [1.0, 0.0]
for step in range(10):
w = [m[0][0] * v[0] + m[0][1] * v[1], m[1][0] * v[0] + m[1][1] * v[1]]
size = (w[0] * w[0] + w[1] * w[1]) ** 0.5
v = [w[0] / size, w[1] / size]
print(round(size, 3))
print([round(v[0], 3), round(v[1], 3)])
# Change 10 to 3, then to 1. Watch how the size and the vector change.
Machine learning finds the directions of most spread
A data set has several measurements for each row. Which directions hold most of the variation? Build the correlation matrix, a grid of how strongly each pair of measurements moves together.
Its eigenvectors are the directions of spread. A bigger eigenvalue means more spread along that direction. Keep the biggest few and drop the rest: that is dimension reduction, the idea behind principal component analysis.
Find the spread of temperature and hires
docker exec jupyter python3 -c "import numpy as np, pandas as pd; h = pd.read_csv('hires.csv'); print(np.linalg.eigvalsh(np.corrcoef(h['temp_c'], h['hires'])).round(2).tolist())"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Worked example: how the spread divides
| Direction | Eigenvalue | Share of the total spread |
|---|---|---|
| Biggest | 1.77 | 89 in 100 |
| Smallest | 0.23 | 11 in 100 |
| Total | 2.00 | 100 in 100 |
The cells students fill in are left blank in this preview.
One direction could stand in for two numbers
The Run printed [0.23, 1.77]. The two measurements, temperature and hires, move together closely, so one direction holds about 89 in 100 of the spread.
Keeping only that direction turns two numbers per day into one. Machine learning does the same with hundreds of measurements.
Where each idea of the module turns up
| Idea | Where it is used |
|---|---|
| Vectors | Each row of data, each pixel colour, each word count |
| Dot product | Scoring how alike two documents or songs are |
| Matrix product | One layer of a neural network, many predictions at once |
| Transformations | Turning, scaling and moving shapes on a screen |
| Systems of equations | Fitting a line or a model to data |
| Eigenvectors | Finding the directions that matter, to reduce the data |
The cells students fill in are left blank in this preview.
Going further
In your own notebook, make the correlation matrix of temp_c, rain_mm and hires with np.corrcoef and ask np.linalg.eigvalsh for its three eigenvalues.
Work out what share of the total spread the biggest one holds. Say how many numbers per day you could keep.