The vector space model and cosine ranking

The vector space model turns every document and every query into a list of numbers, then ranks the documents by the angle between them. A small angle means the document is about the same things as the query.

This is how a search system ranks pages of different lengths fairly. By the end you will rank four documents by cosine similarity.

Fix a vocabulary, a list of words in a set order. For each word, count it in the text. The list of counts is the text's vector.

A query is treated as a very short document. "late bus" has one "late" and one "bus", so it has a vector too, and now it can be compared with every document.

vocabulary = ["bus", "train", "late", "timetable", "library"]

def to_vector(tokens):
    vector = []
    for word in vocabulary:
        vector.append(tokens.count(word))
    return vector

print(to_vector(["bus", "bus", "late"]))
print(to_vector(["late", "bus"]))
# Make a vector for the words "library" and "timetable".

Multiply the two vectors place by place, then add the results. A word that both have adds a positive number. A word only one has adds 0.

Query and document 1: 1 times 2, plus 1 times 1, is 3. Query and document 4 share nothing, so the dot product is 0.

A long document has a big dot product just for repeating words. Divide by the two lengths to remove the effect of size. This is the cosine similarity, the cosine of the angle between the vectors.

For document 1 the length is the square root of 5, and for the query the square root of 2. So 3 divided by 3.162 is 0.949.

docker exec jupyter python3 -c "import math; print(round(3 / (math.sqrt(2) * math.sqrt(5)), 3))"

docker exec jupyter python3 -c "import math; q=[1,0,1,0,0]; ds=[[2,0,1,0,0],[0,1,2,0,0],[1,0,0,1,0],[0,0,0,0,1]]; print([round(sum(a*b for a,b in zip(q,d))/(math.sqrt(sum(a*a for a in q))*math.sqrt(sum(b*b for b in d))),3) for d in ds])"

import math

def cosine(a, b):
    top = 0
    size_a = 0
    size_b = 0
    for n in range(len(a)):
        top = top + a[n] * b[n]
        size_a = size_a + a[n] * a[n]
        size_b = size_b + b[n] * b[n]
    return top / (math.sqrt(size_a) * math.sqrt(size_b))

query = [1, 0, 1, 0, 0]
documents = [[2, 0, 1, 0, 0], [0, 1, 2, 0, 0], [1, 0, 0, 1, 0]]
for n in range(len(documents)):
    print(n + 1, round(cosine(query, documents[n]), 3))
# Add the vector [4, 0, 2, 0, 0], document 1 written twice. Does its cosine change?

Replace the counts with tf-idf weights and the same method ranks by how special the shared words are. Compare a new support ticket with old ones to find the closest answer.

It does not know what words mean. "car" and "automobile" share no place in the vectors, so they are not similar to it.

Write three short texts about your own week and a query, and put them in a table like the one above. Rank them by hand, then check with the function.

Then find a query and a document that mean the same thing but score 0, and say why.