Explaining a model's predictions

Explaining a prediction means saying which inputs pushed the answer and by how much. A person whose loan is refused will ask why, and a team needs an answer it can check.

In this lesson you explain one decision from a simple loan score by changing one input at a time. You also see what such an explanation cannot tell you.

A simple model turns inputs into a score. Each input is multiplied by its weight, and the results are added. A bank approves when the score reaches the threshold.

Here the score is 2 for each income band, plus 1 for each savings band, minus 5 for each missed payment. The Run scores one applicant four times: as they are, then with income one band higher, then savings, then one more missed payment.

docker exec jupyter python3 -c 's=lambda i,sv,m:2*i+sv-5*m;print(s(5,3,1),s(6,3,1),s(5,4,1),s(5,3,2))'

To see how much an input matters for one person, raise it by one step and keep every other input the same. Then compare the new score with the old.

The input that moves the score most matters most for this person. In the code below, one missed payment moves it by 5, far more than a band of savings, which moves it by 1.

weights = {"income": 2, "savings": 1, "missed": -5}
person = {"income": 5, "savings": 3, "missed": 1}

def score(w, p):
    total = 0
    for name in w:
        total = total + w[name] * p[name]
    return total

base = score(weights, person)
for name in weights:
    changed = {"income": person["income"], "savings": person["savings"], "missed": person["missed"]}
    changed[name] = changed[name] + 1
    print(name, score(weights, changed) - base)
# Change a weight, then run again.

A counterfactual answers "what would have to change for a different outcome?" An applicant with two missed payments scores 3, below the threshold of 8.

One fewer missed payment gives 8. Or three more income bands would work. That tells the person what to do, and it tells the team which inputs carry the decision.

docker exec jupyter python3 -c 'w=[2,1,-5];print(sum(a*b for a,b in zip(w,[5,3,2])),sum(a*b for a,b in zip(w,[5,3,1])))'

Changing one input at a time is a probe, not proof. In a bigger model, inputs combine, so the effect of one depends on the rest.

Some inputs also move together, such as income and savings. Raising one alone describes a person who may not exist. And an explanation shows what the model used, not whether using it was fair.

Take the weights in the code block and add a fourth input, age, with a weight of 0 and a value of 30. Run it again and write what you see.

Then say what you would worry about if its weight became 1. Name one way an age could stand in for something you should not use.