Working out a model’s accuracy

A bank’s new fraud model is "99% accurate", says the report. The team is pleased.

Then someone looks closer. Only 1 payment in 100 is fraud. A model that says "not fraud" every time would also be 99% accurate, and it would catch nothing.

To trust a number like that, you need to know how it was worked out.

Put each prediction beside the real answer from the test set. Count how many match.

Accuracy is that count out of the total, as a percentage. Divide the right answers by the total, then multiply by 100.

8 right out of 10 is 8 divided by 10, which is 0.8. Times 100 gives 80%.

# A fraud model checked 10 card payments from the test set.
# True means fraud.
predicted = [False, True, False, False, True, False, False, False, True, False]
actual    = [False, True, False, True, False, False, False, False, True, False]

correct = 0
for i in range(len(actual)):
    if predicted[i] == actual[i]:
        correct = correct + 1

accuracy = correct * 100 / len(actual)
print("Right:", correct, "of", len(actual))
print("Accuracy:", accuracy, "%")

# Change the fourth value in predicted to True, then press Run.
# What happens to the accuracy?

A wrong answer is one of two kinds.

A false alarm says fraud when there was none. A customer’s card is blocked for no reason, and they ring the bank.

A miss says no fraud when there was some. The bank loses the money.

Accuracy counts both the same. So always look at the misses and false alarms as well.