Model cards and monitoring a model

A model card is a short document about one model. It says what the model is for, what data it learned from, how it performs for each group and where it must not be used.

A model keeps running long after the people who built it have moved on. In this lesson you write down a model and set a check that tells you when it has drifted.

A model card is written so that a stranger can decide whether to use the model. It names the intended use, the data it learned from, and how it performs for each group.

It also says where the model must not be used and who answers for it. The Run below prints a short invented card for a model that ranks loan requests for a person to read.

docker exec jupyter python3 -c 'card={"purpose":"rank loan requests for a person to review","not for":"a final decision with no person"};print("\n".join(k+": "+v for k,v in card.items()))'

A model learns from data from the past. After launch, the people who apply can change with a new product or a new season.

Drift is that change. A model that was tested once is only tested on the past, so monitoring compares what it sees now with what it learned from. A large gap is a signal to ask the owner to look.

docker exec jupyter python3 -c 'old=[40,42,38,44,36];new=[52,50,46,54,48];print(sum(old)/len(old),sum(new)/len(new))'

old = [40, 42, 38, 44, 36]
new = [52, 50, 46, 54, 48]
old_mean = sum(old) / len(old)
new_mean = sum(new) / len(new)
change = (new_mean - old_mean) / old_mean
print("Change:", change)
if change > 0.2:
    print("Drift: ask the owner to look")
# Change the new values so the change falls below 0.2, then run again.

Drift in an input is one signal. The results by group are another. A model can keep its overall accuracy while the gap between two groups grows.

The Run below prints the gap each month between the number selected per 100 people in group A and in group B. It rises from 30 to 37.

docker exec jupyter python3 -c 'a=[50,50,48];b=[20,19,11];print([x-y for x,y in zip(a,b)])'

Monitoring only helps if a check leads to a person who can act. Write down who is told when a check fires and who can switch the model off. Also write down how long a person waits to have a decision looked at.

Add a review date to the card. A card with no owner and no date tells the reader nobody is looking.

Write a model card for the line you fitted in the data science module. Use the six fields from the worked example.

Then pick two inputs you would monitor, say what change would make you look, and name who you would tell.