Capstone: report on a live feature

A week later, the questions start

Running an AI feature means knowing its cost, its speed and its quality every week. Those three numbers decide whether the feature is worth keeping.

A summary feature went live last Monday. Finance asks what it costs. Support asks why some answers are slow. Impressions will not do. In this capstone you work out the numbers from a recorded week and write them on one page.

By the end of this lesson

Note

I can report the cost, speed and quality of a live AI feature.

Key words

Word	What it means	Example
Token	a small piece of text a model reads or writes
Latency	how long one request takes
Median	the middle value once sorted
Percentile	the value a given share of requests stay under	95th percentile
Quality score	the share of sampled answers rated good

The cells students fill in are left blank in this preview.

Where to go deeper

Note

This capstone reads numbers and reports them. The AI lessons on the Resources page teach how to collect them. Cost, tracing and reliability. Evaluating AI systems in depth. Operating AI features.

Cost comes from tokens

A model is usually paid for by the token, in and out. The text you send is tokens in. The answer is tokens out, and those often cost more.

So cost depends on how long your prompts are, not only on how many requests come in. The program works out each day's cost from a recorded week, with made-up prices.

Try it: the cost of a week

Code: python

# A recorded week of a summary feature.
# Tokens are counted in thousands. Prices are made up for practice.
PENCE_PER_THOUSAND_IN = 1
PENCE_PER_THOUSAND_OUT = 4

days = [
    {"day": "Mon", "requests": 400, "in": 800, "out": 120},
    {"day": "Tue", "requests": 420, "in": 840, "out": 130},
    {"day": "Wed", "requests": 380, "in": 760, "out": 110},
    {"day": "Thu", "requests": 900, "in": 2700, "out": 300},
    {"day": "Fri", "requests": 410, "in": 820, "out": 125},
]

total_pence = 0
total_requests = 0
for day in days:
    pence = day["in"] * PENCE_PER_THOUSAND_IN + day["out"] * PENCE_PER_THOUSAND_OUT
    print(day["day"], pence, "pence")
    total_pence = total_pence + pence
    total_requests = total_requests + day["requests"]

print("Week:", total_pence, "pence for", total_requests, "requests")
# Which day stands out? Change Thursday to look like Monday and run again.

An average hides slow requests

Latency is how long one request takes. The average sounds useful, but two very slow requests can hide in it.

So report the median, the middle value, and the 95th percentile. Sort the times. The 95th percentile is the time that 95 out of 100 requests stay under. It is what your slowest users meet.

Try it: median, average and 95th

Code: python

# A recorded sample: 20 response times in seconds.
times = [1.2, 1.1, 1.4, 1.3, 1.2, 1.5, 1.1, 1.3, 1.2, 9.8,
         1.4, 1.2, 1.3, 1.1, 1.6, 1.2, 1.3, 8.9, 1.4, 1.2]

ordered = sorted(times)
middle = ordered[len(ordered) // 2]
print("Median:", middle, "seconds")

total = 0
for t in times:
    total = total + t
print("Average:", round(total / len(times), 2), "seconds")

# 95 percent of 20 is 19, so the 19th time in order.
print("95th percentile:", ordered[18], "seconds")

Quality, then one page

Cost and speed mean little if the answers are poor. Each week, a person rates a fixed sample, say 50 answers. The share rated good is the quality score.

Then all three go on one page: the number, the target, and what you will do. A manager should be able to decide from that page alone.

Worked example: the one-page report

Measure	This week	Target	Action
Cost	£90.60 for 2,510 requests	Under £80	Shorten the prompt sent on Thursday
Median latency	1.3 seconds	Under 2 seconds	None
95th percentile	8.9 seconds	Under 5 seconds	Find the slow requests in the logs
Quality score	44 of 50 good	45 of 50	Add the 6 failures to the eval set