A question with a number in the answer
AI engineer interview questions often ask how you checked, ran or fixed a model-backed feature. "Tell me about a time a model gave a wrong answer." A question like that is common.
A strong answer is a short story with a number in it. This lesson shows the usual kinds of question and how to answer each with evidence.
By the end of this lesson
Note
I can answer an AI engineering interview question with evidence.
Key words
| Word | What it means | Example |
|---|---|---|
| Evidence | a number or result that backs a claim | 18 of 20 right |
| Concept question | asks you to explain an idea | |
| Design question | asks how you would build something | |
| Experience question | asks what you did before | |
| Model answer | an example answer to compare yours with |
The cells students fill in are left blank in this preview.
Questions come in three kinds
Most questions fall into three kinds. A concept question asks you to explain an idea, such as retrieval or a token.
A design question asks how you would build something, such as a safe agent. An experience question asks what you did. Formats differ between employers, so treat these as practice, not a script.
Common questions, by kind
| Kind | Example question | What they listen for |
|---|---|---|
| Concept | How does an app give a model the right facts? | Retrieval, in plain words |
| Concept | Why is the average latency not enough? | Slow requests and the 95th percentile |
| Design | How would you stop an agent doing harm? | Allowlist, budget, human check, in code |
| Design | How would you know a feature is ready? | A spec, an eval set, acceptance numbers |
| Experience | Tell me about a model giving a wrong answer. | What you saw, what you did, the result |
The cells students fill in are left blank in this preview.
Answer with what happened and a number
A good experience answer has three parts. What the situation was. What you did, and why. What the numbers showed afterwards.
Keep it under two minutes. Use one real example from your capstones, not three vague ones. The program checks two answers for a number, the simplest sign of evidence.
Try it: which answer has evidence?
# Two answers to: "Tell me about a feature you checked."
answers = [
"I built a chatbot and it worked really well. Everyone liked it.",
"I checked a document assistant with 20 eval questions. "
+ "It scored 16, so I fixed retrieval and it reached 19.",
]
for answer in answers:
has_number = False
for ch in answer:
if ch.isdigit():
has_number = True
if has_number:
print("Has evidence:", answer[:40])
else:
print("No number: ", answer[:40])
# Rewrite the first answer with a number in it, then press Run.
Practise with model answers
Read a question, answer it aloud, then compare with the model answer. Do not learn the model answer by heart. Notice what it has that yours lacks: a reason, a number, a limit.
If you do not know something, say so, and say how you would find out. That is an honest answer, and it shows judgement.
A practice set with model answers
| Question | Model answer, in short |
|---|---|
| How do you check a document assistant? | A fixed eval set of 20 cases, some with no answer; release at 18 right and no guesses. |
| What limits would you give an agent? | An allowlist and a step budget in code, and a person approves risky actions. |
| A feature feels slow. What do you look at? | The 95th percentile, then the slowest requests in the logs. |
| What would you change in your project next? | One known limit, why it matters, and how I would measure the fix. |
| What if you do not know the answer? | Say so, then say how I would find out and check it. |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think the best answer names as many tools as possible. A list of names says nothing about judgement. Name one choice, say why you made it, and say what the numbers showed.