What an AI engineer does

An AI engineer builds useful features on top of a language model, then proves they work. Without that proof, a clever feature can give people wrong answers.

Say a support team wants a feature that answers staff questions from the company handbook. Somebody has to build it, prove its answers are right and keep it running after launch. In many teams that person is called an AI engineer.

An AI engineer rarely trains a new model. They take a model that already exists and build a feature with it. That means a prompt, some code, and often a search over the company's own documents.

Then they check it with an eval set. Last, they run it: they watch cost, speed and quality with monitoring, and fix what drifts.

The job joins skills from several areas. You need enough Python to read and change code. You need to call a model from a server safely. You need to write test cases and read numbers.

Most of all, you need to explain a trade-off: what you gained, what it cost, and how you know. The table shows where each skill is taught.

# Rate yourself from 1 (new to it) to 5 (confident).
ratings = {
    "Python": 3,
    "Calling a model from code": 2,
    "Writing an eval set": 1,
    "Reading logs and numbers": 2,
    "Explaining a trade-off": 4,
}

TARGET = 3

for skill in ratings:
    if ratings[skill] < TARGET:
        print("Work on next:", skill)
    else:
        print("Ready:", skill)

# Change the ratings to match you, then press Run.

A model can give a different answer to the same question on another day. So a feature is never proved by one good demo.

An AI engineer keeps an eval set: questions with the answers they expect. After every change, they run the whole set and count the right answers. The program below does that over a recorded sample.

# A recorded sample: five questions, the answer we expect,
# and what a support feature answered in testing.
expected = ["14 days", "Friday", "Form B2", "No", "2 weeks"]
answered = ["14 days", "Monday", "Form B2", "No", "I am not sure"]

right = 0
for position in range(len(expected)):
    if expected[position] == answered[position]:
        right = right + 1
    else:
        print("Wrong on question", position + 1)

print("Right:", right, "out of", len(expected))

# Make the second answer "Friday", then press Run again.