Evaluating a Model

A single accuracy number can make a genuinely useless model look impressive — how you evaluate a model matters just as much as how you build it.

When "99% accurate" means almost nothing

Imagine a model built to catch fraudulent transactions, where only 1% of all transactions are actually fraudulent. A model that does zero real work and simply predicts "not fraud" every single time scores 99% accuracy — and catches exactly zero fraud.

>>> a lazy "model" on imbalanced data

transactions = 10000
actually_fraud = 100  # 1% of transactions

# A "model" that always predicts "not fraud"
correct_predictions = transactions - actually_fraud
accuracy = correct_predictions / transactions

print(f"Accuracy: {accuracy:.1%}")
print(f"Fraud cases caught: 0 of {actually_fraud}")

Output

Accuracy: 99.0%
Fraud cases caught: 0 of 100

99% accuracy sounds great in a slide deck and is completely worthless here — the model exists specifically to catch fraud, and it catches none. This is what "imbalanced data" (where one outcome is much rarer than the other) does to plain accuracy as a metric: it stops measuring what you actually care about.

Precision and recall

Precision answers: of everything the model flagged as fraud, how much of it actually was fraud? Recall answers: of all the fraud that actually happened, how much of it did the model catch? A model can score high on one while scoring badly on the other — flagging almost every transaction as suspicious gets you near-perfect recall (you'll catch nearly all the real fraud) but terrible precision (you'll also flag huge numbers of legitimate transactions).

The tradeoff is usually deliberate

Which one matters more depends entirely on what a mistake costs. Missing a fraud case might mean a real financial loss, so recall matters a lot; wrongly flagging a legitimate transaction annoys a customer but costs less, so a bit of lost precision is often an acceptable trade for higher recall in fraud detection specifically. A cancer-screening test typically leans the same way — for a different reason with the same shape: missing a real case is far worse than a false alarm that gets ruled out by a follow-up test.

Always ask what a mistake costs: before picking which metric to optimize for, ask what happens when the model is wrong in each direction. There's rarely a single "best" model in the abstract — there's a model that fits the actual cost of the two kinds of mistakes it can make for this particular problem.

PreviousIntro to Machine Learning Concepts NextCommunicating Results