You train a model, the metrics look excellent, and three weeks later it is quietly failing in production on exactly the cases you care about. The model is rarely the culprit. The evaluation was. Metrics are not a formality you run after the interesting work is done; they decide whether the rest of the work means anything at all. Here are the mistakes that come up most often in code reviews, and how to fix each one.

Data leakage: the score that never had a fair chance

Leakage means information from outside the training fold slipped into the model, so your offline score describes a situation that cannot exist in production. The common version is procedural: you scale features, impute missing values, or encode categories on the full dataset and only then split into train and test. The test rows shaped the means and encodings the model learned from. Scores go up, honesty goes down.

The subtler version hides in the features. A churn model with a variable like "number of cancellation emails sent" looks brilliant offline because those emails exist only after the customer has already decided to leave. Duplicate records do the same damage: the same user, order, or session appearing in both splits lets the model memorise rather than generalise.

• Split first, then fit every transformer inside a pipeline so it only ever sees training folds.
• For each feature, ask: would this value be available at prediction time, before the outcome happened? If not, drop it.
• Deduplicate by entity — user, account, store, patient — not by row index.
• For anything time-ordered, split by date and never shuffle.

When a score looks too good to be true, assume leakage until you can explain, step by step, where every input value came from and when.

Accuracy as the headline on imbalanced data

Suppose 0.4% of transactions are fraudulent. A model that predicts "not fraud" for every single row scores 99.6% accuracy and catches nothing. Accuracy is a weighted average of performance across classes, and with skewed classes the weight sits almost entirely on the majority. It rewards the laziest possible model.

Replace it with measures that survive imbalance. Look at the confusion matrix at the threshold you actually intend to deploy, not the default 0.5. Report precision and recall for the class you care about, or F1 if you need one number. For ranking problems, precision-recall AUC is far more informative than ROC AUC when positives are rare.

Pick the threshold with costs in mind

A false negative in fraud review may cost a few hundred euros; a false positive may cost ten minutes of an analyst's time. Those are not equal, so a threshold that treats them as equal is the wrong threshold. Estimate the cost of each error type, then choose the operating point that minimises total expected cost. Report the metrics at that point. It is also worth checking the score's variance across folds rather than quoting a single lucky number — a model at 0.78 ± 0.06 is a different proposition from one at 0.78 ± 0.01.

Splitting grouped or time-ordered data the wrong way

Random splits assume rows are independent. Often they are not. If your dataset has repeated measurements per patient, multiple orders per customer, or several stores in the same region, a random split puts near-identical rows on both sides and the model gets credit for recognising a neighbour. Use grouped cross-validation — one entity belongs to exactly one fold — and hold out entire groups for the final test.

Time series have the same problem in a different costume. Shuffling timestamps lets the model train on December and be tested on March, which never happens in the real world. Use forward-chaining splits: train on the past, predict the next period, roll forward. If your data has both structure and time, respect both.

Overfitting the test set

A test set is a resource you spend once. Try twenty model configurations, keep the one with the best test score, and that score stops being an estimate of future performance — it becomes the maximum of twenty noisy numbers, and maxima are always optimistic. Tune hyperparameters and compare architectures on a validation split or via cross-validation on the training data, then touch the test set once, at the end, when the decision is already made.

If you have looked at the test set more than twice, it has quietly become a validation set. Say so in your report, and carve out a fresh holdout you never touch.

Seeds matter here too. Retrain your final model with a handful of random seeds and look at the spread. If the metric swings by more than the difference between your top two candidates, you have not found a winner — you have found noise.

No baseline, and no link to the decision

An F1 of 0.71 tells a reader nothing on its own. Better than what? Always establish a baseline first: the majority class, a simple logistic regression on obvious features, last quarter's rules, or the current manual process. If the fancy model barely beats the baseline, that is a finding worth reporting, not a failure to hide.

Then connect the metric to the decision it supports. Who acts on these predictions, and what changes when the model is right or wrong? A model that improves ranking quality but never changes a decision is an expensive dashboard. Document the evaluation protocol before you train: the split strategy, the metric, the threshold, the baseline. Deciding this in advance is the cheapest protection against fooling yourself later.

A short checklist before you report a number

1. Fit transformers only on training data, inside a pipeline.
2. Check for duplicate entities across splits and remove them.
3. Match the split strategy to the data: grouped, temporal, or random.
4. Report precision, recall, and the confusion matrix at your chosen threshold — not just accuracy.
5. Compare against a trivial baseline and the current system.
6. Tune on validation folds; touch the test set once.
7. State the metric's variability across folds or seeds.

None of this is glamorous. It is, however, the difference between a model that impresses in a notebook and one that holds up when it meets real data on a Monday morning. Run the checklist, write down your assumptions, and your future self will thank you.

Photo: Pexels / Pixabay