A Beginner's Guide to A/B Testing for Data Scientists
A practical guide to A/B testing for data scientists: framing testable hypotheses, sizing samples, reading...
The model is finished. The meeting is tomorrow. In the room: a marketing director, a finance analyst, and someone from compliance who has read the word "opaque" in a report somewhere. What they need from you is not a tutorial on gradients. They need to know what this thing will do to their week — how many calls it generates, which customers it flags, and what happens when it is wrong.
Explaining a model is a translation job. It has a vocabulary, a structure and a few predictable traps, and it gets much easier once you treat it as a craft rather than a personality trait.
Before you open a notebook, finish this sentence in one line: this model changes the decision of who to contact about X, every Monday morning. If you cannot finish it, you are not ready to present. A model that does not change anyone's behaviour is a hobby.
Then name three things your audience actually cares about:
Those three answers determine the entire rest of your explanation. A model feeding a human review queue can be described very differently from one that auto-declines applications.
Feature importance answers one question: which columns did the model lean on most, averaged across everyone it scored? It does not tell you direction, and it does not tell you cause. A model can lean heavily on a field that is merely a proxy for something else.
Be precise about which kind you are showing. Impurity-based importance from a tree model is cheap but biased toward variables with many distinct values; permutation importance, measured on held-out data, is usually the more honest number to put on a slide. Whichever you use, say so in one sentence and move on.
Useful translation swaps:
Pick two numbers, not eight. A stakeholder who remembers two numbers will use them; a stakeholder who remembers none will ask you to send the deck again.
SHAP is the hardest idea to convey and the easiest to demonstrate. Skip the theory and show one prediction as a receipt.
Start from what the model says about an average customer. Add the small amounts contributed by each fact about this one person. The total is the score you are looking at.
That framing does most of the work. A customer's score of 0.71 is not a mystery; it is a baseline of 0.30, plus 0.15 because they have not ordered in five months, plus 0.12 because they contacted support twice, minus 0.06 because they are on an annual plan, and a handful of small adjustments.
Talking through a single, recognisable case beats any chart. If your audience knows the account, use it — anonymised if needed. The moment someone says "oh, that's the Wexford account," the method stops feeling like a black box.
Two caveats worth stating out loud. SHAP explains the model, not the world: a feature can look important because it correlates with the real driver. And when two features carry the same information, SHAP splits the credit between them, so neither looks as strong as it is in isolation.
Risk conversations go badly when they stay abstract. Anchor them to three concrete questions.
Be candid about what the model cannot see: seasonality that has not happened yet, a competitor's price change, a policy shift. If the model touches credit, hiring, insurance or health decisions, bring legal and compliance in early — the expectations there are stricter than anything covered here, and they are worth professional advice rather than guesswork.
Five beats, in this order, work for almost any audience:
Rehearse this in five minutes with someone outside your team. If they cannot repeat your recommendation to a colleague afterwards, the explanation failed — not the model.
Every audience asks a version of the same four things. Prepare a one-line answer for each.
Run this before every stakeholder conversation. It takes less time than finding the right chart.
Explaining models well is mostly about restraint. You know far more than the room needs, and the temptation is to prove it. Resist that. Give them the decision, two numbers, three drivers and an honest account of the weak spots — then let them ask. The questions they ask are the clearest signal you have that the translation worked.
Photo: ClickerHappy / Pixabay
A practical guide to A/B testing for data scientists: framing testable hypotheses, sizing samples, reading...
A practical guide to pulling your city's open bike data, cleaning trip records without bias, and mapping...
A practical walkthrough of cleaning daily sales history, separating trend from holiday seasonality, and...