A Beginner's Guide to A/B Testing for Data Scientists
A practical guide to A/B testing for data scientists: framing testable hypotheses, sizing samples, reading...
Every January, someone in retail analytics gets the same two questions: why did we sell out of gift boxes on 18 December, and why is the warehouse still stacked with advent calendars? The answer is rarely one mistake. It is usually a forecast that treated December like any other month, built on history that was never cleaned.
This walkthrough uses a realistic case: a mid-sized retailer with eighteen months of daily sales across stores and product categories, a gift-heavy Q4, and a promotion calendar that changes every year. The tools are plain Python — pandas and statsmodels, plus a little plotting. The goal is a holiday demand forecast that holds up when the stakes are highest.
Assume one row per store, category and date: units sold, revenue, average price, a promotion flag and an availability flag. That is the shape most point-of-sale exports take. Eighteen months is workable but tight — one full annual cycle plus a partial second. Enough to estimate weekly seasonality and a rough holiday lift, not enough to be casual about it.
Three problems show up immediately. Dates arrive as strings, or worse, as week-ending labels. Some series have gaps where a store was closed or an export failed. And zero-sales days are ambiguous: no demand, or no stock? Those are different facts, and they lead to very different forecasts.
Parse dates into a proper datetime column, sort, and set a daily DatetimeIndex per series. Reindex each series to a complete daily calendar so missing days appear as visible gaps instead of disappearing quietly. Then work through the list:
Aggregate to a level with enough history to learn from. Category per store is usually more forecastable than SKU per store; split the category forecast down to individual products later using recent share.
Before fitting anything, look at the series. statsmodels gives you seasonal_decompose for a quick additive or multiplicative split, and STL when the seasonal shape is allowed to change over time. If December's lift scales with the overall level of sales, use a multiplicative decomposition or work in log units. If the lift is roughly flat regardless of scale, additive is fine.
The three components tell you different things. Trend captures slow shifts in demand. Seasonal captures the weekly rhythm and the annual holiday shape. The residual is where promotions, weather and everything unexplained lives. If residuals spike whenever a promotion runs, the model is missing a regressor, not mis-specified seasonality.
A forecast is only as honest as the history it learned from. If December was a stockout month, the model learned the wrong lesson.
Forecast at the level you can act on. A category-store forecast that is 8 percent off is more useful than a SKU-store forecast that is 20 percent off and impossible to reconcile.
A single accuracy number for the whole year hides the failures that cost money. Evaluate on holiday weeks separately: late November through early January. Use rolling-origin backtesting, training up to a cut-off and predicting forward, then moving the cut-off and repeating.
Prefer WAPE or MASE over MAPE. Percentage errors explode on low-volume days and quietly reward models that under-forecast the peak. And look at bias, not just average error: a model that consistently under-forecasts December by 12 percent is a bigger problem than one with noisier but unbiased errors.
Break error down by week and by category. If the model is fine in October and falls apart in the final ten days of December, the cause is usually gift-basket mix or last-minute buying behaviour, not a smoothing parameter. Prediction intervals matter here too — order to the upper bound when lead times are long and stockouts are expensive.
Pick one category with two clean years of history and an obvious holiday peak. Reindex the calendar, flag the stockouts, decompose, and build the seasonal naive baseline before anything more sophisticated. Then backtest on holiday weeks only, and write the bias number somewhere the whole team can see it.
Do that once and the rest is repetition: add exogenous flags, extend to more categories, automate the weekly refit, and compare forecast against actual every Monday rather than once a quarter. The forecast will never be perfect. It only has to be better than the spreadsheet — and honest about its own uncertainty.
Photo: Max Fischer / Pexels
A practical guide to A/B testing for data scientists: framing testable hypotheses, sizing samples, reading...
A practical guide to pulling your city's open bike data, cleaning trip records without bias, and mapping...
A practical guide to structuring data science projects for reproducibility: folder layout, locked...