Incrementality Testing
Retail media incrementality tests
Plan a retail media incrementality test with a defined outcome, credible comparison, delivery checks and a bounded reading of the result.
A retail media incrementality test estimates what changed because advertising ran, against what would have happened without it. Decide the outcome and comparison before launch, then check whether the planned groups remained comparable. Report uncertainty with the estimated effect. Attributed sales alone cannot supply the missing comparison.
Start with the decision
Specify whether the result will inform a repeat booking, an expansion or a change in spend. Name the advertised products, retailer, media activity, dates and eligible population. Choose a primary outcome, such as units of the promoted range or brand sales at that retailer. Promoted-item sales answer a narrower question than total brand sales.
Set the analysis period, and any period allowed for later purchases, in advance. Record which transactions the retailer can observe, how returns are treated, and whether sales elsewhere are visible. A result within one retailer should not be described as growth across the UK market.
Choose a credible comparison
| Approach | What it can assess | Condition to check |
|---|---|---|
| Randomly assigned shopper or household holdout | Outcomes among eligible people assigned an opportunity to receive ads versus those withheld | Comparable eligibility, effective withholding and outcome measurement for both groups |
| Randomly assigned or carefully matched locations | Outcomes in media locations versus suitable locations without that media | Pre-campaign patterns, availability, price, promotions and spillover |
| Modelled counterfactual | A possible untreated outcome when a clean holdout is unavailable | Input coverage, assumptions and checks for bias |
Random assignment reduces selection bias when it is feasible and properly implemented. Matching and modelling rely more heavily on recorded differences and assumptions. Comparing sales with the previous month shows change, but cannot by itself isolate the advert’s effect.
For a shopper holdout, agree whether the primary analysis compares everyone assigned to the ad opportunity. People who actually receive an impression can differ from those who do not, even within the assigned treatment group. For a location test, record eligibility and pre-campaign sales patterns before assigning media.
Check what happened during delivery
Keep a dated record of media delivery, product availability, prices, promotions and other relevant campaigns. Ask how the study identifies control-group exposure, failed treatment delivery and unlinkable sales. These issues can change the population or effect the estimate describes.
Retain the original outcome, product set and end date. If changes after launch are necessary, record when and why they occurred, and label analyses added after seeing results.
Use the result within its limits
Request the estimated group difference, underlying rates or totals, group sizes, analysis method, uncertainty interval and material exclusions. State whether the outcome is units, sales value or another measure, and whether it concerns assigned shoppers, reached shoppers or locations. Give the absolute difference alongside any percentage uplift and its comparison base.
An interval spanning both a loss and a gain does not support a confident increase claim; it does not show that the effect was exactly zero. A positive retailer-level result also leaves open possible shifts between the brand’s products and sales at other retailers.
Base the next decision on the study’s design, uncertainty and the advertiser’s economics. If delivery or measurement weakened the comparison, narrow the claim or plan a better test. If no credible comparison survived, report observed and attributed sales under their own definitions, and leave incrementality unresolved.
In this guide
- Choosing a control group before a campaign beginsChoose an eligible retail media control before launch, with documented assignment, comparable outcomes and checks for exposure spillover.
- Measuring whether ads shift sales between a brand's productsCompare advertised-product lift with other brand products and the brand total to assess possible switching in a retail media test.
- Reviewing retailer uplift claims against the stated methodologyCheck a retailer’s uplift headline against its outcome, comparison, sales population, uncertainty and disclosed study method.
- Reporting a result that remains uncertainPresent an inconclusive retail media test with its estimate, interval, study limits and a decision that fits the evidence.
