Reporting a result that remains uncertain

Report an uncertain incrementality result with its estimate, uncertainty interval, study population and decision limit. An inconclusive result does not establish no effect; a positive point estimate alone does not establish a gain.

Key Elements of an Uncertain Incrementality Result

Show the estimate and interval together

State the primary outcome, treatment and comparison, estimated difference, unit, period and interval in the same sentence or chart. Give group sizes, the interval’s stated confidence level or other interpretation, and the analysis method nearby. Show the absolute change alongside any percentage and its comparison base.

A hypothetical statement might read: ‘Among eligible shoppers in this retailer’s study, the estimated difference was +4 units per 1,000 shoppers, with a reported interval from −2 to +10 units per 1,000 shoppers.’

The interval spans a loss and a gain. Without a stated interval method and level, the example should not be treated as a real study result or as a precise probability statement. It does not establish that the effect was exactly zero.

Explain the limit

Separate imprecision from design and data problems. Few eligible shoppers or purchases may produce a wide interval even when the test ran as planned. Stock gaps, control exposure, product changes or incomplete sales linkage can add other limits. A narrow interval cannot correct bias in a poor comparison.

Show what was planned and what happened: assignment, delivery, sales coverage, exclusions and analysis changes. Label outcomes selected after seeing results as exploratory. Keep observed sales and retailer-attributed sales separate from the causal estimate.

Limitations of a Narrow Interval

Make a decision under uncertainty

Define the range of effects that would make the campaign commercially worthwhile using the advertiser’s own economics. If the interval includes both worthwhile and unattractive outcomes, the study has not settled that spending decision. It may indicate a useful next test, provided a longer period, larger eligible pool, cleaner withholding or steadier product availability is feasible. None guarantees a precise answer.

If the result cannot justify expansion, state the spending choice as a decision made under uncertainty. If the comparison failed, do not use its favourable-looking interval as causal evidence. Carry the estimate, interval method, extract date and material caveats into any presentation so a short headline does not overstate the study.

Decision-Making Under Uncertainty