Woman presenting business data on clipboard with success graph in an office.
Photograph by RDNE Stock project on Pexels

Incrementality Testing

Part of Retail media incrementality tests

Reporting a result that remains uncertain

Present an inconclusive retail media test with its estimate, interval, study limits and a decision that fits the evidence.

This is an editorial pen name used by the publication, not a claim of an independently verified personal identity. Original editorial credit: Measurement Desk.

undefined

Report an uncertain incrementality result with its estimate and interval, a defined study population, and a commercial decision limit set before interpreting results. An inconclusive result does not establish no effect; a positive point estimate alone does not establish a gain.

Key Elements of Reporting an Uncertain Incrementality Result

  • Interval TypeConfidence interval using bootstrapping
  • Study PopulationDefined eligible shoppers at [retailer and market], with pre-set eligibility rules and exclusions

Show the estimate and interval together

Give the primary outcome, treatment and comparison, estimated difference, unit, period and interval together. Define the population by retailer, market, pre-set eligibility rule, study dates and exclusions; report group sizes. Name the interval type and construction method used, for example a confidence interval calculated using bootstrapping, and state its confidence level or other interpretation. Show the absolute change alongside any percentage and its comparison base.

For example, a report could identify [retailer and market], [eligibility rule], [study dates] and [exclusions], then state: ‘Among [defined eligible shoppers], the estimated difference was +4 units per 1,000 shoppers, with an [interval type] from −2 to +10 units per 1,000 shoppers, calculated using [method] and reported with [confidence level or interpretation].’

The interval spans a loss and a gain; it does not establish that the effect was exactly zero. Fill in the interval type, method and level actually used, and do not treat the interval as a precise probability statement. The IAB and IAB Europe guidelines for incremental measurement in commerce media refer to bootstrapping and confidence intervals that exclude zero.

Interpreting the Interval: Gain vs. Loss vs. No Effect

Zero in Interval?
Yes – spans both loss and gain
Establishes No Effect?
No – inconclusive result does not prove zero impact

Explain the limit

Separate imprecision from design and data problems. Few eligible shoppers or purchases may produce a wide interval even when the test ran as planned. Stock gaps, control exposure, product changes or incomplete sales linkage can add other limits. A narrow interval cannot correct bias in a poor comparison.

Show what was planned and what happened: assignment, delivery, sales coverage, exclusions and analysis changes. Label outcomes selected after seeing results as exploratory. Keep observed sales and retailer-attributed sales separate from the causal estimate.

Pros and Cons of Using a Narrow Interval in Poorly Designed Tests

  • ProsMay suggest precision; useful when design and data quality are strong
  • ConsCannot correct bias from poor comparison; narrow interval may mislead if data issues exist

Make a decision under uncertainty

Before seeing results, convert the advertiser’s break-even requirement, using campaign cost and its own commercial economics, into a minimum incremental effect in the outcome’s unit: the decision limit. Compare the whole interval with that limit. If the interval includes effects both below and above it, the study has not settled the spending decision.

An inconclusive result may justify a further test if a longer period, larger eligible pool, cleaner withholding or steadier product availability is feasible; none guarantees precision. If it cannot justify expansion, state the spending choice under uncertainty. If the comparison failed, do not use its favourable-looking interval as causal evidence. Carry the estimate, interval method, extract date and material caveats into presentations to avoid an overstated headline.

Steps for Decision-Making Under Uncertainty

  1. Define Decision LimitSet minimum incremental effect required based on campaign cost and commercial economics
  2. Compare Interval to LimitCheck if interval includes values both above and below the decision limit
  3. Assess Feasibility of Further TestingEvaluate whether longer duration, larger pool, or cleaner withholding is possible
  4. State Spending ChoiceDeclare decision under uncertainty if no further test is feasible

Further reading: How to Read Experiment Results (Formerly "Pulse") (docs.statsig.com).

More from Incrementality Testing