Incrementality Testing
Part of Retail media investment reviews
Identifying spend that produced no useful learning
Audit campaign questions, delivery and outcomes to find spend that could not inform a decision.
A poor sales result can still teach an advertiser what to change. Spend produced no decision-grade learning when the available record cannot answer the intended question or a defensible narrower one. Audit the question, execution and evidence before reaching that conclusion.
Recover the decision behind the spend
For each material charged line, identify the approved product, audience or placement, dates, intended outcome and decision it was meant to inform. Ask what observation would have changed the plan. Some lines may have served a contractual or brand purpose rather than a learning purpose; judge them against that purpose.
Reconcile charged spend with credits and undelivered bookings. An approved budget or booked amount is not necessarily the amount spent.
Locate the break in the evidence
| Possible gap | Record to inspect | What may still be concluded |
|---|---|---|
| Delivery cannot be verified | Booking, placement, dates, charged units and exceptions | The intended activity cannot be confirmed. |
| Product was unavailable | Dated stock, ranging and destination records | Delivery may be known, but sales are hard to interpret. |
| Outcome definition is absent | Product set, channels, sales basis and attribution rule | The sales figure may not answer the agreed question. |
| Several conditions changed together | Dated creative, price, targeting and stock changes | A narrower period may be usable, but one cause cannot be isolated. |
| A causal claim lacks a comparison | Study method, comparison and uncertainty | Attributed sales cannot establish additional sales. |
These are checks, not alleged defects in a retailer’s service. IAB Europe’s online standards distinguish media delivery, attribution and incrementality and define different sales and attribution measures. IAB and IAB Europe’s incrementality guidance explains why the counterfactual and method limitations matter to a causal claim. Neither source can fill a missing campaign record.
Key Evidence Gaps and Their Impact on Decision-Making
- Delivery cannot be verified
- Intended activity cannot be confirmed.
- Product was unavailable
- Sales are hard to interpret despite delivery knowledge.
- Outcome definition absent
- Sales figure may not answer the agreed question.
- Multiple conditions changed together
- One cause cannot be isolated; narrower period may be usable.
- Causal claim lacks comparison
- Attributed sales cannot establish additional sales.
Classify the learning
Usable learning answers the defined question within stated limits, even when the answer is unfavourable. Partial learning answers a narrower question, such as whether media ran when the sales definition is missing. No decision-grade learning means a material gap prevents both the intended decision and a defensible narrower conclusion.
The cause of that gap may be known; the label describes what the spend can establish, not whether anyone can explain the failure.
Do not turn a measured zero into missing data, or a missing field into zero. Preserve the original retailer extract and any dated correction. A reliable change log may allow the review to isolate a narrower period; without one, state the limit.
Types of Learning from Retail Media Spend
- Usable learningAnswers the defined question within stated limits, even if the result is unfavourable.
- Partial learningAnswers a narrower question, such as whether media ran when sales definition is missing.
- No decision-grade learningMaterial gap prevents both intended decision and defensible narrower conclusion.
Prevent the same gap next time
For a line with no decision-grade learning, record the amount charged, the unanswered question, the cause where known, and one control with an owner. That might be an agreed report field list, an availability check or a dated change log.
The control is a proposal for the next booking, not a tested remedy. Keep the unresolved question beside any recommendation so a later budget decision does not rely on evidence the campaign never produced.
