Insights/Measurement

Proof, Not Attribution: How to Know What Marketing Actually Caused.

Attribution reports what marketing touched. It cannot say what marketing changed. Three kinds of test close that gap.

Bespoke Research6 min readNovember 20263 exhibits
Cover exhibit

Only the gap between actual revenue and the modelled baseline is counted.

Exhibit

Every platform is happy to report how much revenue it drove. Add the claims together and the total is often larger than the revenue the business actually earned. Each platform counts every sale it touched, and most sales are touched more than once.

Attribution answers the question "which ads were near this sale". The question a board cares about is different: "what would revenue have been without this spend". That is a question about cause, and it needs a different kind of evidence.

Test one: hold something back

The most direct evidence is an experiment. Run a campaign in some regions and not in comparable others, then compare. The gap between the two groups is the lift the campaign caused. Just as important is the interval around it. A lift estimated from a small test can be large and still be indistinguishable from zero.

Exhibit 01Read the interval, not only the estimate
Exhibit

Single regions are noisy. Pooling them gives an estimate narrow enough to act on, which is why a test should be designed across several regions from the start.

A result that could be zero is not a result. It is a reason to run a better test.

Test two: model what would have happened

Not everything can be held back. When an experiment is impractical, the alternative is a baseline: a model of what revenue would have been, built from history, seasonality and markets that were not affected. Actual revenue is then measured against it, and only the gap is credited.

A baseline is an estimate and should be shown with its uncertainty. If the gap sits inside the band, the honest conclusion is that the effect is not yet proven.

Test three: decide with probabilities

Day-to-day decisions, such as which creative or landing page to keep, do not need a regional experiment. They need a clear rule. Reading a test as "the probability that B beats A" gives one: ship when the probability is high enough, stop when it is clear there is no difference, and keep testing in between.

Exhibit 02A test ends when the evidence is strong enough
Exhibit

Key takeaways

  1. Treat platform-reported revenue as a claim, not a measurement.
  2. Use holdout tests for the largest budget lines, and read the interval.
  3. Where a test is impractical, measure against a modelled baseline and show its uncertainty.
  4. Set the decision rule for every test before it starts.

How to apply it

  1. List the three largest budget lines and ask what evidence exists that each one causes revenue.
  2. Design one holdout test for the line with the weakest evidence.
  3. Report incremental revenue next to attributed revenue in every review.

Notes

Charts in this article are modelled examples and do not represent any client.