Sampling Theory, Internet Ads and the rule of 25 conversions.

Facebook and Google now control > 50% of all Internet Ad spend, and virtually every company has tried some form of advertising on these two platforms. Typically, we proceed as follows:

  1. run a few hundred dollars to a thousand dollars of advertising using a single “best idea” ad
  2. get an overall “CPA” (Cost of Customer Acquisition)
  3. Repeat above steps for a few different creatives and target audiences.

The general methodology is correct, but the real question is how much money do you have to spend to get a “statistically significant” results?

I would suggest, as a good rule of thumb, you need 25 conversions.

Lets put this in context. If each conversion is an app install, and they cost, on average 4 bucks each, you need to spend $100 to start getting meaningful results. Not $1,000, not $10,000 (like many ad agencies would have you believe) and not $10 either.

With a little math, we can shed some light on this subject.

You can model the advertising process as a series of events where hundreds of millions of users are shown one of your ads. If you cap the number of times any user sees each ad, these are independent events, meaning that user x’s response to an ad is not related to user y’s response, and they are likely independently distributed.

In mathematical terms, the events are iid (independent and identically distributed). In other words, they are coin tosses, with unknown probability p of conversion.

If I do n coin tosses, what is the distribution of results? Well, we know from elementary probability theory that this is a binomial distribution, with parameters n and p. Now binomial distributions are messy, complex things to evaluate, but for large n, the central limit theorem of statistics says that binomial distributions are approximately normal with mean np, and variance n*p*(1-p).

Now in our case p is very small. So 1-p is essentially 1. Which means that the variance we are looking for n*p is essentially the same as our expected value n*p. This is a key result.

Now suppose we observe n*p = 25. Then we know that the distribution is 25 +/- 5 with one standard deviation. There is approximately 68% shot of being within one standard deviation (see https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule)

As you can see, 25 “successes” is thus statistically significant. You have real confidence that the number is in the 20–30 range. Doing 10 times more testing, will increase your confidence from 68% to 95%, but at a 10x cost in media — probably not worth it.

You can play around with lower numbers and see that anything much less than 25 results is really un-trustworthy. For example, lets suppose you got 9 results. Then all you could say is that you are between 6 and 12. Thats at 33% band either way — not very precise.

This doesn’t mean that you need to keep on spending money in all cases until you get 25 conversions. In fact we can also use statistics to get information on what very low conversions tell us about bad results. But that is a different question.

The main point here is to give some guidance on how many results you need to get a sense of how well an ad is working and the answer is 25.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store