3.11. Statistical Significance (Evan Miller)

In order to control something, you need good information channels. Not only that, but you also need to understand how much information you need to obtain to make a right decision. In business, you can’t make spontaneous choices or decide on the basis of your inner feelings. Business requires analytical and data-based thinking, determined and planned organization, and anticipation of risks and further steps.

Who’s Evan Miller?

Let’s suppose that we decided to implement a change to increase a specific metric and already came up with an idea that we turned into a hypothesis. Nevertheless, before running any tests or conducting experiments, we need to determine the size of the sample on which we are to run the test. Should we ask 10 user, 100 users, or 1 000 000 users to understand the consequences?
Evan Miller is the guy who’s going to help us. The sample size calculator that he has developed is especially efficient when working with split tests. Its first advantage is that even before the start of the method of verification, you already have information about the sample size you need to have, i. e. how many people you are to interview or test on, as well as the duration of the method of verification, to get a statistically significant result.
Sometimes the calculator might be considered as a wake-up call or a warning about the non-expedience of further work. When, for example, for your solution to be mathematically sound for you to be able to make a decision, your sample size should be millions of millions, or the experiment should last 20 or more years. As a matter of fact, the duration of tests should be as short as possible, because if it takes 6+ months to test a specific hypothesis, when you engage all team members and put all your effort, spend a lot of time, and in the end the hypotheses proves wrong, then you simply spend your precious time and resources on nothing. All decisions should be made timely and in the shortest time possible.

What to Test?

You can test anything. Anything that may influence the behaviour of customers: colours of buttons, sizes of pictures, fonts, background, text of the message, position of various elements, alignment, etc. Just keep in mind that while spending time and efforts on something not so important, which you know will not bring any value (or, if any, then that value is not going to make a big difference), you lose an opportunity of gaining benefit and useful knowledge. It is advisable to test only those hypotheses that you have confidence about and strongly believe in.
Let's say we have a website and we wish to experiment with its title, because there’s usually enough traffic on the title page and the title of the page really affects registration. We build a hypothesis: If we change title to 'Free Website Builder', then we will double C0. We should change it, because competitors mostly use this title. Currently, our C0 is 10%, C1 is 5%, there are 20 000 visitors, of them 2 000 registrations and 100 purchases. Keep in mind that we are trying to increase C0 and this is the metric of importance. All other metrics simply provide additional information.
We fill in information in the calculator. Baseline conversion rate is 10% and since we would like to double C0, the minimal relative detectable effect is 100%. In order to understand if we are outside of the range, we need a sample size of 157 people per variation to perform the experiment. There were 2 000 visitors to the website. The experiment showed that the number of registrations has slightly increased. It means when people see 'Free Website Builder', they are more inclined to register. On the other hand, C1 and purchases decreased (2.5% and 6 respectively). Our plan was that C0 reached 20%, but we achieved only 12%. Did we increase registrations? The answer is 'No'. We did not increase registrations. The experiment proved to be negative.
The time has come to test another hypothesis. Members of our team said that we needed a faster website. A faster website will provide us with more users who have more money. Nevertheless, it’s very complicated to make the website faster. What should we do? We test what will happen if we make the website slower. A hypothesis is created: If we decrease the loading speed from 1 second to 3 seconds, then nothing will change in C0, because all websites are slow in this region.
We decreased the loading speed. What impact will it have on us? Let’s compare. We had a fast website and a slow website. By the way, when we talk about the results that will not change or have a minor change, we also need to think about the accuracy. If the minimum detectable effect is to be 1%, then the sample size should be almost 1.5M and the duration of the test then would be 5 years. It is obvious that nobody will test for that long. Maybe we can then take less accuracy? If we take 10% of accuracy, we would need 14 000 users. The difference is outstanding, right?

What results do we get? We got enough traffic, we got registrations, when we decreased the speed there were less registrations with the slower website. C0 was 10% and then became 9.33%. The number of purchases also slightly changed. Our hypothesis was that if we decrease loading speed, nothing will change. It became obvious that that C0 decreased from 10% to 9.33% and we might start thinking that we have lower conversion. As a matter of fact — no. Another advantage of the calculator, is that it also indicates that conversion rates in the grey area will not be distinguishable from the baseline (in our case it’s 9.9%-10.1%). We would have got significant results if we got outside of 9-11. The result that we got — 9.33% — proves that nothing will changed, i.e. we obtained statistical proof that if we make our loading speed slower, from 1 to 3 seconds, C0 will not change. With accuracy of 10%. Of course, you can always have a better accuracy, but then you’d have to test for 5 years. All in all, our hypothesis proved positive.

Summing Up

We make decision every day, hundreds of times a day. Tea or coffee? Car or bus? Go or stay? Do or skip? People usually make a choice of those simple daily activities on the basis of their mood, preferences or the weather, but when speaking about the world of entrepreneurship, to count on intuition or one’s mood is really not an option. Your thoughts or opinion is nothing, data and information are everything. Therefore, in order to make decisions that will bring positive results and make your business grow and flourish, it is necessary to have statistically significant results and proof.