Site.pro
  • Website Builder
    • Website Builder
    • Prices
    • Website Import
    • Online Stores
    • Plugins
    • Collaboration
    • Templates
    • Languages
    • Affiliate Program
    • Examples
    • AI Website Builder
    • For Design Studios
    • We Build for You
  • Domains
  • Email
  • Learn
    • Tutorials
    • Blog
    • FAQ
    • Growth Hacking
    • For Hostings
    • Domainity
    • For Education
  • For Resellers
    • Prices
    • White Label
    • Revenue Share
    • Panels
    • How it Works
    • Mass Import
    • Our Partners
    • Reselling Examples
    • Marketing Materials
    • Documentation
    • Free Websites
  • Website Builder
    • Website BuilderEverybody can easily create a website, landing page, or e-commerce store. Easy-to-use website builder.
    • PricesSite.pro Prices: Templates 200+, Websites, Basic Builder Functions, Online Stores, Get more website builder features.
    • Website ImportTransfer your existing website from anywhere to Site.pro website builder. Website import from different website builders.
    • Online StoresSell your goods or services anywhere on the website. The Easiest Way to Build an Online Store
    • PluginsMany functions and plugins depending on geographical preferences. Plugin description.
    • CollaborationCollaborative Website Builder, Create a website with teammates in real-time in one space, Real-Time Collaboration.
    • Templates
    • Languages
    • Affiliate Program
    • Examples
    • AI Website Builder
    • For Design Studios
    • We Build for You
  • Domains
  • Email
  • Learn
    • TutorialsVideos for Beginners. Download video tutorials for free New video tutorial.
    • BlogWebsite Builder Lifehacks: Shared hosting tips. Web Design, Marketing
    • FAQHelp center. Payment Issues. Basic Tutorial. Quick search. Have questions? Enter your question here.
    • Growth HackingBoost your sales. Strategy
    • For HostingsEducation. Generate more website builder sales
    • DomainityDomainity: Domain quantity per 1000 people
    • For EducationFree Website Builder for Online and Offline Classrooms. Collaborate with students in one space or separately.
  • For ResellersWhite Label
    • PricesStart reselling. Pay for live websites. Free Websites. Join reselling program.
    • White LabelThe most popular white label tool is ideal for reselling
    • Revenue ShareWebsite Builder: Provide free website builder and earn 50% commissions
    • PanelsOne product for all platforms. Download plugin for your panel
    • How it WorksCloud or On-Premises. Recommendations for Builder Server. Requirements for builder server (On-premises only) and published websites.
    • Marketing MaterialsVideo. Mockups. Website Blocks. Marketing Materials.
    • Mass Import
    • Our Partners
    • Reselling Examples
    • Documentation
    • Free Websites
  • Register

    Register

    (Buy Domain, Create email)
    (For Hosting Companies)
    By registering, you accept our Terms of Service
    Number of Customers: 1-100
    Number of Customers: 1-100
  • Log in

    Log in

    New user? Create account
    Forgot password?
  • $
  • English

Statistical Significance (Evan Miller)

Your Helper in Conducting Tests

Free Full Growth Hacking Course
Blog: Growth Hacking

00:24 Experiments, A/B testing and statistical proof
00:50 Sample size calculator. Baseline conversion rate and the minimum detectable effect
01:12 Example 1: generating a hypothesis about changing the title. Experiment
02:48 Statistics from the last period (20 000 visitors, 2 000 registration, C0 — 10%, C1 — 5%, 100 purchases)
03:16 How to use the calculator: relative vs absolute detectable effect
04:26 After the experiment: comparing data (more registrations, C1 and purchases decreased)
07:38 Example 2: generating a hypothesis about changing the title. Split testing
09:30 After split testing: comparing data (less registrations, C0 decreased, C1 and purchases increased)
10:05 The results of split testing: split testing is positive. Writing down the knowledge
12:30 Example 3: generating a hypothesis about decreasing the loading speed. Deterioration test
13:30 The importance of accuracy
14:25 After deterioration test: comparing data (registrations and purchases decreased, C0 decreased)
15:00 All attention on the metric that you measure. All other metrics only provide additional information
15:30 What is significant result
16:55 The results of deterioration test: deterioration test is positive
17:00 How many hypotheses on average prove to be positive?

In order to control something, you need good information channels. Not only that, but you also need to understand how much information you need to obtain to make a right decision. In business, you can’t make spontaneous choices or decide on the basis of your inner feelings. Business requires analytical and data-based thinking, determined and planned organization, and anticipation of risks and further steps.

Who’s Evan Miller?

Let’s suppose that we decided to implement a change to increase a specific metric and already came up with an idea that we turned into a hypothesis. Nevertheless, before running any tests or conducting experiments, we need to determine the size of the sample on which we are to run the test. Should we ask 10 user, 100 users, or 1 000 000 users to understand the consequences?
Evan Miller is the guy who’s going to help us. The sample size calculator that he has developed is especially efficient when working with split tests. Its first advantage is that even before the start of the method of verification, you already have information about the sample size you need to have, i. e. how many people you are to interview or test on, as well as the duration of the method of verification, to get a statistically significant result.
Sometimes the calculator might be considered as a wake-up call or a warning about the non-expedience of further work. When, for example, for your solution to be mathematically sound for you to be able to make a decision, your sample size should be millions of millions, or the experiment should last 20 or more years. As a matter of fact, the duration of tests should be as short as possible, because if it takes 6+ months to test a specific hypothesis, when you engage all team members and put all your effort, spend a lot of time, and in the end the hypotheses proves wrong, then you simply spend your precious time and resources on nothing. All decisions should be made timely and in the shortest time possible.

11:38 Statistical Significance. Significance, Statistical power.
14:28 Statistical Significance — why it’s important

What to Test?

You can test anything. Anything that may influence the behaviour of customers: colours of buttons, sizes of pictures, fonts, background, text of the message, position of various elements, alignment, etc. Just keep in mind that while spending time and efforts on something not so important, which you know will not bring any value (or, if any, then that value is not going to make a big difference), you lose an opportunity of gaining benefit and useful knowledge. It is advisable to test only those hypotheses that you have confidence about and strongly believe in.
Let's say we have a website and we wish to experiment with its title, because there’s usually enough traffic on the title page and the title of the page really affects registration. We build a hypothesis: If we change title to 'Free Website Builder', then we will double C0. We should change it, because competitors mostly use this title. Currently, our C0 is 10%, C1 is 5%, there are 20 000 visitors, of them 2 000 registrations and 100 purchases. Keep in mind that we are trying to increase C0 and this is the metric of importance. All other metrics simply provide additional information.
We fill in information in the calculator. Baseline conversion rate is 10% and since we would like to double C0, the minimal relative detectable effect is 100%. In order to understand if we are outside of the range, we need a sample size of 157 people per variation to perform the experiment. There were 2 000 visitors to the website. The experiment showed that the number of registrations has slightly increased. It means when people see 'Free Website Builder', they are more inclined to register. On the other hand, C1 and purchases decreased (2.5% and 6 respectively). Our plan was that C0 reached 20%, but we achieved only 12%. Did we increase registrations? The answer is 'No'. We did not increase registrations. The experiment proved to be negative.
The time has come to test another hypothesis. Members of our team said that we needed a faster website. A faster website will provide us with more users who have more money. Nevertheless, it’s very complicated to make the website faster. What should we do? We test what will happen if we make the website slower. A hypothesis is created: If we decrease the loading speed from 1 second to 3 seconds, then nothing will change in C0, because all websites are slow in this region.
We decreased the loading speed. What impact will it have on us? Let’s compare. We had a fast website and a slow website. By the way, when we talk about the results that will not change or have a minor change, we also need to think about the accuracy. If the minimum detectable effect is to be 1%, then the sample size should be almost 1.5M and the duration of the test then would be 5 years. It is obvious that nobody will test for that long. Maybe we can then take less accuracy? If we take 10% of accuracy, we would need 14 000 users. The difference is outstanding, right?

What results do we get? We got enough traffic, we got registrations, when we decreased the speed there were less registrations with the slower website. C0 was 10% and then became 9.33%. The number of purchases also slightly changed. Our hypothesis was that if we decrease loading speed, nothing will change. It became obvious that that C0 decreased from 10% to 9.33% and we might start thinking that we have lower conversion. As a matter of fact — no. Another advantage of the calculator, is that it also indicates that conversion rates in the grey area will not be distinguishable from the baseline (in our case it’s 9.9%-10.1%). We would have got significant results if we got outside of 9-11. The result that we got — 9.33% — proves that nothing will changed, i.e. we obtained statistical proof that if we make our loading speed slower, from 1 to 3 seconds, C0 will not change. With accuracy of 10%. Of course, you can always have a better accuracy, but then you’d have to test for 5 years. All in all, our hypothesis proved positive.

Summing Up

We make decision every day, hundreds of times a day. Tea or coffee? Car or bus? Go or stay? Do or skip? People usually make a choice of those simple daily activities on the basis of their mood, preferences or the weather, but when speaking about the world of entrepreneurship, to count on intuition or one’s mood is really not an option. Your thoughts or opinion is nothing, data and information are everything. Therefore, in order to make decisions that will bring positive results and make your business grow and flourish, it is necessary to have statistically significant results and proof.

Previous
Growth Process
Growth Process
Next
Unit-Economics
Unit-Economics
© Site.pro 2011. Website Builder. United States.
Contact SalesTerms of Service