Of course, if you want a higher level of statistical significance, you’ll need a larger sample size. The only way to minimize type 1 errors, assuming you’re A/B testing properly, is to raise your level of statistical significance. And when you understand type 1 errors, you increase your odds of getting it right. The goal of CRO isn’t to get it right every time-it’s to make the right choices most of the time. It’s impossible to achieve 100% statistical significance (and it’s usually impractical to aim for 99% statistical significance, since it requires a disproportionately large sample size compared to 95%-97% statistical significance). Recognize when a type 1 error may have caused a drop in conversions so you can fix the problem Understanding type 1 errors allows you to:Ĭhoose the level of risk you’re willing to accept (e.g., increase your sample size to achieve a higher level of statistical significance)ĭo proper experimentation to reduce your risk of human-caused type 1 errors When you make a change to a webpage based on A/B testing, it’s important to understand that you may be working with incorrect conclusions produced by type 1 errors. Type 1 errors can (and do) result from flawless experimentation. As mentioned above, this could be the result of poor experimentation techniques, but it might also be the result of random chance. For example, if you A/B test two page versions and incorrectly conclude that version B is the winner, you could see a massive drop in conversions when you take that change live for all your visitors to see. Type 1 errors can have a huge impact on conversions. There’s really no excuse for a type 1 error like this. Sloppy researchers might start running a test and pull the plug when they feel there’s a ‘clear winner’-long before they’ve gathered enough data to reach their desired level of statistical significance. Improper research techniques: when running an A/B test, it’s important to gather enough data to reach your desired level of statistical significance. In the end, you have to strike a balance between your desired level of accuracy and the resources you have available. That means there’s a 5% chance these results were produced by random chance.You can raise your level of statistical significance by increasing the sample size, but this requires more traffic and therefore takes more time. For example, let’s say you’ve run an A/B test that shows Version B outperforming Version A with a statistical significance of 95%. Statistical significance measures the odds that the results of an A/B test were produced by random chance.

Since researchers sample a small portion of the total population, it’s possible that the results don’t accurately predict or represent reality-that the conclusions are the product of random chance. Random chance: no random sample, whether it’s a pre-election poll or an A/B test, can ever perfectly represent the population it intends to describe. Type 1 errors can result from two sources: random chance and improper research techniques.