back to other calculators

A/B Test Calculator Online

This free online A/B test calculator shows if case A or case B is better. Moreover, it shows how sure it is that there is a real difference between the two cases and it’s not just the work of coincidence. Higher the certainty level, the more sureness for you about the fact that one is better than the other. Everyone needs to measure sureness, certainty, confidence of the results.

How to use this tool?

First of all, find out what kinds of things you want to compare. It can be employee A and employee B, or it can be design A and design B, or product A and product B, and so on. Anything. You also need an event that can be measured, that has a count. For example count of visits, count of emails sent, count of people in the experiment or something like these. Then specify what is your conversion, i.e. a special event or a positive case that is a specified subset of all cases. If you have these all, just fill in the four input fields and you will get the ratios, difference and the certainty of the result. The question, the names of events and cases can also be rewritten, despite they are not input fields, but avoid these A/B test common mistakes.

Why do you need A/B Test?

Whenever you have to make a decision, you will want to compare cases and choose the better one. You may do so by tossing a coin, intuition, common sense, or other unreliable ways but it is better to use your data. You can read more about "Ten Reasons Why A/B Split Testing Is Necessary"

How to measure difference?

Suppose you have two cases, case A has 20 conversions out of 100 events (20%), and case B has 30 conversions out of 100 events (30%). In this example, the simple difference: 30% - 20% = 10%. The proportional difference: (30% / 20%) * 100 - 100 = 50%. This means that case B is 50% better than case A, despite that simple substraction of conversions rates shows only 10%. Proportional difference is generally more useful because that will tell you the increase in the number of conversions.

How does it work?

It's a two-sample hypothesis testing that calculates the statistical significance with a very complex algorithm based on gauss distribution. But we don't want to bore you with math behind it, just use it. You don't need university-level math in order to use a tool that is based on that.

Why you can't be absolutely sure?

Because the difference between the rates of case A and case B may come from the work of coincidence. Your data comes from an experiment or observation that is not exactly repeatable, they are not accurate, there is a fluke in them. If you would measure again, you would get different values. This distribution causes that you can be sure about one case is better than the other, only if the numbers and differences are big. The greater the numbers and the difference level of the ratios, the greater the certainty. Think about it: If you have a red website with 100 visitors and 10 of them buy something, and you have a blue website too with 100 visitors likewise where 12 of them buy, it may be just the effect of randomness, so you cannot say it with complete certainty that the blue website performs better.

What does the certainty percentage show?

You can easily calculate whether case A or case B is better, but can you tell how sure you are? Well, sometimes yes, sometimes no. In case of small numbers and small differences in ratios (e.g. A:100->51, B:100->52) it is not sure that there is a real difference. In case of big numbers and big differences in ratios (e.g. A:1000->45, B:1000->756) it is sure that there is a real difference. But what if A:100->50 and B:100->60, how sure are you? The above certainty value shows you as a percentage how much you can be sure about the fact that one case is better than the other.

What does low sureness mean?

It means that from these numbers it cannot be known whether case A or case B is better. So it does not mean that there is no difference and the difference experienced is the work of coincidence, but it means you cannot be sure, whether it is the work of coincidence or a real difference. The certainty is said to be low below 95%, so you can't be sure about the result. Generally, 95% or greater sureness is required. If you reach 99% or better, then you can be sure already. (But there is 1% chance, that the difference happened because of a very rare coincidence.)

What does high sureness mean?

It means that one case is surely better than the other. Usually above 95% or 99% certainty level is considered to be high. It's important that despite of the certainty being high, it only means that one case is better than the other, but the difference between the two cases may be minimal. This is why you must also check the experienced difference of ratios.

How to increase the certainty?

You need more data. If you continue your experiment or observation with a larger number of events, you will get better certainty, even if the ratios don't change.

Cookies help us in delivering our service. You consent to our cookies and you agree to our privacy policy and cookie policy if you continue to use our website. Learn more about our privacy policy here and cookie policy here.