How to use this tool?
First of all, select from the upper dropdown examples, or find out what kinds of things you want to compare.
It can be employee A and employee B, or it can be design A and design B,
or product A and product B, and so on. Anything. You also need an event that can be measured, that has a count. For example count of visits,
count of emails sent, count of people in the experiment or something like these. Then specify special event, a positive case that is a specified subset of all cases.
If you have these all, just fill in the four input fields and you will get the ratios, difference and the certainty of the result.
The question, the names of events and cases can be rewritten, despite they are not input fields.
How does it work?
It's a two-sample hypothesis testing that calculates the statistical significance with a very complex algorithm based on gauss distribution.
But we don't want to bore you with math behind it, just use it. You don't need university-level math in order to use a tool that is based on that.
Why you can't be absolutely sure?
Because the difference between the rates of case A and case B may come from the work of coincidence.
Your data comes from an experiment or observation that is not exactly repeatable, they are not accurate, there is a fluke in them.
If you would measure again, you would get different values. This distribution causes that you can be sure
about one case is better than the other, only if the numbers and differences are big.
The greater the numbers and the difference level of the ratios, the greater the certainty.
Think about it: If you have a red website with 100 visitors and 10 of them buy something, and you have
a blue website too with 100 visitors likewise where 12 of them buy, it may be just the effect of randomness,
so you cannot say it with complete certainty that the blue website performs better.
What does the certainty percentage show?
You can easily calculate whether case A or case B is better, but can you tell how sure you are?
Well, sometimes yes, sometimes no.
In case of small numbers and small differences in ratios (e.g. A:100->51, B:100->52) it is not sure that there is a real difference.
In case of big numbers and big differences in ratios (e.g. A:1000->45, B:1000->756) it is sure that there is a real difference.
But what if A:100->50 and B:100->60, how sure are you? The above certainty value shows you as a percentage
how much you can be sure about the fact that one case is better than the other.
Nevertheless it does not tell you anything about the strength of the difference.
If you are 100% sure that A is better than B, it means only that A is better somewhat, but it cannot be told how much,
because you can conclude it only from the data of the experiment or observation.
What does low sureness mean?
It means that from these numbers it cannot be known whether case A or case B is better. So it does not mean that
there is no difference and the difference experienced is the work of coincidence, but it means you cannot be sure, whether it is
the work of coincidence or a real difference.
The certainty is said to be low below 95%, so you can't be sure about the result. Generally, 95% or greater sureness is required.
If you reach 99% or better, then you can be sure already. (But there is 1% chance, that the difference happened because of a very rare coincidence.)
What does high sureness mean?
It means that one case is surely better than the other.
Usually above 95% or 99% certainty level is considered to be high.
It's important that despite of the certainty being high, it only means that one case is better than the other,
but the difference between the two cases may be minimal.
This is why you must also check the experienced difference of ratios.
How to increase the certainty?
You need more data. If you continue your experiment or observation with a larger number of events, you will get better certainty,
even if the ratios don't change.