In the world (and of probabilities), nothing is surely certain. So, when making general assumptions, we need to consider the base rate fallacy, which is the chance that – depending on the context – a simple assumption can yield different results.
One popular type is the false positive paradox, where there’s a higher chance that an assumption is wrong because of a low incidence rate in the population. I’ll adapt one of Wikipedia’s examples:
Imagine that the population of Lagos was 1000 people, and 40% of us are mad. Also imagine that the madness test has false positive rate of 5% (0.05) and no false negative rate.
If we all take the test and your results come back positive (hooray, you’re mad), should you accept your fate? No! A 5% false positive rate means 5% of the time, the test results say someone is mad when they actually are not.
But how much does this matter?
We know that 400 people are mad and will get a positive result. That leaves 600 people who are not mad. However, of these 600 people, 5% will be told they’re mad when they’re not. That’s 30 people.
There’s hope, as you can see. Even if the test results say you are, there’s a slim (30:400 ~ 7%) chance that you’re not actually mad. Whew!
This chance is greatly amplified when there’s a low incidence rate in the population. Let’s switch the example up:
Imagine that the population of Lagos was 1000 people, and 2% of us are wise. In this case as well, imagine that the wisdom test has false positive rate of 5% (0.05) and no false negative rate.
If we all take the test and your results come back positive (hooray, you’re wise), should you rejoice in your wisdom? No! If you’re truly wise, you’ll seriously consider that 5% of the time, the test results are wrong.
We know that 20 people (2% of 1000) are wise and will get a positive result. This leaves us with 980 unwise people. Of these people, 5% will be wrongly told they’re wise. That’s 49 people!
More people (49) will be wrongly told that they’re wise than the actual number of wise people (20). So, if the test says you’re wise, there’s a (49:20 ~ 71%) chance that you’re not!