Clarifying Statistical Usability
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Don't Be a Turkey!: Adjustments
A sample is considered small when it includes less than 20 participants. The smaller the sample size the greater the risk that your results do not reflect the actual population. As Nassim Taleb so skillfully relates in his book Black Swan...
Imagine you're a turkey. And everyday you wake up, yawn, stretch, and walk about the pasture before sitting down to a big scrumptious meal (of whatever it is that turkeys eat). Every day the same schedule. As a turkey you think, "My, this is the life!" Every day the same thing, 364 days in a row. It is reasonable to imagine that you, as a turkey, would expect the 365th day to be exactly the same as the 364th and, for that matter, the 30th and the 2nd. Afterall, your sample leads you to believe nothing different.
Unfortunately your sample didn't cover an adequate time range because, alas, the 365th day is Thanksgiving and instead of a leisurely walk around the pasture you're on the chopping block!
Remember the 90% Completion Rate we calculated for our example? That's the turkey!
With such a small sample size, there's a high chance that we haven't seen enough participants to make a reliable prediction. So in order to account for our small population size, we'll need to make some adjustments.
Once more, here's our goal:
To be complete, statistical conclusions should always include:
- Success Probability (a.k.a. the % of users expected to pass the scenario)
- Confidence Level (a.k.a. how confident your are in your conclusion)
- Confidence Interval (a.k.a. worst-case to best-case)
We'll make two adjustments over the course of our calculation. The first adjustment will directly impact the Success Probability (goal #1). The second adjustment will impact the Confidence Interval (goal #3)
The effect of an adjustment is to, essentially, add extra people. It does no harm to use an adjustment when sample sizes are sufficiently large.