First of all, I am not a statistician. I'm a Usability Practitioner, with 3 years experience working on an amalgam of applications, who followed her curiousity about calculating statistics on usability findings. A Statistics 101 class plus the plethora of information at MeasuringUsability.com form the basis for this tutorial.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Statistical analysis is done differently with small sample sizes (as is common in usability testing) than with typical quantitative research studies. This tutorial provides usability practitioners - who might be new to statistics - step-by-step instructions on how to conduct statistical analysis of their lab findings and how it can be used to inform their conclusions.
A brief survey of the web reveals a dearth of instructive information on this topic with one clear exception: Jeff Sauro's MeasuringUsability.com. It is an awesome resource that caters to both the newbie and the expert. In fact, the information there forms the foundation for much that you'll find here.
But it does not cater very well to the newbie who wants to understand the mechanics behind it all, for whom piecing the disparate references together is likely to be an insurmountable task. For such a "curious novice", this tutorial follows example data through the individual steps of the calculation. It aims to lay out these steps in a clear, linear fashion, along the way explaining the context of each step, defining the technical jargon in laymen's terms and then, finally, discussing some of the issues inherent in this type of analysis.
So if what you want is to simply calculate an accurate statistic for your data and you do not care how it's generated, then Sauro's Confidence Interval Calculator is what you need. If you want to understand the mechanics behind the calculator, then this tutorial is for you.
After three years working in the field I have seen more than a few usability success stories: applications saved by insights gleaned from timely testing with real end-users. Most of these conclusions were derived (and recommendations made) without the use of statistics. So beyond the academic exercise of knowing how to calculate accurate statistics for small population sizes, its informative to understand why streamlined usability testing works so well and where more rigorous analysis is warranted. I share what I have discovered in that regard here as well.
Again, much credit goes to Jeff Sauro and Jim Lewis. Please feel free to contact me at andrealspray(at)gmail(dot)com with any questions or concerns.