Usability Satisfaction Questionnaires
James R. Lewis
International Business Machines Corp.
Psychometrics is a well-developed field in psychology, and usability researchers began to use psychometric methods to develop and evaluate questionnaires to assess usability a little over ten years ago ( Sweeney & Dillon, 1987). The goal of psychometrics is to establish the quality of psychological measures ( Nunnally, 1978). Is a measure reliable (consistent)? Given a reliable measure, is it valid (measures the intended attribute)? Finally, is the measure appropriately sensitive to experimental manipulations'? Here is a brief review of some basic elements of standard psychometric practice.
Reliability goals . In psychometrics, reliability is quantified consistency, typically estimated using coefficient alpha ( Nunnally, 1978). Coefficient alpha can range from 0 (no reliability) to 1 (perfect reliability). Measures of individual aptitude (such as IQ tests or college entrance exams) should have a minimum reliability of .90 (preferably a reliability of .95). For other research or evaluation, measurement reliability should be at least .70 ( Landauer, 1988).
Validity goals . Validity is the measurement of the extent to which a questionnaire measures what it claims to measure. Researchers commonly use the Pearson correlation coefficient to assess criterion-related validity (the relationship between the measure of interest and a different concurrent or predictive measure). Moderate correlations (with absolute values as small as .30 to .40) are often large enough to justify the use of psychometric instruments ( Nunnally, 1978).
Sensitivity goals . A questionnaire that is reliable and valid should also be