When Novices Elicit Knowledge: Question Asking in Designing, Evaluating, and Learning to Use Software
Robert Mack and Jill Burdett Robinson
This paper describes a qualitative empirical method aimed at uncovering what computer users need to know to use a computer to accomplish tasks. The method asks users to acquire information about how to use a computer by asking questions of a more experienced user (i.e., the investigator) or a "coach" (a term we prefer). The technique is both similar to and different from qualitive think-out-loud (TOL) methods for eliciting verbalizations related to thinking and problem solving. TOL verbal protocol techniques have been widely applied in cognitive psychology (see Ericsson & Simon, 1980, 1984). Although question asking may have equally wide applicability, the focus of this paper will be on its use in studying human-computer interaction issues connected with the design and evaluation of computer systems.
We begin with a brief overview of question asking and question answering in the broader context of cognitive and social psychology. We then briefly review the few accessable studies of computer user behavior that have applied question asking to the evaluation of computer software. This includes a more in-depth discussion of a software evaluation carried out in our lab using question asking. Applications of question asking have discussed the technique mostly in an anecdotal way, without grounding the method in the larger context of possible qualitative methods, or methodological and interpretive issues (but see Kato, 1986). In a general discussion we summarize and discuss in more depth methodological issues and guidelines for practitioners of the method. We also sketch possible research directions for better understanding the methodology.
Asking questions is a basic linguistic strategy people have for communicating and, in particular, acquiring knowledge about the physical or social world. Not surprisingly, question asking and question answering are basic tools of empirical inquiry in social and cognitive science for learning what someone may or may not know about some domain. More recently, question asking and answering have become topics of research in their own right in cognitive science, particularly through research of Graesser and colleagues. (For review of research in cognitive psychology and artificial intelligence see Graesser & Black, 1985.) In this paper, we focus on question-asking methods in a new domain, that of human- computer interaction (HCl), where a key focus is developing methodologies for evaluating and diagnosing problems with users' interaction with computer systems.
The HCl domain is multidisciplinary, with basic and applied goals, and its methods and professional membership draw on traditional software human factors and ergonomics, as well as cognitive psychology and computer science. Its goals are (a) to understand and