Evaluation of a Computerized Aid for Creating Human Behavioral Representations of Human-Computer Interaction

Article excerpt

INTRODUCTION

In 1983, Card, Moran, and Newell developed the goals, operators, methods, and selection rules (GOMS) methodology for modeling human computer tasks. A GOMS model has four components: The goal is the state to be achieved, or end state. Goals may include subgoals. Operators are low-level actions, which make up tasks. Methods are a set or series of operators used to accomplish a specific goal. Selection rules are sets of discriminating conditions used to choose between different methods of achieving a particular goal. The GOMS methodology is essentially a process of hierarchical task decomposition in which all the operators needed to accomplish a goal are specified. These operators form a method. If more than one method of achieving the goal exists, then a selection rule is specified to determine the appropriate method, depending on the conditions of the environment at the time of task execution. These high-level operators are further decomposed as subgoals, which are then broken down into lower-level operators. When all operators have been broken down to their lowest level (e.g., they cannot be broken down any more), the process stops.

These lowest-level operators can be subsequently classified under three subsystems: the motor subsystem, the perceptual subsystem, and the cognitive subsystem. Although it is theoretically possible to model all human tasks employing these three subsystem operations, execution times for many motor tasks are not available; also, there are an infinite number of knowledge-level tasks and other creative cognitive operations related to performance for which a sequence of operations is unknown. For the case of human-computer tasks, however, empirical evidence exists regarding the time required to execute each of these primitive subsystem operators, making it possible to accurately predict the time it takes to execute any human-computer task once the sequence of operators has been defined.

Although the GOMS modeling technique has proven extremely successful in developing accurate cognitive task models, instruction and training in the details of applying this modeling technique are required. Instruction or training is typically provided in college-level courses in human-computer interaction or in CHI tutorials or workshops. Most expert operators of computer systems interfaces, however, lack the skill necessary to develop these GOMS and natural language GOMS (NGOMSL) models, as prescribed by Kieras (1997). Consequently, a research effort was conducted to build an automated tool to assist domain experts (unfamiliar with GOMS and NGOMSL analysis techniques) in developing cognitive models of human-computer interactions. Williams (2000) developed such an automated tool, the cognitive analysis tool for human-computer interaction (CAT-HCI). A number of tools have developed for eliciting and structuring knowledge within the framework of a GOMS analysis, such as GLEAN3 (Kieras, Wood, Abotel, & Hornof, 1995) and quick and dirty GOMS (QGOMS; Beard, Smith, & Denelsbeck, 1996). These tools, however, have not been evaluated with respect to the accuracy and consistency with which different individuals, skilled in a task domain but unskilled in GOMS analysis, can create such models.

One study (Baumeister, John, & Byrne, 2000) reported a comparison among QGOMS, CAT-HCI, and GLEAN5 relative to building GOMS models. These researchers, however, were skilled in this form of analysis and performed their evaluation primarily focusing on the usability of each tool. A comparison of accuracy was made among the three tools, an automatically generated model, and a hand-done (manual) version of a GOMS keystroke-level model (KLM) on the one task employed for the comparison. The comparisons of execution times among the models generated by the different tools were not validated by empirical observation. The same holds with the present research. The present work, however, builds on this earlier evaluation of tools by providing a more detailed investigation of one of the tools (CAT-HCI) evaluated by Baumeister et al. …