the Cognitive Walkthrough Technique†
Centre for Human-Machine Interaction
Risϕ National Laboratory, Denmark
Niels Ebbe Jacobsen
Department of Psychology
University of Copenhagen, Denmark
Practising system developers without a human factors background need robust, easy-to-use usability evaluation methods. The cognitive walkthrough (CW) technique ( Lewis et al. 1990, Wharton et al. 1994) has been devised to provide such a method and is particularly suited to evaluate designs before testing with users becomes feasible and as a supplement to user testing.
While several studies have evaluated how well CW predicts the problems encountered in thinking-aloud studies (e.g. John and Mashyna 1997, Lewis et al. 1990), only Lewis et al. have assessed to what extent different evaluators obtain the same results when evaluating the same interface. Data from Lewis et al. suggests that the variability in performance among evaluators using CW is much lower than that of evaluators using heuristic evaluation or thinking-aloud studies ( Jacobsen et al. 1998, Nielsen 1994). One reason for this seemingly higher robustness of CW might be that it is a quite structured process. CW has however evolved considerably since the study of Lewis et al. Moreover, their data was limited in sample size and applicability to actual CW evaluators.
To inform practitioners and methods developers about the robustness of CW this paper investigates to what extent novice evaluators who perform a CW of the same tasks detect the same problems in the evaluated interface. While acknowledging the importance of choosing the right tasks in a CW, we have decided to focus on the actual walkthrough process.____________________