descriptions, the evaluators might not fully realise the heterogeneity of the user group or their walkthrough might be anchored to their own experience with the system. Each of the four questions drives the evaluator to think of the user's behaviour in a certain situation. When the fictive user description becomes too fuzzy or lacks details to judge the user's behaviour, the evaluator unintentionally substitutes the description with a particular user much like herself/himself. Thus, evaluators tend to produce success stories if they imagine themselves having no troubles using the feature in question, and they report problems when they imagine themselves having troubles in the particular situation. In this sense a single evaluator using CW resembles an evaluator performing a thinking-aloud study with one user, namely himself/herself.
Wharton et al. ( 1994) state that CWs can be performed by individual evaluators as well as by groups of co-operating evaluators. For inexperienced CW evaluators our study strongly indicates that several evaluators are necessary to achieve a performance that is acceptable for practical use of the CW technique. Additional studies are required to learn how more experienced evaluators perform and to study more closely why we see these individual differences.
Jacobsen, N. E., Hertzum, M. & John, B. E. ( 1998). "The evaluator effect in usability studies: problem detection and severity judgments". In Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting ( Chicago, October 5-9, 1998), pp. 1336-1340. Santa Monica: HFES.
John, B. E. & Mashyna, M. M. ( 1997). "Evaluating a multimedia authoring tool". Journal of the American Society for Information Science, 48( 11), 1004-1022.
Lewis, C., Polson, P., Wharton, C. & Rieman, J. ( 1990). "Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces". In Proceedings of the ACM CHI'90 Conference ( Seattle, April 1990), pp. 235-242. New York: ACM Press.
Nielsen, J. ( 1994). "Heuristic evaluation". In Nielsen, J. & Mack, R. L. (Eds.): Usability Inspection Methods, pp. 25-62. New York: John Wiley.
Perstrup, K., Frϕkjær, E., Konstantinovitz, M., Konstantinovitz, T., Sorensen, F. S. & Varming, J. ( 1997). "A World Wide Web-based HCI-library designed for interaction studies". In Third ERCIM User Interfaces for All Workshop ( Obernai, France, November 1997).
Wharton, C., Rieman, J., Lewis, C. & Polson, P. ( 1994). "The cognitive walkthrough method: a practitioner's guide". In Nielsen, J. & Mack, R. L. (Eds.): Usability Inspection Methods, pp. 105-140. New York: John Wiley.
-1067-
