The result is shown in table 2. The PPPH we obtained shows that the sHEM is more efficient than the cognitive walkthrough method about 8.3 times for the novice evaluator and 6.4 times for the expert evaluator. Compared to the user testing, the sHEM was found to be 4.0 to 13.4 times better.
For the purpose of comparing the cost/performance of different methods, the measure of PPPH was proposed. This measure means the number of usability problems detected by one evaluator in one hour. For the novice evaluators, the PPPH value of the sHEM is 1.54 to 2.33 times larger than the HEM, and 8.34 times larger than the cognitive walkthrough method. And for the expert evaluators, the value of the sHEM is 6.38 times larger than the cognitive walkthrough method. Thus the performance of the sHEM was found to be the best among the methods used in this study.
Kurosu, M. ( 1996) The structure of the usability concept, 11th Symposium on Human Interface 351-356 (In Japanese)
Kurosu, M., Matsuura, S., and Sugizaki, M. ( 1997) Categorical inspection method: Structured heuristic evaluation (SHEM), IEEE System Man and Cybenetics'97, 2613-2618
Kurosu, M., Sugizaki, M., and Matsuura, S. ( 1997) Usability Evaluation Technique with High Efficiency 1. Structured Heuristic Evaluation (sHEM), 2. Experimental validation of sHEM, 3. Experimental validation of modified sHEM 13th Symposium on Human Interface, 481-488, 489-494 (In Japanese)
Kurosu, M., Sugizaki, M., and Matsuura, S. ( 1998a) Structured heuristic evaluation (sHEM), Usability Professional's Association 7th Annual Conference Proceedings, 3-5
Kurosu, M., Sugizaki, M., and Matsuura, S. ( 1998b) Experimental Analysis of the Cognitive Walkthrough Method, 14th Symposium on Human Interface, 37- 42 (In Japanese)
Matsuura, S., Sugizaki, M., and Kurosu, M. ( 1998) Experimental Approach to Cognitive Walkthrough Method 1, 2 14th Symposium on Human Interface, 29- 32, 33-36 (In Japanese)
Nielsen, J. ( 1993) Usability Engineering, Wiley
Nielsen, J. and Mack, R. L. ( 1994) Usability Inspection Methods, Wiley
Nielsen, J. and Molich, R. ( 1990) Heuristic evaluation of user interfaces, Proc. ACM SIGCHI'90 Conf. 249-256
Polson, P. G., Lewis, C., Rieman, J., and Wharton, C. ( 1992) Cognitive walkthrough: a method for theory based evaluation of user interfaces, Int. J. Man-machine Studies, 36, 741-773