Brain Potentials after Clicking a Mouse: A New Psychophysiological Approach to Human-Computer Interaction
Nittono, Hiroshi, Hamada, Aya, Hori, Tadao, Human Factors
In human factors and engineering psychological research, event-related brain potentials (ERPs) have been applied to assess mental workload (Kramer & Weber, 2000). In particular, the P3 (P300) component of the ERP is proposed to reflect the amount of perceptual-central processing resources allocated to the eliciting event (Donchin, Kramer, & Wickens, 1986).
Currently, three ERP techniques are used for assessing mental workload: (a) relevant probe technique (Fowler, 1994; Kramer, Sirevaag, & Braune, 1987); (b) irrelevant probe technique (Kramer, Trejo, & Humphrey, 1995; Ullsperger, Freude, & Erdmann, 2001); and (c) primary task technique (Kramer, Wickens, & Donchin, 1985; Sirevaag, Kramer, Coles, & Donchin, 1989). In the first two probe techniques, mental workload of the primary task is assessed indirectly with probe stimuli to which the participant has to pay attention (relevant probes) of pay no attention (irrelevant probes). It is suggested that a smaller P3 amplitude for the probe stimuli is associated with a higher processing demand of the primary task. However, the probe techniques are valid only under the assumption that the total amount of processing resources allocated to the primary task and the probe stimuli is constant, which is not always guaranteed.
The primary task technique assesses the participant's state of attention directly by measuring the P3 to certain discrete events embedded in the primary task (e.g., infrequent step changes of the target position in a tracking task). The more processing resources that are allocated to the primary task, the larger the P3 amplitude is for primary task events. The primary task technique appears to have an advantage over the probe techniques in that it is not based on the trade-off assumption. However, a paucity of discrete events that can be used as triggers for ERP averaging allows limited opportunities for applying it to real-world tasks.
In the present study, we propose a new candidate for the primary task technique to assess mental workload in human-computer interaction (HCI). When operating a computer, the user executes an intentional action to achieve a certain goal and evaluates whether a response from the computer is fit for the goal (Frese & Zapf, 1994; Norman, 1986). Theoretically, it is possible to record ERPs for a computer's response to the user's action. If a computer is programmed to produce an irregular response infrequently, a P3 will be elicited by that event, and its amplitude will be directly related to the amount of processing resources allocated to the task. We give a nickname, mouse click paradigm, to the procedure for measuring ERPs to the event triggered by the user's intentional action, given that mouse clicking is one of the most commonly executed actions in current HCI. In addition to mouse clicking, this action-perception paradigm is applicable to key pressing and similar actions in HCI tasks.
Although the idea is attractive, several problems should be addressed before using this method for practical applications. First, little is known about the nature of the ERPs to stimuli triggered by voluntary actions. In previous studies, ERPs were typically recorded for stimuli presented by the experimenter. The participant did nothing except keep still and wait for the stimuli. The idea of recording ERPs in an interactive task has a long history (e.g., O'Connor, 1981 ; Papakostopoulos, 1980; Rohrbaugh et al., 1986), yet few empirical studies have compared ERPs to stimuli triggered by voluntary actions with ERPs to the same stimuli presented automatically (McCarthy & Donchin, 1976; Schafer & Marcus, 1973). In the present context, it is necessary to examine whether the P3 is altered by the method of triggering stimuli. In a previous study, this issue was examined using an auditory target detection task (oddball task; Nittono & Ullsperger, 2000). When the stimuli were triggered by voluntary mouse clicks, the amplitude of the P3 was larger than when the same stimuli were presented automatically without mouse clicks. …