Monitoring Task Loading with Multivariate EEG Measures during Complex Forms of Human-Computer Interaction

Beginning of article

**********

Electroencephalographic (EEG) recordings were made while 16 participants performed versions of a personal-computer-based flight simulation task of low, moderate, or high difficulty. As task difficulty increased, frontal midline theta EEG activity increased and alpha band activity decreased. A participant-specific function that combined multiple EEG features to create a single load index was derived from a sample of each participant's data and then applied to new test data from that participant. Index values were computed for every 4 s of task data. Across participants, mean task load index values increased systematically with increasing task difficulty and differed significantly between the different task versions. Actual or potential applications of this research include the use of multivariate EEG-based methods to monitor task loading during naturalistic computer-based work.

INTRODUCTION

This paper documents the development and evaluation of a neurophysiology-based method for deriving a continuous index of task loading from individuals engaged in complex computer-based work. Task loading is used here to refer to the degree to which neural resources are activated by effortful task performance. Because a human operator's functional capacity is limited and can often be exceeded in work environments that demand sustained vigilance to multiple streams of information, the likelihood of performance errors can be high (Card, Moran, & Newell, 1983). The ability to monitor task loading continuously might thus be valuable in task analysis research, in efforts to improve the usability of human-computer interfaces (Raskin, 2000), and in efforts to design appropriate adaptive automation strategies (Byme & Parasuraman, 1996; Morrison & Gluckman, 1994; Parasuraman, Sheridan, & Wickens, 2000).

Although both subjective estimates and assessment of overt behavioral performance can permit detailed inferences about task loading, other modalities might also provide convergent insights into the degree to which a task environment is demanding of neural resources. To be useful in applied contexts, a monitoring method must be robust enough to be reliably measured under relatively unstructured task conditions and sensitive enough to consistently vary with some dimension of interest. Furthermore, it should not interfere with operator performance, it should be applicable across many contexts, and it should have reasonably good time resolution. Some physiological measurements - particularly measures of central nervous system activity, such as the electroencephalogram (EEG) - appear to meet these requirements.

EEG and event-related potential measures have been shown to be fairly sensitive to variations in task difficulty (Gevins, Smith, McEvoy, & Yu, 1997; Humphrey & Kramer, 1994; McCallum, Cooper, & Pocock, 1988; Parasuraman, 1990). In a recent study, Gevins et al. (1998) demonstrated that EEG-based, participant-specific, multivariate pattern recognition methods could be used to discriminate levels of task loading under highly controlled task conditions. In that study, continuous EEG data were recorded from 8 healthy adults performing a simple task in which working memory demands were varied in order to modulate task difficulty. As task difficulty increased, performance accuracy decreased, response times increased, and a change in the power of the EEG spectra in the theta (5-7 Hz) and alpha (8-12 Hz) frequency bands occurred. In particular, during the highest task load there was an increase in theta power over frontal regions of the scalp and a decrease in alpha power over widespread scalp regions, relative to me asures taken during the lowest task load. Two-class, neural-network-based pattern recognition functions applied to EEG spectral features reliably distinguished low load from high load and moderate load from low or high load.

Such results suggest that with further development, it might become possible to use EEG-based methods for unobtrusively monitoring task loading in individuals engaged in computer-based work. However, the task used in the prior study made minimal demands on perceptual and motor systems and required that a participant's effort be focused on only a single activity. The ability to reliably measure task loading in individual participants under such constrained circumstances might in itself be useful. For example, it has been applied to the problem of assessing the effect of environmental stressors on cognitive functions (Gevins & Smith, 1999). Even so, in more naturalistic work environments, task demands are usually less structured, and mental resources often must be divided between competing activities. Thus it remains to be determined whether such methods can successfully generalize to the problem of monitoring task load under a wider range of activities.

Recent studies have demonstrated that more complicated forms of human-computer interaction, such as playing video games, produce a mental effort-related modulation of the EEG that is similar to that observed during controlled laboratory tasks (Pellouchoud, Smith, McEvoy, & Gevins, 1999; Smith, McEvoy, & Gevins, 1999). This implies that it might be possible to extend EEG-based multivariate methods for monitoring task load to such circumstances. The present study was thus performed in order to further evaluate the potential of the EEG as a modality for monitoring the degree to which an individual's mental resources are engaged by this form of computer-based work.

EEG measurements were made as participants performed the Multi-Attribute Task Battery (MATB; Comstock & Arnegard, 1992). The MATB is a computer-based multitasking environment that simulates some of the activities a pilot might be required to perform. It has been used in several prior studies of mental workload and adaptive automation (e.g., Fournier, Wilson, & Swain, 1999; Parasuraman, Molloy, & Singh, 1993; Parasuraman, Mouloua, & Molloy, 1996). The data collected during performance of the MATB were used to test whether it is possible to derive combinations of EEG features that can be used for indexing task loading during a relatively complex form of human-computer interaction.

METHOD

Participants

Sixteen healthy young adults (21-32 years, mean age 26 years; 8 women, 8 men) participated in the study. All were right handed and had normal or corrected-to-normal vision and hearing. Participants received a cash honorarium in exchange for their participation. All participation was fully informed and voluntary.

Task Structure and Procedure

Study participants performed versions of the MATB that varied in difficulty. The MATB included four concurrently performed subtasks in separate windows on a computer screen (for graphic depictions of the MATB visual display, see Fournier et al., 1999, and Molloy & Parasuraman, 1996). Performance data were collected for each task separately from keyboard and joystick controls. The four subtasks were systems monitoring, resource management, communications, and tracking. The systems monitoring task required the operator to monitor and respond to simulated warning lights and gauges. In the resource management task, fuel levels in two tanks had to be maintained at a certain level. Fuel level could be controlled by pressing keys to turn on and off a series of pumps.

The communications task simulated receiving audio messages from air traffic control and required pressing keys to make frequency adjustments on navigation and communication radios. In the communications task, distractor stimuli were occasionally presented in which frequency instructions for a different aircraft (identified by a call sign different from that of the operator's own aircraft) were given. The two-dimensional compensatory tracking task simulated manual control of aircraft position using a joystick with first-order control characteristics. Participants were seated comfortably about 60 cm from the computer screen as they performed the task battery.

Prior to the test day, participants completed a training session to ensure that they were well practiced on the tasks before the EEG recording. On the practice day, participants were first given oral instructions for the four subtasks, emphasizing that effort should be divided equally among them and that response times and accuracy were equally important. Participants initially practiced the tracking component alone for 3 min. In subsequent 3-min runs, the resource management, then monitoring, and finally the communications task were added. Participants then received extensive practice for low-, medium-, and high-load versions of the task (LL, ML, and HL, respectively), first in 3-min runs and then in 5-min runs.

Manipulating the difficulty of each subtask served to vary load. Tracking was set to automatic in the LL condition, but in the ML and HL conditions a varying degree of drift in the joystick control made the tracking slightly or moderately difficult. The forcing function for tracking consisted of a sum of nonharmonic sine waves. The frequency of the forcing function was low in the ML task condition and medium in the HL task condition, according to the MATB program specifications. In the resource management task, a number of fuel pumps "failed" and subsequently reset themselves during the course of the task. Failure of the pumps inhibited the flow of fuel between tanks, rendering it more …