Academic journal article Human Factors

The Utility of Event-Based Knowledge Elicitation

Academic journal article Human Factors

The Utility of Event-Based Knowledge Elicitation

Article excerpt

Ren[acute{e}]e J. Stout

The purpose of this investigation was to describe and evaluate an event-based knowledge elicitation technique. With this approach experts are provided with deliberate and controlled job situations, allowing investigation of specific task aspects and the comparison of expert responses. For this effort a videotape was developed showing an instructor pilot and student conducting a training mission. Various job situations were depicted in the video to gather information pertinent to understanding team situational awareness. The videotape was shown to 10 instructors and 10 student aviators in the community and responses to the videotape were collected using a questionnaire at predetermined stop points. Consistent with expectations, the results showed that more experienced respondents (i.e., instructors) identified a richer database of cues and were more likely than students to identify strategies for responding to the situations depicted, providing some empirical evidence for the validity of the event-based techni que. This method may serve as a useful knowledge elicitation technique, especially in the later stages of a job analysis when focused information is sought.

INTRODUCTION

Knowledge elicitation is a component of knowledge acquisition in which information pertaining to the reasoning and other thought processes needed to perform a job is obtained from a human source. Knowledge elicitation has become an increasingly important task in modem work environments where understanding the cognitive requirements associated with highly complex jobs is critical. It is also a task associated with frustration (e.g., prying information from experts), large time investments (e.g., in coding, collecting, and analyzing verbal protocols), and, worst of all, "art," in that the quality of the information received depends on the interviewer's technique and experience (Cooke, 1994; Duda & Shortliffe, 1983; Hoffman, 1987).

Considering the importance of knowledge elicitation as well as its inherent difficulties, it is necessary that the analyst be equipped with a variety of techniques that can be selected based on their suitability to the problem at hand. The purpose of this paper is to describe an assessment of one such technique, referred to as event-based knowledge elicitation. Event based means that the expert is provided with known and controlled job situations. These are selected because prior analysis (e.g., interviews with subject matter experts) suggests that experts' reaction to them will reveal meaningful information about specific aspects of the job.

This approach is similar to case study (Cooke, 1994) and test case (Hoffman, Shadbolt, Burton, & Klein, 1995) protocols. They allow a priori expectations to be developed and allow data to be collected from a number of experts on the same stimulus set so that their responses can be compared. Although tradeoffs occur in the application of any knowledge elicitation technique, these approaches appear to yield useful information when focused information is sought. However, apart from their apparent usefulness, little is known empirically about the validity of such approaches. As Hoffman et al. (1995) noted, it is not enough to know that a particular technique appears useful; other types of evidence are needed.

In order to provide some context for event-based knowledge elicitation, Table 1 provides examples of "direct" knowledge elicitation approaches -- that is, approaches that obtain knowledge by directly asking or observing the expert. Several general comments can be made about these techniques. First, in one way or another, all are situated or embedded in a job context. In a structured interview, for example, an expert may be prompted to recall a job situation. In the method of familiar tasks (Hoffman, 1987), experts are instructed to provide a commentary on their actions as they perform their job. The embedded aspect of knowledge elicitation is critical for revealing meaningful information about the job (Hoffman, 1987). …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.