Direct Observation: Factors Affecting the Accuracy of Observers
Direct observation is a method of data collection in which the target behavior is observed and recorded as it occurs. Although thousands of studies have used direct observation, research on the accuracy of the observers who provide data is not commensurate with the widespread use of observation methodologies. There exists a general presumption that observers are collecting accurate data, as well as a belief that adequate reliability scores are synonymous with adequate levels of accuracy. Some research, however, has suggested that these views are not necessarily correct (e.g., DeMaster, Reid, & Twentyman, 1977).
The first purpose of this article is to review research on the accuracy of observers. The review is organized around seven major factors that may potentially affect accuracy: (a) reactiv-ity; (b) observer drift; (c) the recording procedure; (d) location of the observation; (e) reliability; (f) expectancy and feedback; and (g) characteristics of subjects, observers, and settings. The second purpose is to offer recommendations for increasing the accuracy of observers.
FACTORS AFFECTING OBSERVER
In direct observation, we concurrently observe and record the behaviors of interest (Repp, 1983), and in many of these situations, the observer's presence is known to the subject. A common presumption would seem to be that the subject's behaviour is the same as it would be if the observer were not present. However, reactivity surely occurs in some of these cases, as subjects respond to the presence of observers by changing their behaviors. Haynes and Horn (1982) presented a comprehensive review of studies related to reactivity and suggested that behaviors may be increased, decreased, made more variable, or not be affected at all. For example, some subjects may present themselves in their "best light," and socially desirable behaviors may increase in probability as a function of observer presence; other subjects may have the opposite reaction. When such reactivity occurs, the study's internal validity is threatened as the effects from reactivity would not have been separated from any effects of the experimental variable. External validity, or the extent to which the findings of a study can be generalized, may also be affected, and two concerns arise (Kazdin, 1980). Once is whether the findings of a study in which reactivity occurred apply to nonreactive situations. A second is pretest sensitization, wherein reactivity during baseline may sensitive subjects to the intervention and make them either more or less receptive to the experimental variable. Therefore, the results may not apply to individuals who are not similarly sensitized.
Whereas most research has focused on subject reactivity, some has focused on the effect that observation has on the behavior of the observers. This effect may be termed observer reactivity, and it has been demonstrated in two studies. In one (Hay, Nelson, & Hay, 1977), teachers who were instructed to record the behavior of students in their classrooms began giving more prompts to the observed students. In another (Hay, Nelson, & Hay, 1980), one of four teachers acting as observers increased her rate of instructing and of giving positive feedback.
Drift is a cognitive phenomenon that involves a gradual shift by the observer from the original response definition, and it results in behavior being inconsistently recorded (Hersen & Barlow, 1976). As Lipinski and Nelson (1974) noted, it is relevant to both between-group and within-group designs and thus may warrant considerable concern. When drift occurs, the data collected are no longer directly comparable across conditions, because they no longer quantify the same precise response. For example, an intervention study cannot be properly evaluated if the target response has been defined differently in the baseline and treatment phases. …