Academic journal article Human Factors

Information-Processing Slope Scores Can Be Reliable!

Academic journal article Human Factors

Information-Processing Slope Scores Can Be Reliable!

Article excerpt

INTRODUCTION

A number of cognitive tasks that increasingly are being used for selection or performance assessment characterize performance in terms of the relation between average correct reaction time (CRT) and levels of the independent variable. These tasks include Sternberg's memory search task (1969), Neisser's visual search task (1963), the mental rotation task (Shepard and Cooper, 1986), and the choice reaction time task (Hyman, 1953). For many of these tasks, the relation between the average CRT and levels of the independent variable is linear and interest typically focuses on the slope of the function. This slope is hypothesized to represent the rate at which certain cognitive processes can be performed.

Over the last few years, a number of articles on the unreliability of slope scores have appeared in the human factors literature (e.g., Carter, Krause, and Harbeson, 1986; Dunlap, Kennedy, Harbeson, and Fowlkes, 1989). These articles have attracted attention because the reliability of dependent measures affects the power of a repeated-measures experiment and the upper limit of the strength of association found in correlation studies (Carter et al., 1986). These articles have convinced many human factors practitioners that the slope is a worthless dependent measure, if not actually misleading.

Neglect of slope scores is unfortunate for at least two reasons. First, as noted in Carter et al. (1986), the slope in many cases has a theoretical meaning that is not reflected in the average CRT at one level of the independent variable. Because the slope often measures the construct of most interest, much information is lost. Second, the assumption that slope or difference scores are necessarily unreliable is unwarranted.

Tables 1 and 2 present data from a mental rotation experiment (see Damos, 1989, 1991, for more details). Table 1 shows the intersession reliability matrix of the slope scores for a group of 10 male subjects using the letter F as the stimulus. Table 2 presents comparable data for a different group of 10 males using an abstract polygon as the stimulus. For both groups the slope scores were calculated by fitting a linear regression to the average CRT at 0, 60, 120, and 180 deg of absolute rotation. In each of 10 sessions, the subject saw a total of 540 standard stimuli and 540 mirror stimuli. Mean values for the slopes and the standard deviations by session are given in Table 3.

[TABULAR DATA FOR TABLE 1 OMITTED]

Clearly, the reliability of these slope scores is unusually high relative to data commonly available in the literature, which raises two questions. First, can an investigator know a priori whether slope scores from a given task will be reliable? Second, are there any practical steps an investigator can take to increase the reliability of the slope scores?

BACKGROUND

Before discussing factors affecting the reliability of slope scores, the relation between difference scores and slope scores must be clarified. Rogosa, Brandt, and Zimowski (1982) noted that "the difference score is proportional to a slope that is computed from only two data points" (p. 730). Carter et al. (1986) provided an explanation of this proportionality. Dunlap et al. (1989) pointed out that the slope concept is represented [TABULAR DATA FOR TABLE 2 OMITTED] by a weighted difference of treatment means in analysis of variance (ANOVA). An algebraic argument can be extended to any number of levels of the independent variable to show that slope scores are differences between weighted sums of scores on each side of the mean of the independent variable. (See the appendix.) Thus discussions concerning the reliability of individual differences among difference scores also are applicable to the reliability of slope scores.

ESTIMATING RELIABILITY

The first question raised earlier concerned the investigator's ability to know a priori whether slope scores for a given task could be reliable. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.