Data collected and analyzed by the Operator Licensing Branch at the U.S. Nuclear Regulatory Commission (NRC) has shed additional light on the discriminatory effectiveness of validated, modified, and new test items in examinee test performance. Examination data gathered over a period of five years shows remarkable and consistent differences in test item performance among validated (bank items), modified items, and new items used on the NRC Generic Fundamentals Examination (GFE). Since there is a paucity of empirical research in this area, these recent and substantive findings may be valuable to organizations and agencies who use test item banks in developing examinations and serve as useful information in setting examination development policy.
Although the NRC had not originally set out to measure the effectiveness of item discrimination among three categories of test items, the data on examination results, as collected and presented over time, began to yield a clear and distinctive pattern on how each category of item functioned. This serendipitous finding served as the basis to continue data collection and analysis. For five continuous years and over 28 separate examinations, the pattern of differences in item discrimination among item categories has remained consistently distinctive as will be shown and discussed in the Tables and discussion that follow.
Since 1989, the Operator Licensing Branch of the Office of Nuclear Reactor Regulation at the NRC has administered the Generic Fundamental Examination (GFE). This examination is administered twice yearly to candidates seeking a Reactor Operator (RO) or Senior Reactor Operator (SRO) license in nuclear power plants nationwide. In 1991, the NRC initiated a successful approach to GFE examination development through the combined use of validated, modified, and new items to yield a discriminating, content-valid examination.
The GFE are two separately administered 100 item examinations specific to Boiling Water Reactors (BWR) and Pressurized Water Reactors (PWR). The license examinations measure candidate knowledge in three areas: (1) reactor theory (2) plant components, and (3) thermodynamics. This examination must be passed with a minimum score of 80 percent before candidates are eligible to take the plant-specific written examination and operating test at their facilities.
The GFE examination measures fundamental knowledge applicable to all reactor operators and senior reactor operators; for example, test items that measure candidate knowledge of plant components include questions on valves, sensors and detectors, controllers and positioners, pumps, motors, generators and so forth while reactor theory questions include questions on topics covering reactivity coefficients, control rods, neutrons, fission product poisons, and others. In essence, the GFE assesses candidate basic knowledge for understanding the elements of nuclear power and plant behavior, and in this regard, represents the underpinnings for later control room related problem-solving, trouble-shooting, and decision-making.
The GFE multiple choice item format has proven successful for objective scoring and standardizing a nationally-administered exam of this type. The multiple choice format has also proven to be effective for testing higher cognitive thought processes. Carefully, well-designed multiple choice test items require candidates to think through responses and alternatives where they must weigh and consider the conditions posed in the stem of the question and further discriminate and eliminate among plausible distractors before choosing the correct answer. The mental processes involved in arriving at the correct answer often challenge candidates to analyze and synthesize information in a problem solving context. An example follows of a reactor theory test item taken from a past PWR exam:
Following a reactor trip, the power decrease rate initially stabilizes at negative one-third decade per minute when: