Academic journal article Measurement and Evaluation in Counseling and Development

Two Versions of the Contrasting-Groups Standard-Setting Method: A Review

Academic journal article Measurement and Evaluation in Counseling and Development

Two Versions of the Contrasting-Groups Standard-Setting Method: A Review

Article excerpt

This article reviews the literature on the contrasting-groups standard-setting method and describes 2 versions of the method-tire person-focused version and the response-focused version. Of the 2, the response-focused version is preferable. It can be enhanced by adopting some practices of the modified Angoff method.

For nearly 30 years, educators have used criterion-referenced test scores to make judgments about student knowledge and skills. This began in earnest in the 1970s with the minimum-competency testing movement. Minimum-competency test scores were typically used to show whether graduating high school students achieved acceptable levels of knowledge and skills. The process known as standard setting was used to classify students as masters or nonmasters. As part of the standards-based assessment movement that began in the 1990s, many districts began testing students in several grades and setting standards in each grade. Standard setting is exemplified by the National Assessment of Educational Progress (NAEP), which began classifying and reporting scores in performance-level categories in the early 1990s. The recent No Child Left Behind Act requires states to administer assessments in three grades and to report the percentages of students meeting standards; to meet this requirement, standard-setting activities will increase considerably.

Standard setting involves developing performance standards and setting cut scores. A performance standard is a written description of the knowledge or skills that students must demonstrate to show that they meet a specified level of performance in a subject-matter area; and a cut score is the test score separating students who achieve the performance standard from those who do not (Kane, 1994). A performance standard is a concept, and a cut score is the operational definition of the concept.

Many standard-setting methods have been developed. Most of these methods are classified either as test-centered methods or examinee-centered methods. Test-centered methods focus on items as the objects of judgment. The best known and most studied test-centered method is the modified Angoff method (Angoff, 1971; Brandon, 2002). With this method, judges develop a performance standard and then estimate the probability that an examinee (or a group of examinees) will correctly answer each item on the test or assessment at the level of the performance standard. Item estimates are summed for each judge, and the sums are averaged across judges, resulting in the cut score. In large-scale assessments, districts and states typically set cut scores at three or four levels.

In contrast to test-centered methods, examinee-centered methods focus on examinees as the objects of judgment. The most common examinee-centered methods are the contrasting-groups method and the borderline-group method. Of these, the former has been empirically examined the most. The contrasting-groups method is commonly described (e.g., Livingston & Zieky, 1982) as a procedure in which standard-setting judges classify a group of examinees (or potential examinees) according to the adequacy of their knowledge of the subject matter measured by the test, with a cut score then set at the point that divides the two groups. This version of the contrasting-groups method is a person-focused procedure. That is, it focuses on the people taking the examination. In an alternative version, judges classify examinees' completed examinations. This version of the contrasting-groups method is response-focused. That is, it focuses on judging examinees' responses to tests. The distinction between the person-focused and response- focused versions, which has not been made in previous articles about the contrasting-groups method, is important because the procedures carried out in the primary steps of the method vary between the two versions. The primary steps of the method that have been carried out in operational standard-setting studies are (a) selecting judges and examinees (for person-focused studies) or examinations (for response-focused studies), (b) training the judges, (c) developing the definition and description of the performance standard (i. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.