Academic journal article Contributions to Music Education

Adjudicator Reliability: A Comparison of the Use of Authentic State Festival Choir and Global Score Audition Forms

Academic journal article Contributions to Music Education

Adjudicator Reliability: A Comparison of the Use of Authentic State Festival Choir and Global Score Audition Forms

Article excerpt

The purpose of this study was to investigate adjudicator reliability by, comparing utilization of an authentic State Festival Choir audition form to utilization of a global score form. Participants (N = 30) included experienced adjudicators (n = 10), music educators (n = 10), and non-music educators (n = 10). Participants adjudicated CDs with performances (N = 20) of choristers singing a state festival choir audition. Results suggested that adjudicator reliability failed to exhibit acceptable levels. Findings also revealed no significant differences between participant groups. Outcomes suggested that state festival choir audition form category scores reflected adjudicators' overall impressions of the performances rather than discrete assessments. Suggestions for further research included an investigation of a more descriptive performance assessment rubric.

A recent survey of Music Educators National Conference and American Choral Directors Association state presidents concluded that methods employed in adjudicating students for state festival choirs remained relatively diverse nationwide (Wine, 1996). Audition components included sight singing, solo singing, singing in a quartet, director recommendations, and music theory tests. Findings revealed that adjudicators consisted of high school choir directors, university and college choir directors, retired high school choir directors, private voice teachers, church musicians, and doctoral students. Wine reported widespread concerns among respondents regarding audition fairness and adjudicator reliability. A later questionnaire of 150 choral directors representing a cross-section of choral programs in the United States reported the most often used audition component for state festival choirs to be solo singing (Barkey, 2005).

Reliability of adjudicators responsible for assessment of student performances has long been a concern among music educators. Radocy (1989) argued, "any measure that involves human judgment is inherently subjective because it involves human impressions" (p. 30). He suggested that music educators must recognize that all measurement procedures are inherently subjective, either in construction, application, or interpretation.

Fiske (1983) suggested that by (a) matching an adjudicator's scores for one hearing of a particular performance to those of the second hearing of the same performance and (b) subjecting those data to appropriate statistical analyses, a reliability index could be obtained for that adjudicator. He argued that such indexes rarely exceeded r = .50 or 25% consistency, while the average intra-adjudicator reliability (within a single adjudicator's ratings) was likely to range between 9% and 16% consistency. Fiske suggested that such consistency improved with an increase in the number of adjudicators on a panel, although he posited that this result was an effect of panel size rather than enhanced individual reliability.

Efforts to discern factors related to adjudication fairness have provided some insights. In a study of 402 students auditioning for a state music educator association's festival choir, Adderly (2001) observed that hour of the day could influence selection of such participants. Adderly's findings, however, were not statistically significant.

Studies of festival ratings conducted by Bergee and Platt (2003), Bergee and McWhirter (2005), Bergee and Westfall (2005), and Bergee (2006), however, concluded that extra-musical variables could significantly influence performance evaluations in solo and group contexts. They reported that certain factors such as performing in the morning rather than the afternoon, attending smaller schools with lower per pupil expenditures, performing as an instrumentalist, and performing as an ensemble member all militated against receiving superior ratings. Their findings also suggested that methods of adjudicator selection appeared to influence ratings awarded. Other extra-musical factors found to significantly influence ratings in music contest settings included physical attractiveness (Ryan & Costa-Giomi, 2004), stage behavior and dress (Wapnick, Mazza, & Darrow, 1998), and gender and ethnicity (Elliott, 1995/1996). …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.