Academic journal article The Journal of Faculty Development

Helping Stakeholders Understand the Limitations of SRT Data: Are We Doing Enough?

Academic journal article The Journal of Faculty Development

Helping Stakeholders Understand the Limitations of SRT Data: Are We Doing Enough?

Article excerpt

Student Ratings of Teaching (SRT) are widely employed to evaluate teaching for both formative and summative purposes. The appropriateness of items on these forms is one factor among many determining the utility and validity of SRT. Another is whether stakeholders are sufficiently knowledgeable to interpret the resulting data. Evidence gathered from 30 faculty developers suggests that professional development efforts to help decision-makers understand the limitations of SRT data are extremely uncommon. This article presents six research-based guidelines for using SRT data and explores ways that faculty developers can help institutions provide this information to decision-makers and encourage them to incorporate these guidelines into their evaluation practices.

Because teaching is a complex human endeavor, no single measure can accurately evaluate it. Employing a single measure to evaluate an instructor is analogous to appraising a baseball pitcher by using just one o/the statistics: earned run average, win-loss record, walks and hits per inning pitched, opposing team's batting average, or number of saves. The research literature has long called upon institutions of higher education to use multiple measures when evaluating teaching (Arreóla, 2007; Cashin, 1995, 1999; Centra, 1993; Knapper & Cranton, 2001). In practice, Student Ratings of Teaching (SRT) are the most commonly used measure (Seldin, 1999) and, typically, the primary measure (Loeher, 2006). Even worse, "many institutions reduce their assessment of the complex task of teaching to data from one or two questions" (Fink, 2008, p. 4).

Getting valid and reliable data about teaching from SRT depends on a number of factors. The forms must be properly constructed and administered. The ratings data summaries must also include essential information such as response rates, averages and measures of dispersion, and appropriate benchmarks, if possible (Franklin, 2001). It is also crucial to ensure that individuals making personnel decisions, especially for retention, tenure, and promotion, understand what conclusions can be reliably and validly drawn from SRT data and what conditions are necessary to do so. Unfortunately, as this article will show, it appears that many faculty, department chairs and deans receive little in the way of assistance to understand the limitations of SRT data despite documented concerns about the qualification of the users of this data (Franklin & Theall, 1989, 1990).

Background

Concern about best practice in interpreting and using SRT data arose during a successful two-year revision process (2007-2009) of an SRT form at a medium-sized comprehensive university. The initial goals of the revision were to:

1. Produce a pedagogically neutral form

2. Capture important dimensions of teaching (Feldman, 2007) that students are capable of assessing

3. Have all items meet criteria for survey construction (Berk, 2006).

A few words may be in order regarding the somewhat unusual first goal. The "old" form was perceived by many faculty on campus as a hindrance to pedagogical innovation because it contained several items worded in such a way as to privilege the lecture method. For example, The professor's presentations facilitated learning of the course content. These items were seen as disadvantaging faculty employing seminar-type discussion, cooperative learning in small groups, service learning, or performance-oriented approaches in the arts. As McKeachie (1997) noted: "Most student rating forms . . . focus almost completely on conventional classroom teaching" (p. 1220).

Through a review of the literature the faculty on the committee charged with revising the SRT form became aware that they themselves were not fully knowledgeable about the limitations of SRT data. One former department chair on the committee confessed her inclination to give more weight to written comments than to the numerical data. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.