Challenges in Using Outcome Data to Evaluate School Leadership
This chapter discusses challenges associated with evaluating efforts to improve school leadership. The statistical models in the New Leaders evaluation enable us to estimate the overall effect of the New Leaders program on selected student outcomes, independent of the effect of a set of control variables. It is this overall, aggregate estimate that informs whether the program is having an effect on the student outcomes.
The available outcome measures for the evaluation of efforts to improve school leadership typically include students’ scores on state or district assessments, along with other student-level information, such as whether a student graduated or progressed to the next grade. Student outcome data are critical to understanding how well an effort is working to improve the principalship, but they have a number of limitations. Below we discuss six broad issues that many evaluations are likely to encounter.
Challenge: When evaluating efforts to improve school leadership, there are often inconsistencies in the availability of outcome measures across states, districts, and CMOs. Even within a district, changing reporting needs and inadequate record-keeping can lead to differences in the availability of outcome measures from year to year. Depending on whether charter school data are reported separately or by the district, charter schools may track different outcomes than the district with which they are affiliated. These inconsistencies make it challenging to accurately evaluate the improvement effort, as the same outcomes are needed to compare results between years and across districts, states, and CMOs. Such inconsistency may require the estimation of separate program effect measures for different districts, states, or types of schools. This limits the interpretation of results derived from each type of estimation, as the results may vary greatly and may not be generalizable beyond the specific district, state, or school type.
For example, for the New Leaders evaluation, some districts provided us with detailed attendance data that included days attended and days enrolled at each school that a student attended throughout a given school year. These detailed data allowed us to create an average attendance rate using information from all schools that a student attended. Other districts initially provided attendance data that only included information from the school at which the