Many Schools, One Complex Measure
Kedro, M. James, Short, William E., Journal of Staff Development
St. Louis district synthesizes the lessons of reform
Measuring the extent of professional development in a large school system is complex. It's a challenge to gauge training and its effects when schools choose different instructional reform models, and the task is more formidable when the models are chosen at different times and change.
In 1999 and 2000, the St. Louis Public Schools identified schools that didn't do well on the state's highstakes performance test and selected 40 to receive about $55,000 a year in extra funds for instructional innovations. The goal for the schools, identified as Schools of Opportunity (SOP), was clear: meet state standards and earn full accreditation.
The schools' staffs selected models from a menu of research-based programs on the list of Comprehensive School Reforms (American Institutes for Research, 1999; NWREL, 2001). The extra money they received was budgeted for teacher professional development and instructional materials. A dozen different instructional models were off and running.
The district decided it needed to evaluate the overall project to find out whether the schools were achieving the desired results. Was professional development adequate? Did training in the models make its way into classrooms? Did the instructional changes help - did student achievement improve?
DESIGNING A STUDY
To assess the professional development in instructional models, researchers in the St. Louis district asked: (1) How many teachers at the elementary, middle, and high school levels believe they were adequately trained in the models? (2) How does the length of time with a model affect the teachers' perception of the training and use of the model? (3) How appropriate did teachers think the model was for their students? (4) Does length of time with the model, training, and teachers' perceptions of its use and acceptance relate to improvements in student achievement?
For three consecutive years (2000-02), the district's research division surveyed all instructional staff, including site administrators, special needs teachers, and ancillary teachers in the schools that implemented reform models. The district eliminated the survey in 2003 due to budget constraints, but researchers continued with school observations and analyzing student achievement.
In the first year, we used a standardized, validated instrument, the Concerns-Based Adoption Model (CBAM) questionnaire, to measure to what degree teachers accepted the new programs (Hall & George, 1979; Hall, Wallace, & Dossett, 1973). We repeated the CBAM questionnaire in the second and third years, along with a questionnaire the district developed as a cross-check on the CBAM results.
Items on the staff-developed questionnaire pertained specifically to teacher training in the instructional models (Parsad, Lewis, & Farris, 2001). The survey asked teachers to rate their level of use of whichever new model their school had adopted, choosing one of five levels: nonuser, novice, intermediate, old hand, and past user.
In 2001, 36 schools returned the survey, for a response rate of 90%, compared with 25 schools (62.5%) in 2002. The 2001 surveys had been sent out in February, while the 2002 surveys were sent out near the end of the year, which likely lessened the response rate.
We then compared the data from the staff-developed survey over time. We looked at the responses for each level - elementary, middle, and secondary - the number of respondents (552 in 2002; 715 in 2001), and how representative the respondents were of all the participating schools and their selected models. We analyzed the data for schools that had used their model for three consecutive years to determine what teachers thought of the extent of their professional development and their perception of the degree to which the models were being used (see chart below).
What we found was:
* More teachers seemed to have difficulty adapting to change in the second year and fell back on traditional strategies despite the reform initiative training. …