Who's Scoring Those High-Stakes Tests? Poorly Trained Temps
Fortner, Cameron, The Christian Science Monitor
Growing up in California's public schools, I took more standardized tests than I can remember: Teachers at every grade level stressed their importance. I didn't want to let anyone down, so I approached each test with all the solemnity and effort a child can muster.
I never questioned that obedience. As a child, I imagined my test answers being flown across the country to a room of educated, professional test scorers who possessed a zeal for essays written on such topics as "A moment that changed my life."
My summer as a test-scorer disabused me of that notion. As a recent college graduate, I worked in a Boston testing company, and instead of the professionals I'd envisioned painstakingly grading exams, I found a room full of temporary employees who had little respect for - and minimal investment in - their jobs.
It was my first assignment after registering with a temporary- employment agency in June. For a fee, the grading company they placed me with scores exams and summarizes the results. My job was to score the essay portion of a test taken last spring by eighth- grade students all across the United States.
Before I began working, I attended a three-day training seminar during which I studied a scoring rubric, learned a numerical scoring system, and read hundreds of sample essays graded by experts.
Several other temps and I read the same essays, scored them, and compared our results. Often there were wide disparities between our scores, and it surprised me that these differences decreased only slightly as the training progressed. It surprised me even more to learn that those disparities were acceptable. We were told - by trainers who were themselves temps - that our scores need be only within one number of the standard on a five-point scale. This meant that if an essay "should" have gotten a score of three, as long as we gave that student a two or a four, we were close enough.
All of the scorers - even all the supervisors - were temps, some with poor English skills, and many without a college degree. Perhaps the grading system seemed subjective because most of us had no background in education, were minimally trained, and therefore weren't well qualified to evaluate standardized tests.
It's no wonder our scores differed to such an alarming extent. The turnover rate was high, because our compensation was low and many people left after finding higher-paying jobs. In addition, many of my fellow scorers were young college students, and some of them scored essays written by high school students only a few years their junior.
Some scorers had been there for months before I arrived, and the work environment lacked any sense of purpose or professionalism. …