Growing up in California's public schools, I took more
standardized tests than I can remember: Teachers at every grade
level stressed their importance. I didn't want to let anyone down,
so I approached each test with all the solemnity and effort a child
I never questioned that obedience. As a child, I imagined my test
answers being flown across the country to a room of educated,
professional test scorers who possessed a zeal for essays written on
such topics as "A moment that changed my life."
My summer as a test-scorer disabused me of that notion. As a
recent college graduate, I worked in a Boston testing company, and
instead of the professionals I'd envisioned painstakingly grading
exams, I found a room full of temporary employees who had little
respect for - and minimal investment in - their jobs.
It was my first assignment after registering with a temporary-
employment agency in June. For a fee, the grading company they
placed me with scores exams and summarizes the results. My job was
to score the essay portion of a test taken last spring by eighth-
grade students all across the United States.
Before I began working, I attended a three-day training seminar
during which I studied a scoring rubric, learned a numerical
scoring system, and read hundreds of sample essays graded by
Several other temps and I read the same essays, scored them, and
compared our results. Often there were wide disparities between our
scores, and it surprised me that these differences decreased only
slightly as the training progressed. It surprised me even more to
learn that those disparities were acceptable. We were told - by
trainers who were themselves temps - that our scores need be only
within one number of the standard on a five-point scale. This meant
that if an essay "should" have gotten a score of three, as long as
we gave that student a two or a four, we were close enough.
All of the scorers - even all the supervisors - were temps, some
with poor English skills, and many without a college degree.
Perhaps the grading system seemed subjective because most of us had
no background in education, were minimally trained, and therefore
weren't well qualified to evaluate standardized tests.
It's no wonder our scores differed to such an alarming extent.
The turnover rate was high, because our compensation was low and
many people left after finding higher-paying jobs. In addition,
many of my fellow scorers were young college students, and some of
them scored essays written by high school students only a few years
Some scorers had been there for months before I arrived, and the
work environment lacked any sense of purpose or professionalism. …