Newspaper article International Herald Tribune

Algorithms That Can Cope with Essays

Newspaper article International Herald Tribune

Algorithms That Can Cope with Essays

Article excerpt

A competition to see how well algorithms could predict students' test scores assigned by human graders showed that the predictive algorithms were eerily accurate.

As a professor and a parent, I have long dreamed of finding a software program that helps every student learn to write well. It would serve as a kind of tireless instructor, flagging grammatical, punctuation or word-use problems and also showing the way to greater concision and clarity.

Now, unexpectedly, the desire to make the grading of tests less labor-intensive may be moving my dream closer to reality.

The standardized tests administered by the states at the end of the school year typically have essay-writing components, meaning humans have to be hired to grade them. This spring, the William and Flora Hewlett Foundation sponsored a competition to see how well algorithms submitted by professional data scientists and amateur statistics wizards could predict the scores assigned by human graders. Winners were announced last month -- and the predictive algorithms were eerily accurate.

The host for the competition was Kaggle, a Web site that runs predictive-modeling contests for client organizations -- thus giving them the benefit of a global crowd of data scientists working on their behalf. The site says it "has never failed to outperform a pre- existing accuracy benchmark, and to do so resoundingly."

Kaggle's tagline is "We're making data science a sport." Some of its clients offer sizable prizes in exchange for the intellectual property used in the winning models. For example, the Heritage Health Prize ("Identify patients who will be admitted to a hospital within the next year, using historical claims data") will bestow $3 million on the team that develops the best algorithm.

The essay-scoring competition that just concluded offered a mere $60,000 as a first prize, but it drew 159 teams. At the same time, the Hewlett Foundation sponsored a study of automated essay-scoring engines now offered by commercial vendors. The researchers found that these produced scores effectively identical to those of human graders.

Barbara Chow, education program director at the Hewlett Foundation, said: "We had heard the claim that the machine algorithms are as good as human graders, but we wanted to create a neutral and fair platform to assess the various claims of the vendors. It turns out the claims are not hype. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.