Flaws in the Evaluation Process
Clayton, Mark, The Christian Science Monitor
The Department of Education's expert panel, assembled to find the best math programs in the United States, had a key problem, critics say. It relied heavily on studies of student achievement that were authored or co-authored by the directors of the programs themselves - or by people with close institutional or other ties to the program.
In the cases of Core-Plus Mathematics Project and the Connected Mathematics Project, a middle-school program, studies showing positive student achievement were submitted to the Department of Education's expert panel.
But neither Core-Plus nor Connected Math has yet published in a peer-reviewed journal the findings from the field-tests of their programs - though these were the primary studies supplied to the expert panel as proof the programs work.
"Peer review is essential," says Ronald Green, director of the Institute for the Study of Applied and Professional Ethics at Dartmouth College in Hanover, N.H. "To be independent, an evaluator's judgment must not be distorted by institutional relationships or personal ties or financial considerations."
Core-Plus was one of the best of the programs reviewed, panel members say. But studies of its effectiveness were co-authored by Harold Schoen, a University of Iowa professor.
Dr. Schoen, who is listed as a co-director of the program, admits he is in line to receive royalties from the sales of Core-Plus textbooks. His studies, he says, are not motivated by the prospect of royalties, of which he has received little.
But some critics have concerns. "You simply cannot have one of your principal investigators [in a research project] also be the outside evaluator," says R. James Milgram, a Stanford University mathematician and critic.
Nobody, including research ethicists, argue that Schoen's studies are invalid. As a co-director of the program, Schoen's studies provide a valuable basis for analysis. However, experts say, his co- authorship and receipt of royalties mean his reports should require independent peer review.
Still, Schoen and Steven Leinwand, the co-chair of the expert panel, both contend that the ultimate peer review for the winning programs was the education panel's process itself.
But outsiders say this was not a true peer review. For instance, the studies and programs were not anonymous to reviewers, thus opening the door to bias, says Thomas Loveless, senior analyst at the Brookings Institution in Washington.
Of 61 programs evaluated, 25 survived the quality-review process. Then, nine "impact reviewers," professional evaluators, examined supporting studies of student achievement for those programs.
Their summaries were then relied upon by the expert panel as evidence of a program's effectiveness. It was a last, critical step before voting.
Copies of the impact reviews for all 10 programs were obtained by the Monitor. …