The Department of Education's expert panel, assembled to find the
best math programs in the United States, had a key problem, critics
say. It relied heavily on studies of student achievement that were
authored or co-authored by the directors of the programs themselves -
or by people with close institutional or other ties to the program.
In the cases of Core-Plus Mathematics Project and the Connected
Mathematics Project, a middle-school program, studies showing
positive student achievement were submitted to the Department of
Education's expert panel.
But neither Core-Plus nor Connected Math has yet published in a
peer-reviewed journal the findings from the field-tests of their
programs - though these were the primary studies supplied to the
expert panel as proof the programs work.
"Peer review is essential," says Ronald Green, director of the
Institute for the Study of Applied and Professional Ethics at
Dartmouth College in Hanover, N.H. "To be independent, an
evaluator's judgment must not be distorted by institutional
relationships or personal ties or financial considerations."
Core-Plus was one of the best of the programs reviewed, panel
members say. But studies of its effectiveness were co-authored by
Harold Schoen, a University of Iowa professor.
Dr. Schoen, who is listed as a co-director of the program, admits
he is in line to receive royalties from the sales of Core-Plus
textbooks. His studies, he says, are not motivated by the prospect
of royalties, of which he has received little.
But some critics have concerns. "You simply cannot have one of
your principal investigators [in a research project] also be the
outside evaluator," says R. James Milgram, a Stanford University
mathematician and critic.
Nobody, including research ethicists, argue that Schoen's studies
are invalid. As a co-director of the program, Schoen's studies
provide a valuable basis for analysis. However, experts say, his co-
authorship and receipt of royalties mean his reports should require
independent peer review.
Still, Schoen and Steven Leinwand, the co-chair of the expert
panel, both contend that the ultimate peer review for the winning
programs was the education panel's process itself.
But outsiders say this was not a true peer review. For instance,
the studies and programs were not anonymous to reviewers, thus
opening the door to bias, says Thomas Loveless, senior analyst at
the Brookings Institution in Washington.
Of 61 programs evaluated, 25 survived the quality-review process.
Then, nine "impact reviewers," professional evaluators, examined
supporting studies of student achievement for those programs.
Their summaries were then relied upon by the expert panel as
evidence of a program's effectiveness. It was a last, critical step
Copies of the impact reviews for all 10 programs were obtained by
the Monitor. …