On the Treatment of Authors, Outliers, and Purchasing Power Parity Exchange Rates
Jaeger, Richard M., Phi Delta Kappan
IN HER RESPONSE to my October article, Ruth Stotts indulges in and ad hominem attack that lacks cohesive logic. Within her 2 1/2 pages of typescript, Stotts accuses me of "violating [my] own methodology," of submitting an article that is "replete with factual errors," of harboning "prejudiced opinions," of distoring information for the purpose of supporting my opinions, of misrepresenting information, and of "violating the ethics of [my] profession." What happened to the canons of schoolarly debate?
What evidence or argument does Stotts offer in support of these accusation? First, she advances an alleged "error in methodology" resulting from my interpretation of a finding that average class size predicted just over 10% of the variation in 13-year-old' mean scores in mathematics in the 14 nations that "supplied reasonably comprehensive sampling frames for the 1991 IAEP study." I conclude that, "despite these results, it is difficult to believe that larger class size is generally associated with higher average student achievement. Perhaps a larger sample of nations would reveal a total absence of relationship."
It is important to note several facts. First, the methodology used was simple linear regression analysis. It was applied correctly, and the findings were accurately and factually reported.
Second, the same methodology was used in investigating the prediction of math achievement in the 1991 IAEP study, using average number of minutes of math instruction per week as the predictor variable, as well as the predictive value of a number of other variables. The method used was clearly described.
Third, both of the regression analyses cited by Stotts left more than 90% of the between-nation variation in 13-year-olds' average math achievement unexplained, a fact clearly noted in my article. The slopes of these regression lines did not differ significantly from zero when tested using a Type I error probability of .05. Thus the data at hand fully warrant the conclusion that the slope of the regression of 13-year-olds' math achievement on class size is equal to zero in the population of nations from which the sample of 14 nations might have been drawn.
Fourth, because of space limitations, only two of 28 figures contained in my original manuscript were included in the published Kappan article. Had my original Figures 14 and 15 (presented herewith as Figures 1 and 2) been printed, they would have shown clearly that Korea and Taiwan were outliers in the relationship between average class size and 13-year-olds' average achievement on the 1991 IAEP in mathematics - so much so that it seems reasonable to conclude that the positive regression line found in the analysis was not likely to generalize beyond the sample of 14 nations for which data on these two variables were obtained. Indeed, in the absence of Korea and Taiwan, the slope of the regression line is negative.
The screening of outliers is recommended practice in regression analysis. Note, in contrast to Figure 1, that in Figure 2 there are no outlying data values that would materially alter the slope of the regression line were they to be omitted from the data set.
Finally, a substantial research literature exists that supports the inverse relationship between average class size and student achievement. Although none of these studies used nations as its unit of analysis and the within-nation results often differ from between-nation results, it is not unreasonable to expect the regression of student achievement on class size to have a negative slope in light of the more than 75 studies summarized in this body of research. Having been a reviewer of Gene Glass' meta-analysis prior to its publication, my personal expectation, clearly stated as just that in my October Kappan article, is well justified. In sum, the first objection Stotts raises provides no support for her conclusions or for her attributions of bad faith on my part. …