Material Fallacies of Education Research Evidence and Public Policy Advice
O'Neill, John, New Zealand Journal of Educational Studies
Rational debates about education policy choices are only possible when all those involved use language and evidence correctly. Material fallacies in logic are "mistakes in understanding the meaning or use of terms" (Kreeft, 2010, p. 85). Just as in practical logic, public policy discourse becomes problematic when the terms used are ambiguous, unclear or vague. This commentary analyses one recent case of material fallacy in education policy discourse in order to promote discussion of the responsibilities researchers may have whenever research evidence is used mistakenly.
Treasury's briefing to the incoming Minister of Finance after the 2011 election included a controversial assertion about school funding policy:
Student achievement can be raised by improving the quality of teaching, which the evidence shows is the largest inschool influence on student outcomes. Increasing student/ teacher ratios, and consolidation of the school network, can free up funding that could be used to support initiatives to enhance the quality of teaching, such as more systemic use of value-add data and a more professionalised workforce. (Treasury, 2011, p. 21)
In reply to a Parliamentary question about this briefing, the Minister of Education declined to rule out increases in class size. In short, this was because the "independent observation" of Treasury and the research findings of an influential government adviser, Professor John Hattie, were that schooling policy should instead focus on improving the quality of teaching (House of Representatives, 2012).
Government ministers have many portfolio responsibilities and rely largely on officials and advisers to provide accurate policy advice. At first glance, the Minister's reply appeared to draw logically on such advice: student/ teacher ratios do not have a large influence on student outcomes; the quality of teaching does have a large influence on student outcomes; therefore in order to produce a larger effect on student outcomes, school policy and funding should prioritise the quality of teaching over student/ teacher ratios.
Closer analysis reveals errors in both argument and evidence. Much of the terminology is ambiguous and inconsistently used by politicians, officials and academic advisers. The propositions are not demonstrably true- indeed, there is evidence to suggest they are false in crucial respects. The conclusion is, at best, uncertain because it does not take into account confounding evidence that larger classes do adversely affect teaching, learning and student achievement, particularly the achievement of already disadvantaged students such as those prioritised by the Ministry of Education (2011) throughout its current school reform agenda.
Precisely because material fallacies are written or spoken by persons, these limitations give cause to reflect more broadly on the practices, motivations and subject positionings of the various policy actors (politicians, officials and academics) within the contemporary discourse of schooling reform in Aotearoa New Zealand. It has been observed, for example, that, in the English context, the equivalent discourse seeks to portray the public sector as "ineffective, unresponsive, sloppy, risk-averse and innovation- resistant" yet at the same time it promotes "celebration of public sector 'heroes' of reform and new kinds of public sector 'excellence'" (Ball, 2007, p. 3). Relatedly, Mintrom (2000) has written persuasively, in the American context, of the way in which "policy entrepreneurs" position themselves politically to champion, shape and benefit from school reform discourses.
The case discussed in this commentary provides a timely opportunity to ask whether and to what extent New Zealand may too exhibit "a confusing interplay of trust/ distrust" (Ball, 2007, p. 3) in the education policy process. It also encourages a more systematic, disinterested analysis of the quality of evidence and advice, and how these are used tactically by various policy actors to facilitate a reformist ideology of public schooling. …