I was walking along the road towards Athens with Admetus one day last summer. We were talking of this and that when we ran into Socrates. He was just coming out of the gymnasium where he had been working out and watching the younger men wrestle. "Come with us," we said, "for indeed, we won't take no for an answer." He readily agreed to walk with us. "What shall we discuss today?" he asked. Just then a pedagogue walked by with his young charge on their way to school. Admetus quickly spoke, "Let's speak of education, school reform, and No Child Left Behind."
SOC: Is this an important issue for you?
AD: Yes, I believe that an educated populace is important for a democracy to survive. In addition, economic competition requires that all of our young have a world-class education.
SOC: I also believe that education is very important. Indeed, I have been thinking about education lately. I wonder if I could ask you some questions that we might arrive at a clearer understanding together of these important matters.
AD: That is just what I had in mind.
SOC: Let us for the moment assume that education needs reform, although there is much that could be argued here either way, so that we might address ways of bringing reform about.
AD: Yes, that makes sense.
SOC: Tell me what you believe to be the main lack that schools have today.
AD: It seems to me that schools and teachers lack a quality knowledge base of practices which they could implement confidently knowing they are effective in producing student learning. The model I have in mind is the medical field: doctors have a wide variety of medicines and other interventions which they can dispense, knowing that their effectiveness and safety have been verified in large-scale clinical trials.
SOC: So you believe that randomized trials are the only sure method for determining the effectiveness of education programs and practices?
AD: Yes, I do, because of the relatively strong internal validity of a true experiment in connecting the researcher's hypothesis and result. Because the experimenter is intervening, actually manipulating the independent variable, and because the different groups involved start at the same point (within the limits of random variation) in every variable, important or not, many of the classic ways to go "wrong"--that is, to have viable alternative hypotheses (also called threats to validity)--are controlled. For example, in a correlational study (not an experiment), we find that teacher salary is correlated with student performance on the ITBS. We might conclude that raising teacher salary will increase student performance. More likely, however, a "third variable" is at work; teachers are paid more in higher SES districts, usually located in more homogeneous suburbs, and students from these type of districts also tend to core better on standardized achievement tests. Although teachers would love higher salaries, there is not necessarily a cause-and-effect relationship here, and it is unlikely that, in the absence of anything else being done, that student performance would rise.
SOC: I understand that argument, and it does have value. I found myself using it just the other day. My wife was telling me about a study which found that music students who practiced within 24 hours of their lesson retained most of the lesson; those who waited until between 24 and 48 hours about 40%, and more than 48 hours might as well never had the lesson. I immediately asked if this was a controlled experiment or if it relied on self-selection, as those who practice immediately are probably those who are more interested in learning to play as well.
However, I wonder if you have considered some of the limitations of true experiments. For example, not all variables can be manipulated feasibly or ethically. Gender is a classic example. Cigarette smoking is another. In addition, real life in the schools does not always allow random assignment. …