A Methodological Review of Computer Science Education Research
Randolph, Justus, Sutinen, Erkki, Julnes, George, Lehman, Steve, Journal of Information Technology Education
One of the most influential books on computer science education research (Fincher & Petre, 2004) begins with the following statement: "Computer science education research is an emergent area and is still giving rise to a literature" (p. 1). Top computer science education researchers like Mark Guzdial and Vicki Almstrum argue that the interdisciplinary gap between computer science education and educational research proper, including methods developed in the broader field of behavioral research, must be overcome before computer science education research can be considered to be a field which has emerged (Almstrum, Hazzan, Guzdial, & Petre, 2005). (In this methodological review, we use the term behavioral research as a synonym for what Guzdzial, in Almstrum et al. (2005, p. 192), calls "education, cognitive science, and learning sciences research.") Addressing this lack of connection with behavioral research, Guzdial, in Almstrum and colleagues (2005) wrote,
The real challenge in computer education is to avoid the temptation to re-invent the wheel. Computers are a revolutionary human invention, so we might think that teaching and learning about computers requires a new kind of education. That's completely false: The basic mechanism of human learning hasn't changed in the last 50 years. Too much of the research in computing education ignores the hundreds of years of education, cognitive science, and learning sciences research that have gone before us. (pp. 191-192)
One way to bridge the gap between computer science education research and behavioral research proper is to review current methodological practices in computer science education and to compare those practices with existing knowledge about best practices from the field of behavioral research. In this article, we do just that; we review the current methodological practices in computer science education and present recommendations, from a behavioral science standpoint, on how computer science education research might be improved. It is our hope that our results and recommendations will improve practice and inform policy about and, ultimately, help computer science education research transition from an emerging research area to an area that has already emerged. (Here we use Denning et al.'s (1989, p. 12) definition of the discipline of computing: "the systematic study of algorithmic processes that describe and transform information: their theory, analysis, design, efficiency, implementation, and application.")
Several groups stand to gain from this review. The creators of computer science education research will benefit from knowledge of how their peers tend to conduct research--such as what variables they investigate, how they tend to measure those variables, and how they analyze and present their data--and from getting suggestions on how to improve their own research. The consumers of computer science education research, such as funders, practitioners, and educational administrators, will become more aware of the strengths and weakness of the current computer science education research and can use that knowledge to make decisions about policy or practice. Finally, the gatekeepers of computer science education, such as the funders, editors, and reviewers of computer science education research, are especially important stakeholders in this review because they set the standard for what types of research activities are acceptable and what types of reports merit publication.
The next section begins with a discussion of three pre-existing reviews of the computer science education research and a rationale for the need for the current research. In the remaining sections of this paper, after a short discussion of biases, the reader will find a description of the methods used, including a description of the coding book development, the sampling strategy and sampling frame, interrater training procedures, and data analyses. …