Statistical Analysis for Field Experiments
and Longitudinal Data in Writing Research
Robert D. Abbott, Dagmar Amtmann, and Jeff Munson
In this chapter we begin with a focus on the strengths of randomization and experiments in writing research, and discuss statistical implications for power and type I error rates when scaling laboratory or small-scale research up to field experiments. Because statistical methods for exploratory and confirmatory analysis of cross-sectional data are well known, we then focus on statistical analysis and the design of newer longitudinal approaches that are useful in research on growth in writing processes. These longitudinal methods include latent variable growth mixture modeling for change when measures are continuous and latent transition analysis (LTA) for change when measures of change are stages or categories. We have minimized the number of equations in the text, instead pointing readers to the relevant sources in the statistical literature. In many ways, this chapter is a complement to Abbott, Amtmann, and Munson (2003), where the reader can find discussion of graphical and quantitative (e.g., factor analysis) exploratory methods of analysis for cross-sectional data, confirmatory methods for measured and latent variable structural equation modeling of cross-sectional data, and new directions in confirmatory methods, including permutation-based statistics and graph– theoretic approaches to causal modeling. Consequently, we make only brief reference to these methods in this chapter.
Researchers investigating writing processes have used a variety of qualitative and quantitative methods (Allal, Chanquoy, & Largy, 2004; Beach & Bridwell, 1984; Kamil, Langer, & Shanahan, 1985; Rijlaarsdam, van den Bergh, & Couzijn, 1996). The complementary nature of qualitative and quantitative methods is well illustrated by the work of MacArthur, Graham, and Harris (2004), whose program of research on revising draws on theory about writing processes, descriptive interview studies, and randomized experimental studies to tease apart the multiple processes involved in revising.
While complementary and compatible within a broader epistemological framework (MacArthur, 2003), applications of qualitative and quantitative methods in writing research need to be evaluated based upon the standards of construct validity, internal validity, statistical conclusion validity, and external validity (Levin & O'Donnell, 1999; Mosteller & Boruch, 2002; Shadish, Cook, & Campbell, 2002) or their translation into scientifically based research standards (Eisenhart & Towne, 2003). As research on effective strategies in teaching writing is extended into experiments in schools and classrooms, researchers need to continue to focus