After all, the problems of nonrandomized field studies go back as far as Student's (1931) discussion of the Lanarkshire nutrition experiment (Chapter 3, Appendix, pages 81–82).
Third, field studies are important and randomization is not always possible. The emphasis on randomized experiments is not intended to slight the potential value of quasi-experiments. The foregoing nursing home study seems useful, even though the results are less trustworthy than if some form of randomized design had been used.
Too often, however, field studies are uninterpretable, as with much of the Head Start program (see similarly Premarital Cohabitation and Low Lead Controversy in Section 16.2.2). Adequate technical preparation is essential to assess whether a given field study will be meaningful and/or to embed meaningful field studies within socially mandated programs.
Kirk (1995, Chapter 11) considers hierarchical designs similar to the natural groups design considered in Section 15.1.2 (following page); see also Bogartz (1994, Chapter 6), Lindman (1992, Chapter 8), Myers and Well (1991, Chapter 10), and Winer et al. (1991, Sections 5.14 and 6.6).
Establishing that natural group differences are small in some line of investigation would be desirable as it would allow pooling of SS G/A and SS S/G. The only way to do this is through empirical assessment over a sequence of studies within a common experimental setting. To build up a knowledge base for pooling decisions, mean squares for all sources deserve to be reported. Even more important, it would seem, is development of procedures to reduce differences between groups, as with screening tests and clean procedure.
Error terms for more complex mixed models, with both fixed and random terms, can be tricky, especially for confidence intervals. Schwarz (1993) commented in adverse detail on current texts and computer packages, and his article should be understood by anyone doing serious work with random factors. As one example, he says “Only an experienced [statistical] analyst should attempt the analysis of a mixed model using SPSS [Version 4.0]” (p. 56).
The relevance of random factor analysis to language studies was pointed out by Coleman (1964) and amplified by Clark (1973), who used an example of comparing reading speed for nouns versus verbs, also adopted by Wickens and Keppel (1983). This example does not seem too pertinent because nouns and verbs are heterogeneous classes