Methodological and Conceptual Challenges
in Conducting and Interpreting
Michael J. Burke Ronald S. Landis
Meta-analysis has proven to be a powerful tool for researchers from a variety of social science disciplines interested in testing whether bivariate relationships or effects generalize across conditions or settings. In addition to examining such relatively simple relations, researchers are beginning to use meta-analysis to address broader research questions and hypotheses including the use of metaanalytic results as input to structural equation models. Because the ultimate goal of most research is to develop an understanding of the associations among constructs, meta-analysis represents a powerful technique for explicating nomological networks. The use of meta-analytic procedures to test broader research hypotheses than posited in primary studies as well as the diverse applications of meta-analytic procedures present new methodological challenges and construct validity issues.
The purpose of this chapter is to discuss five important issues that can affect the use of meta-analytic procedures in making inferences about relations between constructs or the effectiveness of behavioral interventions, and to offer suggestions for dealing with these issues. More specifically, this chapter presents discussions of three topics that are likely to affect the conduct of meta-analyses: the estimation of effect size in studies that use repeated measures designs, the use of hypothetical artifact distributions versus sample-based artifacts when making corrections for statistical artifacts,