Academic journal article Journal of Instructional Psychology
Moderating Meta-Analysis: "Meta-Analytic Homogeniety"
The authors over the years have expressed criticism about meta-analysis. However, it could not be more popular. Recently a major news station reported the use of meta-analysis in a new finding.
To try to reduce the errors in this statistical and methodological strategy, we have suggested that all data be nominalized and use chi-square as a goodness of fit study. (Snell & Marsh: 2003) We thought that perhaps allowing for all the heterogeneity built into the methodical strategy that this may be more defensible. However, we retracted this position when more studies using numerous samples, methods, and analysis were compiled into meta-analysis. (Snell& Marsh: 2009) (Begley: 2008)
Further, Wikipedia and other sources have many supporters and statistical strategies to homogenize the research strategy. There are now numerous books on the strategy, software, and related analysis that appear to make it more valid and reliable. There are literally thousands of articles.
Our position is that this approach has metasized through out numerous disciplines like a virulent cancer. Thus, we suggest that the following be used by editors in journals across the academic fields.
1. Has the author used "best evidence meta-analysis" or has claimed to use this. That means weaker studies are deleted.
2. Have they attempted to deal with the "file drawer effect" (only published studies are used).... Simpson paradox" (two studies have opposing results, but when combined show one direction) and "coding bias." (anecdotal, convenience samples, and related are combined together.)
3. Acknowledge the controversial status of the research methodology and that other studies have shown a difference of up to 35% in outcomes. (Ioannidis, J. et. al. 1998) when meta-analysis is compared to randomized controlled trials.
4. Have they clearly stated the hypothesis? Used only studies where the test-retest or cross sectional methods are same or similar? Thus a study on clinical depression uses the same index or scale. As an example Hamilton Depression is used in all studies compiled as opposed to Langer's Psychiatric Screen or Sroles Anomie Scale or numerous other measures. …