Academic journal article Canadian Psychology

The New Caucus-Race: Methodological Considerations for Meta-Analyses of Psychotherapy Outcome

Academic journal article Canadian Psychology

The New Caucus-Race: Methodological Considerations for Meta-Analyses of Psychotherapy Outcome

Article excerpt

In Alice's Adventures in Wonderland, the Dodo bird adjudicates a "Caucus-race": a contest in which contestants begin at different starting points, run in different directions, and end whenever they please (Carroll, 1865/2008, Chap. 3). The Dodo concludes the race and declares: "Everybody has won, and all must have prizes" (Carroll, 1865/2008, Chap. 3, para. 20). The Dodo Bird Verdict (DBV) has been used to describe a particular conclusion about psychotherapy outcomes-namely, that all types of psychotherapy are equivalently effective (i.e., that all therapies have "won" and all of them deserve "prizes"; Luborsky, Singer, & Luborsky, 1975; Rosenzweig, 1936). Ironically, the DBV has acted as a starting gun for a new Caucus-race, with researchers going in different directions to find evidence in support of, or against, the verdict.

The primary strategy used by the runners of the new Caucusrace has been meta-analysis, with effect sizes from a number of relevant primary treatment studies collected, weighted by their precision, and then aggregated to produce a weighted average effect size. This strategy allows for a much more precise estimate of the overall effect than could be obtained by any one primary study, due to the increase in sample size and statistical power that arises from aggregation. Because most primary studies in the psychological literature have low power to detect significant effects, this also yields a clearer overview of the literature than could be obtained by counting the number of studies with and without significant results (e.g., Borenstein, 2000).

Meta-analyses, however, are limited by the both the nature of the primary studies included in the analyses and the manner in which the analyses are conducted. A high-quality meta-analysis should have a cogent justification for the studies included and excluded, and appropriate analyses should be used to combine data from included studies, evaluate possible biases in the data set, and evaluate variables that may moderate the overall findings. However, meta-analyses on the DBV have historically been run as races with different starting points (i.e., variability in which studies are included), in different directions (i.e., variability in analytic strategies), and with arbitrary finish lines (i.e., variability in how the resulting mean effect sizes are interpreted)-much like the Dodo's Caucus-race. This has resulted in competing claims that all psychotherapies are essentially equivalent (e.g., Wampold & Imel, 2015), that certain therapies provide superior outcomes for some clinical conditions (e.g., Marcus, O'Connell, Norris, & Sawaqdeh, 2014), and that there may be insufficient evidence to decide one way or the other (e.g., Hunot, Churchill, Teixeira, & Silva de Lima, 2007).

In this article we review critical methodological issues that have arisen in the context of using meta-analysis to evaluate the DBV. These issues, we contend, have interfered with researchers' ability to draw clear conclusions from the results of their statistical analyses and continue to impede attempts by mental health professionals to use the results of these meta-analytic studies to guide their clinical training and clinical service activities. The range of issues we address include questions of (a) how to aggregate effect sizes, (b) how to determine which outcomes from primary studies should be included in the analyses, (c) whether to conduct direct or indirect comparisons between treatments, (d) how best to conduct moderator analyses, and (e) how to provide contextually appropriate interpretations of effect sizes obtained in the meta-analysis. We have two goals in reviewing these issues. First, we identify key issues that contribute to the DBV literature's methodological heterogeneity and that underlie some of the ongoing disagreements about the appropriate use of meta-analysis in evaluating the DBV. Second, based on our analysis of these issues, we recommend options for ensuring that meta-analytic investigations address the DBV in ways that are both methodologically sound and statistically appropriate and that allow for the DBV hypothesis to be tested fairly. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.