Replication Is Not Coincidence: Reply to Iverson, Lee, and Wagenmakers (2009)

By Lecoutre, Bruno; Killeen, Peter R. | Psychonomic Bulletin & Review, April 2010 | Go to article overview

Replication Is Not Coincidence: Reply to Iverson, Lee, and Wagenmakers (2009)


Lecoutre, Bruno, Killeen, Peter R., Psychonomic Bulletin & Review


Iverson, Lee, and Wagenmakers (2009) claimed that Killeen's (2005) statistic p^sub rep^ overestimates the "true probability of replication." We show that Iverson et al. confused the probability of replication of an observed direction of effect with a probability of coincidence-the probability that two future experiments will return the same sign. The theoretical analysis is punctuated with a simulation of the predictions of p^sub rep^ for a realistic random effects world of representative parameters, when those are unknown a priori. We emphasize throughout that p^sub rep^ is intended to evaluate the probability of a replication outcome after observations, not to estimate a parameter. Hence, the usual conventional criteria (unbiasedness, minimum variance estimator) for judging estimators are not appropriate for probabilities such as p and p^sub rep^.

Iverson, Lee, and Wagenmakers (2009; hereafter, ILW) claimed that Killeen's (2005) p^sub rep^ "misestimates the true probability of replication" (p. 424). But it was never designed to estimate what they call the true probability of replication (the broken lines named "Truth" in their Figure 1). We clarify that by showing that their "true probability" for a fixed parameter δ-their scenario-is the probability that the effects of two future experiments will agree in sign, given knowledge of the parameter δ. We call this the probability of coincidence and show that its goals are different from those of p^sub rep^, the predictive probability that a future experiment will return the same sign as one already observed. ILW's "truth" has nothing to do with the "true probability of replication" in its most useful instantiation, the one proposed by Killeen (2005).

The "True Probability of Replication"

Statistical analysis of experimental results inevitably involves unknown parameters. Suppose that you have observed a positive standardized difference of d^sub obs^ = 0.30 between experimental and control group means having n = 10 subjects each.1 You assume the usual normal model with an unknown true effect size δ and (for simplification) a known variance. What is the probability of getting again a positive effect in a replication (d^sub rep^ > 0)? If you are ready to assume a particular value for δ, the answer is trivial: It follows from the sampling distribution of d^sub rep^, given this δ. The true probability of replication is the (sampling) probability [varphi]+|δ (a function of δ and n) that a normal variable with a mean of δ and a variance of 2/n exceeds 0: [varphi]+|δ = φ(δvn/2). If you hypothesize that δ is 0, then [varphi]+|0 = 0.5. Some other values, for different hypothesized δs, are [varphi]+|0.50 = 0.868, [varphi]+|1.00 = 0.987, [varphi]+|2.00 [asymptotically =] 1. These values do not depend on d^sub obs^: It would not matter that d^sub obs^ = 0.30 or d^sub obs^ = 1.30. Of course, for reasons of symmetry, [varphi]+|-δ = 1-[varphi]+|δ.

What was novel about Killeen's (2005) statistic prep was his attempt to move away from the assumption of knowledge of parameter values, and the "true replication probabilities" [varphi]+|δ that can be calculated if you know them. The Bayesian derivation of p^sub rep^ involves no knowledge about δ other than the effect size measured in the first experiment, d^sub obs^. This is made explicit by assuming an uninformative (uniform) prior before observations-hence, the associated posterior distribution for δ: a normal distribution centered on d^sub obs^ with a variance of 2/n. To illustrate the nature and purpose of p^sub rep^, consider the steps one must follow to simulate its value, starting with a known first observation:

Repeat the two following steps many times:

(1) generate a value δ from a normal(d^sub obs^,2/n) distribution;

(2) given this δ value, generate a value d^sub rep^ from a normal(δ,2/n);

and then compute the proportion of d^sub rep^ having the same sign as d^sub obs^. Each particular value of d^sub rep^ is the realization of a particular experiment assuming a true effect size δ, and corresponds to a "true probability of replication" [varphi]+|δ (if d^sub obs^ > 0) or 1-[varphi]+|δ (if d^sub obs^ , 0). …

The rest of this article is only available to active members of Questia

Already a member? Log in now.

Notes for this article

Add a new note
If you are trying to select text to create highlights or citations, remember that you must now click or tap on the first word, and then click or tap on the last word.
One moment ...
Default project is now your active project.
Project items

Items saved from this article

This article has been saved
Highlights (0)
Some of your highlights are legacy items.

Highlights saved before July 30, 2012 will not be displayed on their respective source pages.

You can easily re-create the highlights by opening the book page or article, selecting the text, and clicking “Highlight.”

Citations (0)
Some of your citations are legacy items.

Any citation created before July 30, 2012 will labeled as a “Cited page.” New citations will be saved as cited passages, pages or articles.

We also added the ability to view new citations from your projects or the book or article where you created them.

Notes (0)
Bookmarks (0)

You have no saved items from this article

Project items include:
  • Saved book/article
  • Highlights
  • Quotes/citations
  • Notes
  • Bookmarks
Notes
Cite this article

Cited article

Style
Citations are available only to our active members.
Buy instant access to cite pages or passages in MLA, APA and Chicago citation styles.

(Einhorn, 1992, p. 25)

(Einhorn 25)

1. Lois J. Einhorn, Abraham Lincoln, the Orator: Penetrating the Lincoln Legend (Westport, CT: Greenwood Press, 1992), 25, http://www.questia.com/read/27419298.

Cited article

Replication Is Not Coincidence: Reply to Iverson, Lee, and Wagenmakers (2009)
Settings

Settings

Typeface
Text size Smaller Larger Reset View mode
Search within

Search within this article

Look up

Look up a word

  • Dictionary
  • Thesaurus
Please submit a word or phrase above.
Print this page

Print this page

Why can't I print more than one page at a time?

Help
Full screen

matching results for page

    Questia reader help

    How to highlight and cite specific passages

    1. Click or tap the first word you want to select.
    2. Click or tap the last word you want to select, and you’ll see everything in between get selected.
    3. You’ll then get a menu of options like creating a highlight or a citation from that passage of text.

    OK, got it!

    Cited passage

    Style
    Citations are available only to our active members.
    Buy instant access to cite pages or passages in MLA, APA and Chicago citation styles.

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences." (Einhorn, 1992, p. 25).

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences." (Einhorn 25)

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences."1

    1. Lois J. Einhorn, Abraham Lincoln, the Orator: Penetrating the Lincoln Legend (Westport, CT: Greenwood Press, 1992), 25, http://www.questia.com/read/27419298.

    Cited passage

    Thanks for trying Questia!

    Please continue trying out our research tools, but please note, full functionality is available only to our active members.

    Your work will be lost once you leave this Web page.

    Buy instant access to save your work.

    Already a member? Log in now.

    Author Advanced search

    Oops!

    An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.