Sample selection issues are relevant for any empirical exercise with human subjects. We study this problem directly, in the context of laboratory experiments in economics. However, the issue is at least as salient for "field experiments" (see Harrison and List 2004 for an overview). Among other issues, they still need to recruit their subjects, and thus there is the possibility for selection bias. Indeed, no sample is likely to be fully representative; Gronau (1974) is an early paper that worries about just such an effect on wage selectivity in labor markets. We discuss further in the conclusion the relevance for both field experiments and for fully naturally occurring observed data.
To directly assess such effects, we hypothesize in advance that a particular recruitment procedure will affect the composition of the subjects who show up to participate. We use standard laboratory protocols for comparability, but the implications are similar for either the lab or the field. In particular, we vary the amount of information (about the task to be performed and/or the expected payment) revealed at the time of recruitment. This is a dimension that varies in any case, but that is not often explicitly considered or controlled for. It also has a natural theoretical link to ambiguity aversion: an aversion to uncertainty over states of the world about which the probabilities are unknown. (1) We hypothesize that potential subjects who are more ambiguity averse will be less likely to choose to participate if they have less information about their possible outcomes. Although we focus on this single aspect, we stress that our concern is broader. Selection biases are likely to be present in almost all situations, along a variety of dimensions, and by definition they are unusually difficult to test for and to control for.
To test for this effect here, we begin by inducing a representative sample of undergraduates, namely almost all students in several pre-occurring groups, to voluntarily participate in the first phase of our experiment in which we measure ambiguity preferences (specific procedures are described in Section II). This is by no means representative of the population at large, but if anything it is more homogeneous--making it more difficult for us to observe selection effects within that group. In fact, even in this case, we do find a significant selection effect when those same students are invited to participate in a follow-up experiment via a randomly varied recruitment e-mail. In particular, none of the e-mails that we used successfully led to the same underlying distribution of types as existed in the base population (sample frame).
The general issue of potential bias both in subject pools and in subject behaviors has been considered by experimental psychologists for many years (Orne 1962; Rosenthal and Rosnow 1973). Note that there are two distinct considerations: Who volunteers to participate in an experiment to begin with? And does their behavior change relative to other settings? The latter effect is sometimes referred to as a demand characteristic, with many studies finding that subjects appear to conform their behavior to that which is "demanded" by the researcher. But of course, without good evidence as to what the baseline population looks like, it is difficult or impossible to separate these effects. Experimental economics may suffer slightly less from both effects: the first because there is always payment for participating and the second because often we have been interested not in individual differences but rather in comparing institutions or testing theories that are supposed to apply to everyone equally. Even within economics, this potential problem was discussed quite early (e.g., Kagel, Battalio, and Walker 1979), but it has received little attention.
As it matures, however, experimental economics has become increasingly interested in behavior differences among groups, and here the selection effects are more acute. …