Experiments are a powerful method for understanding causal relationships in journalism and mass communication research. In this essay, the authors examine seven aspects of experimental quality that reviewers should include as criteria in their evaluations. They note that there are complex interrelationships among these indicators. In cases where aspects of the standards are controversial, the authors attempt to summarize the conflicting arguments. Where different methodological conclusions can be rationalized as appropriate, the authors' suggestion is that the researcher make clear what decisions were made in the experimental design and why, so that readers can evaluate those decisions.
experiments, control, publication criteria
Experiments are important to the theoretical development of fields like journalism and mass communication because they provide the most rigorous way to establish causal relationships between independent and dependent variables (as well as moderators and mediators), relationships critical for building and evaluating theory. Experimentation is not a dominant approach in Journalism & Mass Communication Quarterly. Indeed only 12% of social-scientific manuscripts submitted to the journal in 2008-9 were based on experiments.1 Grabe and Westley reported only 9% to 13% of studies in mass communication use experimental methods.2 Nevertheless it is important that the experiments that do appear are of the same quality as they are in fields where they play a more dominant methodological role. In this essay we identify common threats to standards of experimental research quality. We do not provide a how-to summary for all experimentation, which is widely available in experimental design texts. Rather, we focus on issues that are often complex and even controversial.
Specifically, we examine seven attributes of well-executed experiments. They all involve situations where threats to the quality of social science experiments commonly occur. Conflicting arguments are presented as a means of explaining the complex interrelationships that exist among these indicators. Our objective is to explicate the reasons why these debates exist and to provide guidance to both authors and reviewers so that reasoned decisions can be made during the process of developing publishable work in the field. For the sophisticated student of experimental methods, there are many additional complexities to deal with. The references included here will also provide some suggestions for further study.
Seven Attributes of Well-Executed Experiments
1. Explication of the theory being tested and explanation of how the posited relations among independent, dependent, moderator, mediator, and control variables relate to that theory
2. Explication of how the experimental design will demonstrate causal relations between independent and dependent variables
3. Clarity in conceptualizing media stimuli
4. Clear identification of hypotheses and research questions
5. Clear specification of the sample and acknowledgment of its limitations
6. Correct specification of effect size, power, number of participants, and alpha levels
7. Consideration and empirical assessment of alternative explanations of experimental findings such as the following:
b. Message-related variance
c. Manipulation checks and/or pilot testing
d. Counterbalancing or randomization of presentation orders and conditions
1. Explication of the Theory Being Tested and Explanation of How the Posited Relations Among Independent, Dependent, Moderator, Mediator, And Control Variables Relate to That Theory
Reviewers should check that posited independent-dependent variable links (including moderators, mediators, and controls, if applicable) are rationalized within a compelling theory. A central problem for the communication field is that there are often many theoretical approaches that can prove useful. …