Academic journal article Review - Federal Reserve Bank of St. Louis

Replication and Scientific Standards in Applied Economics A

Academic journal article Review - Federal Reserve Bank of St. Louis

Replication and Scientific Standards in Applied Economics A

Article excerpt

SINCE EARLY 1993, the Research Department of the Federal Reserve Bank of St. Louis has made the data and programs for articles published in the Bank's Review available to the public on its electronic bulletin board.(1) During the first year, files from articles in the Review were downloaded from the bulletin board more than 200 times. More recently, about 30 files have been downloaded each month.

The Research Department of the Bank develops the program and data files on our bulletin board during a replication of each article prior to publication. A research analyst first checks the author's data against original sources. Because databases may have been updated or revised after the research began, this can require searching for the original published data. In a few cases, data errors have been corrected, fortunately with only minor impact on the author's results. Next, an annotated version of the computer program is prepared and all statistical results recalculated. Finally, bibliographic and other references are checked by the analyst against original source documents. We believe this practice both assures the accuracy of the empirical results and allows the interested reader to delve into the details of the author's research.


Although empirical knowledge in both the physical and social sciences arises from repeated experiments, the role of data differs. In the physical sciences, scientists control a relatively small number of variables such as temperature, atmospheric pressure, diet or family characteristics. Since some variables are neither observed nor controlled, no two repetitions of an experiment will be identical. Response surface analysis and the newer field of research synthesis provide tools for analyzing the dependence of experimental results on the settings of the conditioning variables.(2)

In economics, however, unlike the physical sciences, researchers can only condition on the observed values of the environmental variables, not control them. Consider a simple model of an economic experiment:

1. Form hypotheses.

2. Collect data.

3. Develop theoretical and econometric framework.

4. Estimate.

5. Test hypotheses, draw conclusions.

The values of the conditioning variables are collected in step 2. Published articles typically describe steps 1, 3 and 5, but are most often silent on step 2. In principle, a researcher armed with the values of the conditioning variables and the computer code for step 4 should be able to exactly reproduce an economic experiment.(3) Unlike the physical sciences, the experiment is deterministic, given the data.

Appraising the robustness of the results of an economic experiment requires knowing the values of the conditioning variables used by the researcher. Obtaining the data may sometimes be difficult. Datasets and programs may be mislaid or lost during the interval between completion of the research and publication of an article. Further, requests to authors for data may raise suspicions that the reader hopes (or expects) to find errors in the authors' research. An individual researcher has strong incentives not to share data and programs. If the materials are shared and results confirmed, the confirmation provides little (if any) reward to the researcher beyond the original publication of his findings. If results are found faulty, however, the researcher faces the likelihood of some professional embarrassment.

The trepidation of authors aside, scientific progress depends on challenging received wisdom. In applied economics, these challenges fall into three categories: replication of published results using the previous authors' data and programs; applying new statistical methods or techniques to authors' datasets; and application of existing statistical methods (including those used by previous authors) to new datasets.(4) That most applied economic research falls within the third category is not surprising, since the first two depend on access to previous authors' datasets. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.