Replicability, Real-Time Data, and the Science of Economic Research: FRED, ALFRED, and VDC
Anderson, Richard G., Review - Federal Reserve Bank of St. Louis
This article discusses the linkages between two recent themes in economic research: "real time" data and replication. These two themes share many of the same ideas, specifically, that scientific research itself has a time dimension. In research using real-time data, this time dimension is the date on which particular observations, or pieces of data, became available. In work with replication, it is the date on which a study (and its results) became available to other researchers and/or was published. Recognition of both dimensions of scientific research is important. A project at the Federal Reserve Bank of St. Louis to place large amounts of historical data on the Internet holds promise to unify these two themes.
Federal Reserve Bank of St. Louis Review, January/February 2006, 88(1), pp. 81-93.
REPLICATION AND REAL-TIME ECONOMETRICS
During the past 25 years, two themes have flowed steadily, albeit often quietly, through economic research: "real time" data and replication. In replication studies, the issue is determining which data were used and whether the author performed the calculations as described; in real-time data studies, the issue is determining the robustness of the study's findings to data revisions. These themes share the same core idea: that scientific research has an inherent time dimension. In both real-time data and replication studies, the time dimension is the date on which particular observations, or pieces of data, became available to researchers. Projects at Harvard University and at the Federal Reserve Bank of St. Louis promise to improve the quality of empirical economic research by unifying these themes.1
Although replication studies focus on the correctness of results and real-time studies on their robustness, economic theory suggests that these are related-the likelihood that an author's error will become visible to other researchers is an inverse function of the cost of conducting tests for replicability and robustness. Yet, for the profession, excessive emphasis on the criminaldetection aspects of replication (Did the author fake the results? Or did the author cease experimenting prematurely when a favorable result appeared?) has tended to increase the reluctance of researchers to share data and program code. That is, to the extent that the profession overemphasizes the manhunt of David Dodge's 1952 To Catch a Thief, it risks foregoing the benefits of Sir Isaac Newton's 1676 dictum, "If I have seen further it is by standing on the shoulders of giants."
The incentives and disincentives for a researcher to share data have been discussed by numerous authors (e.g., Fienberg, Martin, and Starf, 1985; Boruch and Cordray, 1985; Dewald, Thursby, and Anderson, 1986; Feigenbaum and Levy, 1993; Anderson and Dewald, 1994; Bornstein, 1991; Bailar, 2003).2 Researchers receive a stream of rewards for the new knowledge contained in a published article, which begins with publication and eventually tapers to near zero. Furnishing the data to other researchers invites the risk that a replication will demonstrate the article's results to be false, an event which immediately ends the reward stream. If the replication further uncovers malicious or unprofessional behavior (such as fraud or other unethical conduct), "negative rewards" flow to the researcher.
Creating original research manuscripts for professional journals is craft work. Although often referred to as "knowledge workers," researchers might equally well be regarded as artisans, with creative tasks that include collecting data, writing code for statistical analysis or model simulation, and authoring the final manuscript.3 Similar to the work of other craftsmen, researchers' output contains intellectual property-not only the final manuscript, but also the data and programs developed during its creation. Yet, for academic-type researchers, some of the intellectual property must be relinquished so the work can be published in peer-reviewed journals. …