Economic Forecasting Evaluation: Re-Examination of the Track Record of Macroeconomic Forecasting
Taseen, Arshad A., Journal of Economic Issues
When a forecasting technique is selected, economists and statisticians would like to evaluate the technique for its ability to forecast accurately. Normally an error--a difference between the actual value and its forecast--is assessed in different forms. There is no single measure of error that would capture accuracy in its entirety. There are techniques to measures accuracy in terms of magnitude and also in terms of direction. Each measure captures different aspect of accuracy and provides a good insight into the capability of the forecasting technique. The measures we have overlap and provide some exclusive information from one another but not exhaustive information in terms of accuracy. As a result, the outcome of a measure may be sufficient and helpful for one user but inadequate or even misleading for the need of the other.
The Wall Street Journal (WSJ) has been polling its panel of economists on their predictions for short-term interest rates, long-term interest rates, consumer price index, gross domestic product, unemployment rate, and dollar to yen exchange rates since 1981. Some research has been done to assess its accuracy (See Cho 1996, Cho and Hersch 1998, and Greer 1999 and 2003 among others; also see C. M. Ford, "Littmann, Top Forecaster, Interprets Nation's Economy Based on Midwest," The Wall Street Journal, Eastern edition, 1, July 7, 1998.)
Mark Greer (1999) tried to determine how often The Wall Street Journal's forecasters correctly predict the direction of change in the macroeconomic variables and found an overall forecast accuracy of 49 percent--not much different from the expected value of a random forecast-generating mechanism.
This paper re-examines WSJ forecasts under an alternative measure of directional accuracy and shows the results to be quite different.
This study uses the same data used by Greer (1999) to compare the results. In more recent years WSJ has changed the format of its reporting. The foreign exchange rate that it used to report was U.S. dollar to yen; now it is U.S. dollar to euro. The long-term interest rates used to be thirty-year bond rates; now they are ten-year bond rates. The unemployment rate became erratic; the methodology of its estimate was changed at least twice. GDP values are revised by the Department of Commerce; forecasters might not possess the new values at the time of making their forecasts. As a result, some forecasts of later years became unsuitable for comparison in our study. However, this study was later extended wherever possible to include data subsequent to the past study but that was not included for comparison.
The structure of this paper is to first review the recent work on directional accuracy and in the third section describe briefly Greer's hit and miss method. An alternative method with its calculation is presented in the fourth section along with the comparisons of the results in two methods. The fifth section presents the conclusion.
Economists and statisticians use different techniques to measure errors of their forecasts. A majority of forecasters and their users are interested in measuring the forecast accuracy in terms of the magnitude of error. Since it is impossible to measure the accuracy in its entirety, forecasters and users often use several measures of error to decide the acceptability of their forecasting technique or forecasts. Though there is an overlap among different measures, there is also some exclusive information in each measure. A number of methods exist to measure the magnitude of error and have been dealt with adequately. (See Theil 1966, Makridakis et al. 1983, Kling and Bessler 1985 and Armstrong and Collopy 1992 among others.)
The direction of the error, however, does not have a similar depth of discussion in the literature. James Cicarelli (1982) and Gerhard Thury (1985) used directional change accuracy to evaluate the accuracy of economic forecasts and pointed out that often "directional accuracy is more important than the magnitude of the accuracy" (Cicerali 1982). …