Accuracy Analysis of Consensus Forecasts
Bowers, William P., Jarrell, Stephen B., The Journal of Business Forecasting Methods & Systems
Many businesses both large and small rely on various forecasts of economic activity when making their own sales and financial projections. Companies that use or may be considering the Journal of Business Forecasting (JBF) as a basis for their projections, no doubt, will be interested in knowing the degree of accuracy inherent in past forecasts. Toward this end, the present analysis gauges the accuracy of past JBF predictions of selected "Consensus Forecast" variables: real GNP, CPI, and unemployment.
The forecasts to be evaluated occur in the Spring 1983 through Fall 1990 issues of the Journal of Business Forecasting, the available data set at the time of the analysis. This time span includes the 1990 downturn in economic activity. These are quarterly forecasts with one, two, three and four quarter horizons. All issues of JBF contain four quarterly horizons except the Fall 1988 issue which includes forecasts for the first three horizons only. The analysis employs Consensus Forecasts, the means of the individual institutional forecasts, for CPI, real GNP and unemployment. Evidence indicates that averages of forecasts generally perform better than individual forecasts so these results generally reflect lower bounds for any individual forecaster in the group.
All real GNP and CPI forecasts are converted to consistent "base" years that coincide with current base periods employed by the U.S. Department of Commerce and JBF. The base period is 1982-84 for CPI and 1982 for real GNP. No modification of published unemployment forecasts are necessary, because unemployment rates do not rely upon a base year. Actual values are taken from Business Conditions Digest and Survey of Current Business.
Measuring the accuracy of an economic forecast depends on whether the objective is to predict the level of a variable or to predict changes in direction, turning points, for the variable. A number of statistical artifacts measure the closeness of the forecast to the actual value by employing the error, the difference between the forecast and actual values. This study includes the mean absolute error (MAE), the root mean square error (RMSE), and the mean absolute percentage error (MAPE). In other cases, where predicting turning points is the objective, prediction-realization ratios depict the forecasts' behavior regarding missed turns and false signals.
The MAE is an average of the absolute values of the forecast errors over the periods spanned in the analysis (up to thirty quarters in this study). This results in a measure of the typical error expressed in units such as dollars and percentage points.
The RMSE also measures the average or typical size of the error using the square root of the average of the squared errors. The result emphasizes large errors, because it weights each error by itself when values are squared. Again, the result is expressed in levels and percentage points.
The MAPE measures the typical size of an error relative to the size of the value being forecast. Here error is always expressed in percentage points which facilitates accuracy comparisons among forecasts of several different variables and among several forecasts of the same variable. In the MAPE, first a relative error, the ratio of the absolute value of the error to the actual value, is calculated for each period. Then the average of these ratios is expressed as a percentage.
In addition to measures of closeness, forecast users often need to know how well the forecast predicts turning points. While each variable follows a general trend during the time spanned by this study, there are occasions where each changes direction, such as the decline in real GNP at the onset of the 1990 recession. The direction of change for the variable during the period spanned by the forecast and during the quarter preceding this forecast period determine the character of the turning point behavior for each horizon. If GNP is increasing when the forecaster predicts a downturn for the next horizon, whether it be one, two, three, or four quarters ahead, a turning point is predicted. …