Testing the Tea Leaves: Evaluating the Validity of Forecasts: A Variety of Formal Methods Exist but Decision-Makers Should Also Apply a Systematic Assessment Framework
Vanston, John H., Vanston, Lawrence K., Research-Technology Management
Planning is, by definition, oriented to the future. No one makes dinner plans for last week. No successful manager is truly interested in the present, except with regard to how it can be changed for the future. Thus, all business plans, all financial plans, and all marketing plans are based on projections about how the future will unfold. These projections--forecasts--can be formal of informal, implicit of explicit, short term or long term. However, regardless of the type of forecast used in business planning, the success of the plan will, in large measure, depend on the validity of the forecast.
Because of the importance of valid forecasts and because the people charged with making key business decisions typically rely, to a great extent, on forecasts made by others, it is essential that planners, executives and other decision-makers be able to assess the validity of various forecasts. In making such assessments, these people typically rely on the reputation of the forecaster, the results of past forecasts, or their personal comfort with the forecast. However, in many cases, a more formal assessment of a forecast can be of significant value to people who must stake their reputations and careers on its validity. The purpose of this paper is to provide a set of tools that can assist in making such an assessment.
The authors' 30 years of forecasting experience involving more than 150 companies, government agencies, associations, and academic institutions, including more than half of the technology-oriented Fortune top 50 companies, has convinced us that there are two primary reasons for forecast failures: the use of inappropriate or outdated information and the use of improper models. It appears reasonable, therefore, to use procedures that will test each of these factors.
Two general types of data are typically used in forecasting: statistical data and expert opinion. Tests for the two types of data are similar, but do have significant differences.
Statistical data should be examined for the following qualities:
Reliability of source
Obviously, data sources with long reputations for reliability and accuracy are more credible than those without such reputations. Data from official government agencies usually have strong credence, as do data from recognized authoritative sources such as professional associations, public service organizations, and media files. For example, a projection of school age population in future years by the U.S. Census Bureau could be accepted as quite reliable, both because the organization has a long history of accurate population projections and because most of the people reflected in the projection will have already been born. On the other hand, data presented in most companies' promotional material may well be subject to question.
A number of commercial organizations provide data for a fee. These vary from those that organize and publish data in a general subject area to those that provide data in narrow and specialized areas. The reliance that can be placed on this information depends on the reputation of the organization, its past record, and the credibility of the organization's own information sources.
E-mail and Internet websites offer a wealth of data to people familiar with their use. Although there are means of checking the accuracy of such data to some extent, in general, data obtained from such sources should be viewed with considerable trepidation. Of course, the most serious questions about reliability arise when the sources of data are not indicated at all.
Because the gathering of primary source data is both difficult and expensive, forecasters often extrapolate
from old information or, for convenience, continue to use information that has grown long-of-tooth. Moreover, many forecasters use data available from other agencies without giving due regard to its timeliness. …