Evaluating the 1996-2006 Employment Projections: On the Whole, the BLS 1996-2006 Employment Projections Outperformed Alternative Naive Models, but Not Projecting the Housing Bubble or the Rise in Oil Prices Did Cause Some Inaccuracies in the Projections
Wyatt, Ian D., Monthly Labor Review
Every 2 years the Bureau of Labor Statistics (BLS, the Bureau) publishes long-term economic projections. In 1997, BLS published the 1996-2006 projections. (1) These projections are used by policymakers, economists, and even students making career decisions, and are one of the most popular products on the BLS Web site. With such a popular product, it is important to ask the question, How accurate are these projections?
BLS has published numerous articles evaluating its earlier projections. (2) Most past articles focused on a specific part of the projections: the labor force, industry employment, or occupational employment. In 2005, two researchers from outside the Bureau (on a contract with BLS), H.O. Stekler and Rupin Thomas, wrote an article evaluating the accuracy of the Bureau's 1988-2000 projections and suggested metrics and methods, including naive models, for evaluating future projections. (3) Subsequently, BLS decided to revamp how it evaluates its own projections and convened a team that used Stekler and Thomas' ideas as a starting point when it developed recommendations for future projection evaluations. Following the team's recommendations, this article evaluates all four parts of the projections in a holistic manner and attempts to show how problems in earlier parts of the projections process affected the later parts.
The first step in evaluating any projection is to compare actual data with projected data. Although such a comparison may seem simple, revisions to GDP definitions, occupation and industry classification systems, and historical data make such a comparison quite complex. The first section of the article discusses how BLS attempted to match and compare projected data, which were created on the basis of one set of definitions, classifications systems, and historical data (pre-1997 data), with actual data, which were created under a different set of definitions and classification systems, and were based on revised historical data. After discussing data comparability, the article will evaluate the accuracy of the four parts of the projections: the population and labor force projections, the macroeconomic projections, the industry employment projections, and the occupational employment projections. The metrics used in this article are described in the appendix.
Following Stekler and Thomas' recommendations and the BLS projection evaluation team's recommendations, this article will not only quantify the accuracy of the projections; it will attempt to explain, when possible, why differences occurred, and will compare the accuracy of the BLS projections with the accuracy of naive models. (See the appendix for an explanation of the naive models used in this article.) The comparison with naive models is necessary because accuracy can only be judged through comparisons. Because of a lack of comparable projections, naive models provide the basis for comparison. Most of the naive models employed in this article assume that the rate of change over the next 10 years will equal the rate of change over the previous 10 years. The section of the article on occupational employment projections describes additional types of naive models employed in that section.
The 1996-2006 projections, created in 1996 and 1997, used 1996 data and the racial, gross domestic product (GDP), occupational, and industrial definitions, and occupational and industrial classification systems, used in 1996. Analyzing the accuracy of economic projections requires comparing projected values with actual data. In the case of the 1996-2006 projections, direct comparisons between the 2006 projections and the 2006 actual data were more difficult because the aforementioned definitions and classification systems changed between 1996 and 2006. In addition to those changes, the results of the 2000 Census brought into question the accuracy of the 1996 estimates of the U.S. population, and the rebasing and revising of GDP data altered the historical data (1986-1996) that were used to create the projections. …