Desperately Seeking Management: Understanding Management Quality and Its Impact on Government Performance Outcomes under the Clean Air Act
Heckman, Alexander C., Journal of Public Administration Research and Theory
THE CHALLENGE OF MEASURING MANAGEMENT
Public management studies consistently find that measures of management quality and the use of certain management practices are positively related to government performance. However, the results of these studies often do not provide useful, generalizable, guidance about what practitioners should do to improve management practice. Unfortunately, management quality measures are frequently based on the subjective opinions of employees within the organizations studied or they have been too narrowly defined for a particular organizational or programmatic context to be of general applicability (Brudney, O'Toole, and Rainey 2000; Donahue 2004; Meier and O'Toole 2002; Nicholson-Crotty and O'Toole 2004; O'Toole, Meier, and Nicholson-Crotty 2005; Selden and Sowa 2004). Additionally, management measures used in the literature are often opaque as to what constitutes good management. For instance, Meier and O'Toole (2002) used the residual from a regression model of a superintendent's salary as a measure of managerial quality and found that good management had a positive impact on student learning outcomes. Although findings from such studies may support the notion that good management produces good performance, they shed little light on general steps for improving public management practice and policy implementation outcomes.
The lack of concrete, yet general, measures of management quality represents a major obstacle to producing pragmatic insights about the public management-government performance relationship (Boyne 2003, 2004; Boyne et al. 2005). Although Meier and O'Toole (2002) have stated that this measurement challenge is "intractable," developing more useful, general measures of management quality is essential if public administration research is to produce practical knowledge for improving public management practice that results in better government performance.
A significant exception to the lack of general and concrete measures of management quality is the management grades assigned to the 50 states in 1999, 2001, 2005, and 2008 by the Government Performance Project (GPP), which is currently managed by the Pew Center on the States. The GPP management capacity measure provides detailed, general, criteria that clearly define good management practice. The GPP grades represent the best, criteria-based, multidimensional measure that is regularly applied to a large number of governments in the United States (Borins 2005; Ingraham, Joyce, and Kneedler 2003). Studies using the GPP suggest that it is a promising way to operationalize the concept of management quality in a general way that provides practical insights into what constitutes good management practice (Burke and Wright 2002; Coggburn and Schneider 2003; Donahue, Selden, and Ingraham 2000; Knack 2002).
This study uses the GPP management measure to test management's impact on environmental outcomes--a wholly different context than past studies using the GPP. The specific questions addressed in the study are:
1) Does state management quality impact air pollution control outcomes?
2) What is the impact of management quality on air pollution control outcomes relative to other state-level factors related to problem severity, resources, and the political environment?
The study incorporates design elements and data that make it well-suited for answering these questions. The important study elements include:
* Model Specification: The model specification was developed using the Mazmanian-Sabatier implementation model (MSIM) illustrated in Figure 1, which is a well-established policy implementation model.
* Outcome Measures: The analysis uses two different measures of performance outcomes including a unique outcome measure that estimates reductions in air pollution emissions due to national Clean Air Act (CAA). This second measure better reflects the outcomes sought by policy makers and achieved by policy implementers, than similar studies using only aggregate emission outcome measures (e.g., Carson, Jeon, and McCubbin 1997; Lowry 1992).
* Causal Measures and Data: The analysis also uses measures of management quality and interest group influence that more accurately capture these concepts than similar studies (e.g., Lester, Franke, and Bowman 1983). Also, the study includes better data on the total amounts spent within each state on air pollution control efforts. The data were collected by the author and provide the most accurate estimates available of these expenditures.
* Comparative, Large-N Analysis: Two different measures of air pollution control outcomes affected by the same causal variables are modeled, which enables a comparative analysis of the empirical results. Also, air pollution outcomes for 47 states are modeled, providing generalizable results not found in many studies examining government performance, particularly studies on policy implementation.
Together, these elements create a unique study that sheds new light on the management-performance connection and the causal factors affecting national air pollution control outcomes.
[FIGURE 1 OMITTED]
CONTEXT FOR ASSESSING MANAGEMENT IMPACT
National air pollution policy provides a useful context for assessing the impact of state management quality on performance outcomes because it provides a common policy design in regards to air pollution control, but state-level implementation efforts and characteristics are critical determinants of the extent to which policy goals are achieved (Woods, Konisky, and Bowman 2009). The CAA is the primary air pollution control legislation in the United States.
Under the CAA, the US Environmental Protection Agency (USEPA) sets national ambient air quality standards to ensure basic protection of human health and natural resources, but state and local governments are primarily responsible for enforcement of the policy. All 50 states have been delegated responsibility for implementing the CAA provisions and states face financial and other sanctions from the USEPA if they fail to adequately carry out their duties (ECOS 2007; USEPA 2007). States and certain local environmental agencies implement the CAA by monitoring air quality, inspecting facilities and enforcing air pollution regulations. Amendments to the CAA enacted in 1990 further increased the regulatory role states play and strengthened the ability of the states to meet national air pollution standards (Rabe 2006). (1)
Evidence of the prominent role states play in carrying the CAA can be seen in USEPA statistics. Nearly all the air pollution data in the USEPA's databases are collected and reported by the states (Brown and Green 2001; ECOS and USEPA 1999). States also bear most of the financial responsibility for enforcing air pollution control programs. On average, more than $1 billion per year is spent on air pollution control activities within each of the 47 states included in this study. However, according to the National Association of Clean Air Agencies (NACAA 2009), less than 25% of all the revenue for state and local air pollution control activities comes from the national government.
Complying with CAA requirements and avoiding federal sanctions is an important goal of each state government, including the state environmental agency, the legislature, and top executive officials. Researchers generally agree that states have been active at carrying out their duties under the CAA, although the level of commitment and competency varies by state (Rabe 2007). Therefore, differences in states' spending, management quality, and political environments should determine the air pollution control outcomes they achieve (Lester 1995; Rabe 2006). This is particularly true for industrial source nitrogen dioxide (NOX) emissions, which are the outcomes modeled in this study.
THEORETICAL MODEL AND MANAGEMENT QUALITY MEASURE
The MSIM is one of the most comprehensive and empirically tested policy implementation theories and was developed to model intergovernmental implementation situations like the one examined in this study--see Figure 1 (McFarlane and Gruebel 2006; Zheng 2000).
The MSIM states that government performance is the result of three categories of factors (Mazmanian and Sabatier 1989):
Statutory: The statutory structure of a policy is set in law by elected officials to direct and constrain policy implementation. According to the MSIM, policy implementation should be more successful when goals are clearly understood, there is a sound understanding of the causes of the problem, and sufficient resources are provided to achieve policy goals. Under the CAA, the policy design and goals are set by the national government and is the same for all states. Total spending on implementing air pollution control policies is the main statutory variation across the states. The combined decisions of national, state, and certain local governments determine the level of resources spent on air pollution control efforts within each state.
Nonstatutory: Nonstatutory factors are not in direct control of national policy makers but impact the policy outcomes achieved by officials implementing the policy. Relevant nonstatutory factors include the support or opposition of the general public and influential interest groups for the policy and the skill of implementing officials or management quality within each state.
Under the CAA, the ability of management to effectively achieve policy goals depends on good management practices, sufficient funding for environmental efforts, and political support that enables states to take the actions necessary to enforce environmental standards that negatively impact, or are perceived to negatively impact, economic output in a state and may be seen as overly intrusive policy interventions.
Problem Tractability: The idea that the technology available to deal with the problem situation, the severity of the problem, and the desired change in the problem state will impact the effectiveness of any policy intervention. No policy can accomplish unrealistic goals that we do not have the knowledge or methods for achieving.
In this context, states vary widely on both the level of industrial NOX pollution they face and the extent of change required for bringing them into compliance with the CAA standards. The technology and techniques for reducing air pollution are available and well understood, so the level of industrial output is the major difference in problem tractability between states that will impact industrial NOX emissions and each state's ability to reduce those emissions.
Empirical Support for the MSIM
Empirical analysis has typically supported the MSIM (Bullock 1981; Goodwin and Moen 1981; Lester and Bowman 1989; McFarlane 1989; Meier and McFarlane 1995; Sabatier and Klosterman 1981; Zheng 2000). However, these studies typically have not incorporated measures of all three types of factors in their analysis. Also, the impact of implementer skill or management quality has not been tested in prior research because a suitable measure was not available (Mazmanian and Sabatier 1989; McFarlane and Gruebel 2006). Additionally, implementation research has typically been limited in regards to the generalizable insights it has produced because such research often analyzes case studies of failures or disasters rather than conducting quantitative analyses of a large number of similar government entities implementing similar policies (Fox 1987; O'Toole 2000, 2004). This study provides generalizable results about factors affecting policy implementation by analyzing nearly all the American states' implementation of air pollution policy and incorporating state-level measures for all the major categories defined in the MSIM, including management quality.
The GPP Management Measure
According to Ingraham, Joyce, and Donahue (2003), the GPP grading process is "the largest and most systematic effort to define and measure good management using rigorous and consistent techniques." The GPP grades are used by public administration scholars, policy makers, and the media as indicators of state management quality because they are based upon transparent, concrete, and detailed criteria developed by expert panels comprised of academics and practitioners from all levels of government (Borins 2005; Clifton 2000; Coggburn and Schneider 2003; Donahue, Selden, and Ingraham 2000; Lewis 2005). The original GPP criteria for each management system are summarized in Table 1.
The management systems presented in Table 1 were selected "because they were judged to be common systems in all levels of government, they have common characteristics that are amenable to comparison, and they compromise a major part of government managers' and organizations' management activities" (GPP 2002). The criteria represent a synthesis of academic research and measure the application of general management principles that can be accomplished using different practices, depending on the context (e.g., Williams 2005). Overall, the GPP criteria emphasize rational decision making, operational transparency, public accountability, and formal planning for and monitoring of the performance of each management system. According to the GPP, sustained, long-term performance at a high level requires effective maintenance, ongoing coordination, continual monitoring, and timely improvement to the management systems.
The final GPP evaluation relies primarily on three main data sources: (a) a management survey completed by each state, (b) documents about each state's management systems (e.g., budget documents, strategic plans, and human resources policies and procedures), and (c) interviews conducted with state government officials and external stakeholders, such as reporters and representatives from citizen watchdog groups. The GPP graders obtain and review similar documents and interview people in similar positions within each state to maximize the comparability of the information obtained and enable an assessment on all the criteria (Ingraham, Joyce, and Kneedler 2003; N. Johnson, personal communication, April 22, 2010). (2)
The final grades are based on an assessment of the management survey responses and documentary evidence by academic experts and an assessment of the interview responses by the journalists who conduct the interviews. In both cases, multiple academics and journalists individually review the data and assign grades for each state. Then, a team of the academics and a team of the journalists separately agree on a grade for each state. Finally, the academic and journalistic groups collaborate to arrive at a final grade. The approach used to assign the GPP grades seems sound given the ambitious and complicated nature of assigning management quality grades to all fifty states (Borins 2005; N. Johnson, personal communication, April 22, 2010).
Empirical Studies Using the GPP
Studies using the GPP have typically found the expected relationships with other management quality measures and government performance variables. The GPP has been found to be related to another measure of management quality (Burke and Wright 2002), to various measures of state quality of life (Coggburn and Schneider 2003), to certain measures of social capital (Knack 2002), and to local government human resources effectiveness (Donahue, Selden, and Ingraham 2000).
This study uses the overall state GPP grades, instead of the management system grades, to focus the analysis on the GPP as a holistic measure of management quality. Also, the management system grades for each year are not comparable for all the years included in this analysis because the management system categories were consolidated after 2001 from the five presented in Table 1 to four--Money, People, Infrastructure, and Information. However, the GPP still applied similar criteria, followed a similar grading process, and reviewed substantially the same evidence in assigning the overall grades used in this study. The 2005 changes to the GPP present a challenge for comparing the subcategory grades to past grades but are only of minimal concern when comparing the overall grades to each other (Pew Center for the Sates, 2008; N. Johnson, personal communication, April 22, 2010). (3)
METHODS, DATA, AND HYPOTHESES
The foundation for my empirical examination is a regression analysis, using a model specification based upon the MSIM, to analyze the impact of management quality and other key state-level factors on two different measures of state industrial NOX emissions. Modeling different dependent variables determined by the same causal factors facilitates comparative analysis of the two models' regression results, including some assessment of the robustness of the empirical findings (Pawson 1989).
Industrial NOX emissions, excluding emissions from power plants, are analyzed because these pollution levels are most likely to be impacted by state implementation efforts. In contrast, pollutants, such as sulfur dioxide, are more likely to be affected by federal cap and trade programs and utility regulation. (4) Similarly, mobile source pollutants are mostly affected by federal enforcement of national regulations, such as compliance with automobile fuel efficiency standards. Focusing on industrial NOX emissions also minimizes the problem of cross-state pollution affecting the findings (USEPA 2001). Therefore, analyzing NOX industrial point source emissions should provide the most favorable context for discerning the impact of state-level factors on these air pollution control outcomes.
Additionally, NOX pollution is an important policy problem because it negatively impacts public health by contributing to the formation of ozone and particulate matter. Reducing ozone levels and particulate matter was a priority of the 1990 CAA because of the severity of the problems caused by these pollutants (USEPA 1999).
The first dependent variable is tons of industrial point source NOX emissions per state. An aggregate emissions measure such as this, often standardized by population or some other factor, is one of the most common air pollution control outcome measures used in this type of analysis (e.g., Carson, Jeon, and McCubbin 1997; Lowry 1992; Stern 1998). Table 2, which presents the descriptive statistics for the dependent and independent variables, shows that most states produce industrial, point-source NOX emissions of less than 115,000 tons.
Also, the distribution of NOX emissions is skewed above the mean, so the log of the emission variable is used in the analysis to bring the distribution close to normal, which helps to ensure more accurate coefficient estimates (Stern 1998).
Although an aggregate emissions measure is a commonly used dependent variable in analyses examining state and national environmental performance, such variables are not necessarily the best measure of the policy goals because environmental policy is ultimately focused on reducing emissions. Therefore, a change in emissions measure is a better outcome variable to use when analyzing the impacts of pollution control efforts because it better reflects policy goals (Ringquist 1993).
To address this issue, the second dependent variable is emissions reductions per state as a result of the 1990 CAA Amendments (ERNE). This variable measures the estimated reductions in state-level emissions between 1990 and 2000 after the CAA was amended in 1990.
The 1990 CAA law required the USEPA to issues a series of reports on the costs and benefits of the policy (USEPA 1997). The second of these reports was issued in 1999 and estimated the national impact of the 1990 CAA Amendments on pollution emission between the years 1990 and 2000. These estimates are the foundation for deriving the second dependent variable used in this study. (5)
In the report on the benefits and costs of the 1990 CAA Amendments, the USEPA (1999) forecasted pollutant emission levels for NOX emissions under two scenarios. One scenario assumed that the CAA Amendments were not passed in 1990 and that no additional control requirements would be passed by the national, state, or local governments between 1990 and 2010. The second scenario estimated emissions based on the forecasted impact of the CAA with the 1990 Amendments. In short, USEPA estimated the impact of the actual policy and the impact under the counterfactual.
I used the USEPA emission estimates and actual data on NOX industrial point source emissions for the years 1990 and 2001 to calculate each state's reduction in emissions as a result of the CAA. A key assumption of the ERNE calculation is that the each state's relative share of national emissions in 1990 would remain the same if more stringent standards and enforcement mechanisms had not been introduced by the 1990 CAA Amendments. Based on this assumption, the actual 1990 relative shares can be used to apportion the USEPA national projection for 2000 under the non-CAA Amendments scenario. Similarly, each state's share of actual 2001 emissions is used to apportion out its share of the USEPA NOX emissions projections for industrial point sources in 2000 under the CAA Amendments scenario. (6)
As shown in Table 2, most states were estimated to have reduced NOX industrial emissions by less than 21,000 tons. Five states were projected to have had increases in NOX emissions after implementation of the 1990 CAA Amendments--Arkansas, Georgia, Hawaii, Kentucky, and Mississippi. The distribution of the ERNE variable is approximately normal, so no transformation of this variable was necessary.
The dependent variables are not standardized by state population or some other factor because it is the aggregate pollution levels that are of concern for public officials. It is the overall levels that create the health and safety impact and the goal of policy is to reduce aggregate levels of pollutants not per capita reductions. However, the analysis does account for size effects on the dependent variables by including a lag of the dependent variable in the model for each outcome measure. I mitigate simultaneity issues in the emissions model analysis by using prior years' values for all the independent variables, which should produce more accurate coefficient estimates. Simultaneity is not a concern with the ERNE model because it is a measure of change (Ringquist 1993; Stern 1998).
Causal Variables and Expected Relationships
Table 3 shows the independent variables used in the analysis and the hypothesized relationships with the dependent variables.
1) The GPP grades are used to measure management quality. As shown in Table 2, almost all states have C-level management or better, according to the GPP grades. The average GPP grade is 2.6 or just below a B-level. The grade distribution is essentially bimodal with 95% of states rating a B-level (26 states) or C-level (21 states) grade.
GPP Hypotheses: The GPP variable should have a negative relationship with both emissions levels and the ERNE because states with better management quality should have lower emissions and achieve larger reductions, which are measured by larger negative values.
2) A state-level citizen ideology index developed by Berry et al. (1998) is used to capture the concept of citizen support for CAA implementation. (7) The index determines how liberal citizens are in each state by calculating the ideology of each state's Congresspeople based upon ratings from the liberal group Americans for Democratic Action and the conservative group Americans for Constitutional Action. The ideology index can range from 0 (extremely conservative) to 100 (extremely liberal).
Berry et al. have demonstrated the reliability and validity of the measure through empirical testing of many of the assumptions upon which the measure is based and testing expected relationships with other relevant phenomenon. This variable avoids the pitfall of measures commonly used in implementation research that merely quantify the partisan composition of state or national officials as a measure of state ideology (e.g., Lester et al. 1983). This is problematic because the meaning of partisan ideology is not consistent across states (Berry et al. 1998). For example, Democrats in Texas are more likely to be ideologically similar to Republicans in Massachusetts rather than Democrats.
As shown in Table 2, states are, on average, somewhat moderate ideologically with a mean of 47 out of a possible 100 on the ideology index. Nearly 60% of states have a score in the 40s or 50s.
Citizen Ideology Hypotheses: More liberal state populations are expected to demand more stringent environmental enforcement, and so there should be a negative relationship between the citizen ideology variable and the dependent variables.
3) The MSIM concept of interest group support is captured with a measure of environmental interest group influence. The variable is the proportion of pro-environmental interest group campaign contributions to successful state-level candidates relative to total interest group campaign contributions to successful candidates from both pro-environmental and pro-manufacturing industry groups. The larger the value of the spending ratio variable the more influential environmental interest groups are in the state relative to industry interests. This is a better measure of the actual relative influence of interest groups than prior studies, which have typically only examined potential influence of environmental groups based upon state membership in certain environmental groups (Lester 1995).
Table 2 shows that environmental interest groups are not very influential relative to industry groups, although there is wide variation across states. Environmental groups' contributions to winning state-level candidates represented an average of only 14% relative to total campaign contributions made during the three election cycles used in the analysis. In most states, environmental contributions did not exceed 34% of total contributions.
Interest Group Hypotheses: States with relatively more influential pro-environment interest groups should achieve lower emission levels and greater emission reductions, so this variable should have a negative relationship with both dependent variables.
Statutory Resources Variable
Total state spending on air pollution control activities is included as a measure of the financial resources used to achieve policy goals. (8) An aggregate measure of spending, rather than a per capita measure, is used because aggregate spending is a better measure of capacity. Following the logic of Ringquist (1993), pollution control is an "aggregate problem" that requires a certain minimum level of spending to effectively generate results, regardless of the amount of pollution to be reduced. It is the absolute level of spending that best captures the fiscal capacity of a state to carry out the activities required to effectively reduce pollution rather than some standardized measure such as spending per capita. For example, a small state may have quite high per capita spending, but its overall spending may not provide sufficient capacity to carry out an effective enforcement program. States that spend more on air pollution control should have higher capacity to implement the CAA and achieve greater emission reductions.
Table 2 shows that within each state, an average of more than $26 million is spent on air pollution control with two thirds of states spending more than $18.5 million and less than $33.5 million.
Spending Hypotheses: The air quality spending variable should have a negative relationship with both dependent variables.
Problem Tractability Variables
The model includes two problem tractability variables. The first variable is a measure of the value of each state's manufacturing gross domestic product (GDP). The value of production in the manufacturing sector in each state should have a direct impact on emission levels and reductions that can be achieved. In the aggregate emissions model, the variable is the total real value of each state's manufacturing GDP. In the ERNE models, the manufacturing GDP variable is the change in manufacturing value between 1990 and 2000.
The second variable is a lag measure of the dependent variable, which also captures the extent of each state's pollution problem. In the emissions model, the variable is tons of NOX emissions from the prior year, and in the ERNE model, it is the tons of NOX emissions in each state in 1990. The lag measures should help account for size effects that can explain variation arising from the use of the aggregated dependent variables.
With all of the problem tractability variables, the aggregate levels of manufacturing output are used, rather than standardizing by population or some other basis, because it is the total amounts that create the negative environmental impacts.
Two-thirds of states having manufacturing output valued at less than $60 billion. The average increase in manufacturing GDP is more than $7.8 billion with two thirds of states generating less than $21 billion in increased manufacturing output.
Manufacturing Output Hypotheses: More manufacturing should mean higher emissions, and so there should be a positive relationship with state emissions levels. The greater the increase in manufacturing, the more difficult it would be to reduce emissions. Therefore, manufacturing GDP should have a positive relationship the ERNE variable.
Lag Variables Hypotheses: The prior year emissions lag measure should have a positive relationship with the aggregate emissions outcome variable, but the ERNE outcome measure should have a negative relationship with the lag measure because states starting with higher levels of emissions in 1990 should be able to achieve greater reductions. (9)
FINDINGS AND ANALYSIS
Table 4 presents the results of the regression analysis with the log of NOX emissions as the dependent variable and Table 5 shows the results of the regression analyses with the ERNE measure as the dependent variable. (10)
Management Quality and Other Nonstatutory Variables
In both the aggregate emissions model and ERNE Model 1, the GPP management grade has the expected relationship with emissions but is not statistically significant. (11) However, there is a notable difference in the magnitudes of the GPP coefficients between the two models. Even if one ignores the lack of statistical significance, the GPP coefficient in the aggregate NOX emissions model is close to zero, suggesting no substantive impact on overall emissions. In contrast, the GPP coefficient in the ERNE Model 1 indicates a substantive impact of management quality on the level of emission reductions achieved. The GPP coefficient in Model 1 is nearly 15% of the mean NOX emissions, suggesting a substantive impact of management quality on the emission reduction outcomes.
The citizen ideology coefficient is statistically significant and has the expected negative relationship with emissions in both the aggregate emissions model and ERNE Model 1; however, the magnitude of the coefficient is notably different in the two models. A one-point increase in the citizen ideology measure is associated with a .02% decrease in NOX emissions. This means that a 10-point increase in the ideology index (10% on the 100-point scale) would be associated with a 2% decrease in NOX emissions. The ideology coefficient magnitude in the ERNE model is notably greater at about 1% of the mean of total NOX emissions. Therefore, a 10-point increase in the citizen ideology index would be associated with a reduction equivalent to 10% of total NOX emissions. This indicates a substantive difference between the two variables' impacts on emissions.
The coefficient for the interest group influence measure is positive and not statistically significant in the emissions model but is negative and statistically significant in ERNE Model 1. Additionally, there is a notable difference in the magnitude of the interest group influence coefficient between the two models. Even if one ignores the lack of statistical significance, the interest group coefficient in the aggregate NOX emissions model is close to zero, suggesting no substantive impact on overall emissions. Conversely, a 10% increase in interest group influence is associated with an emission reduction equal to more than 25% of the mean in aggregate emissions. One plausible explanation for the difference in the coefficient sign between the two models (ignoring the lack of statistical significance) is that pro-environment interest groups are stronger in states with worse emission problems, but they influence states to make greater NOX pollution reductions.
Lastly, the coefficient of the interest group influence variable in the ERNE Model 1 is nearly double the magnitude of the GPP coefficient, suggesting politics affect outcomes more than management.
The spending coefficient is statistically significant and has the expected negative relationship with both outcome measures. However, the impact of spending is modest or negligible in both the aggregate emissions model and ERNE Model 1. A 1 standard deviation increase in spending is associated with a 3% lower level of NOX emissions and a 0.02% reduction in emission levels.
One plausible explanation for the limited impact of spending increases is that the federal government policy design and USEPA oversight causes states to spend at high enough levels that spending differences between states have negligible impact on pollution outcomes. Another plausible explanation for the negligible impact of spending is that higher spending by itself does not reduce emissions. Researchers such as Boyne (2003) have theorized that spending must be coupled with good management to have a substantive and positive impact on performance.
I examined this possibility by including a management-spending interaction variable in the ERNE model. The interaction variable is calculated by multiplying the GPP and air quality spending per capita for each state, therefore weighting spending more heavily when it is done in well-managed states. As shown in Table 5, the management-spending interaction variable is significant, associated with greater emissions, but has a negligible impact on emissions based upon the magnitude of the coefficient.
The inclusion of the management-spending interaction variable also does not impact the total variation explained relative to ERNE Model 1 and does not have any notable impact on the citizen ideology, the environmental interest group influence or the problem tractability variables, nor on the signs of the spending and GPP coefficients. However, the magnitude of the spending coefficient in Model 2 is three times greater than in Model 1, though the impact is still quite modest.
Most interesting is the impact on the significance and magnitude of the GPP coefficient as a result of including the management-spending interaction variable. The GPP measure is not statistically significant in Model 1, but it is significant in Model 2 and the magnitude of the coefficient is nearly double that of the GPP coefficient in Model 1.
The problem tractability variables have the expected relationship with NOX emissions in all the models; however, problem tractability is not statistically significant in the ERNE models. Nonetheless, the magnitude of the coefficients in all the models is quite small, suggesting a negligible impact of problem tractability on these outcome measures. This is not too surprising given that the methods for reducing pollution from industrial point sources are well understood and the technology for doing so is also readily available. Therefore, the problem is quite tractable and achieving policy goals is technically feasible, but whether the methods will be applied and the technology used effectively depend largely upon good management and the financial and political support for implementation.
Summary of the Regression Results
Overall, the findings indicate a consistent yet modest impact of citizen ideology and air pollution control spending on NOX emissions. More liberal citizens and more spending seem to contribute to modestly lower NOX emissions and greater reductions in emissions under the CAA. The problem tractability variables were not consistently significant and their impact was negligible in all models based upon the coefficient magnitudes. This suggests that problem tractability is not a major factor in determining these outcomes.
How one interprets the impact of management quality and environmental interest group influence variables depends, in large part, on one's views on the importance of the statistical significance of the variables, the proper model specification, and the best outcome measure. If statistical significance is critical, then neither management quality nor interest group influence help explain NOX emission outcomes, except that management quality is significant and has a substantive impact if the best model specification is ERNE Model 2. This change in significance and magnitude suggests that analyses designed to examine relationship between management and performance may need to account for the interaction of management quality and spending.
The ERNE Model 2 would seem to be the best model because it includes both the best outcome measure and the management-spending interaction variable, which captures the critical insight that good management is required to make effective use of available financial resources (Boyne 2003).
Similarly, if one is not concerned about statistical significance and considers the ERNE variable to be a better measure of the policy outcomes sought, then environmental interest groups appear to a have a rather substantive impact on the ability to achieve emission reductions in a state. Otherwise, interest groups seem to have little or no impact on NOX emissions.
RESEARCH AND POLICY IMPLICATIONS
The specific findings of this analysis have broader research and policy implications when examined within the context of prior research into public management and government performance, particularly if one emphasizes the results arising from the comparative analysis of the ERNE models. These findings have notable implications for public administration research and policy.
Good Management and Government Performance
The findings of this analysis add to the management, government performance, and implementation literature that provide evidence that management quality and the political environment have a substantive impact on government performance outcomes (e.g., Boyne 2003; Boyne et al. 2005; Lester and Bowman 1989). The impact of management on performance outcomes is important and can exceed the impact of spending and the constraints created by the severity of the problem. Political factors such as citizen ideology and interest group influence seem to have an even greater impact on performance outcomes than management quality. Still, improving management is a viable strategy for practitioners and policy makers seeking to improve government performance.
Because this analysis used the GPP grades as the measure of management quality, the study also sheds light on the management practices that constitute good management. The GPP criteria define relatively specific practices that constitute good management including rational decision making, operational transparency, public accountability, and formal planning for and monitoring of the performance of each management system. This transparent and concrete definition of management should be helpful to policy makers and practitioners seeking to improve public management.
Additionally, the detailed and transparent nature of the GPP measure can facilitate useful dialogue about the nature of good management, which can improve both research and practice. Whatever one thinks of the validity of the GPP, measures like it provide a starting point for professional dialogue about how to define and measure good management. Discussions about what exactly constitutes good management practice should help scholars to better understand the impact of management and help practitioners be more reflective professionals. The result should be management research that makes a more practical contribution to improving management practice and government performance.
Although the GPP is a promising measure, it has limitations that relate to the key limitations of this study. First, the practical guidance based on this study is not as specific as it could have been if the GPP subsystem grades had been able to be incorporated in the study. Certainly, a subsystem analysis would be worthwhile in future research.
More fundamentally, some inconsistencies in the empirical findings in this study may, in part, be explained by the limitations of the GPP as a measure management quality. The GPP is primarily a measure of management capacity, and it does not effectively capture the other major aspect of management as defined in the literature--the quality of the daily execution of policy by states and the execution of agency-level policies within the state environmental agencies (Kotter 1990, Mintzberg 1975; Tsoukas 1994).
However, the inability to account for execution is not limited to the GPP or this study, but is one of the fundamental obstacles to developing any quantitative measure of management quality that seeks to define good management practice. Developing formalized policies, procedures, and structures is the part of good management practice that can be relatively easily observed and evaluated in retrospect, but determining how well structures are utilized, policies are carried out, and procedures are executed is not amenable to after-the-fact examination.
This challenge suggests both opportunities for improving the measurement of management quality and limitations on such quantification efforts. The GPP project and other scholarly efforts to quantify management quality need to better account for management execution in order to develop better measures and a better understanding of the management-performance connection.
However, because management execution is action oriented, communication based, and ephemeral, there are limits to our ability to measure and quantify it (Kotter 1990; Mintzberg 1975; Sull and Spinosa 2007). As such, the study of management execution is more conducive to qualitative research approaches that rely on direct observation and interpretation rather than on after-the-fact quantification using formal documents or secondary data. Therefore, a mixed-methods approach may be the best way to understand good management and its impact on performance (O'Toole 2004).
Analyzing Government Performance
This study also provides insights into how different measures and model specifications may help explain the contradictory findings often found in public administration research into the causes of government performance (Boyne 2003). For example, this analysis provided evidence that inclusion of a management-spending interaction variable and the selection of the performance outcome measure have a notable effect on the empirical findings.
Accounting for Management-Spending Interaction
Boyne (2003) has detailed the contradictory findings in the government performance literature as to the impact of spending on performance and concluded that this discrepancy may be a result of differences in measurement. Specifically, some studies operationalize resources using aggregate measures of spending while others examine real resources actually used to carry out activities designed to achieve outcomes (e.g. people and equipment). Boyne theorizes that spending variables measure financial capacity, which does not directly determine results. Instead, variables that measure real resources capture the actual use of that financial capacity, which should directly impact outcomes.
This study provided evidence for Boyne's claim by including in one of the model specifications a management-spending interaction variable to account for the notion that it is the financial capacity available for achieving a policy goal and how effectively these resources are used that determines performance outcomes. This model produced results most aligned with the MSIM hypotheses and other studies into the impact of management on performance outcomes.
Selecting Outcome Measures
The findings of this study were also notably impacted by the selection of the outcome measure. For example, the model that used aggregate pollution levels as the measure of performance outcomes did not show a significant or substantive impact of management quality on pollution. Conversely, the model that used the estimated reduction in pollution emissions as the performance outcome measure did show substantive impact of management on performance.
This result may suggest that the study findings are unreliable, but I contend that they illustrate the misleading findings that can result from selecting an outcome measure not closely related to policy goals. For example, studies seeking to explain the impact of policy and politics on pollution have often measured performance outcomes using a per capita emissions variable. However, governments are not trying to reduce per capita emissions but seek to reduce total pollution from existing levels (Ringquist 1993). Therefore, a measure of pollution reduction outcomes better captures the goals of air pollution control efforts.
In this study, the model using the emissions reductions achieved as a result of national air pollution policy produced results more in line with theory and other studies into the impact of management on performance outcomes. This finding supports the notion that better outcome measures produce more accurate results.
Applying a Comparative Approach
The findings discussed above were arrived at by conducting a comparative analysis using two different performance outcomes that were expected to be impacted by the same causal factors. The comparative approach facilitated an assessment, albeit limited, of the robustness of empirical findings and produced inconsistencies and discrepancies in the empirical results, which spurred additional questions and analysis (Pawson 1989).
For instance, the citizen ideology and spending variables were consistent across all models in terms of their sign, statistical significance, and magnitude. This provides a high level of confidence in the findings related to these variables.
Conversely, inconsistent results in the statistical significance, direction, or magnitude of other variables' coefficients raised questions that required explanation or additional analysis. For example, interest group influence was associated with higher overall emission levels but also greater reductions in emissions. This discrepancy demanded an explanation and spurred the insight that pro-environmental interest groups become more influential in states with more severe pollution problems, and in turn, these groups are more effective at causing reductions in emissions.
This insight could not have been produced by an analysis that modeled emissions as the only outcome variable. Instead such an analysis would have suggested an opposite finding--stronger pro-environment groups are ineffectual because they are associated with higher emission levels. In fact, this is not an uncommon finding in studies looking at the effect of interest groups on policy outcomes (e.g., Lester and Bowman 1989).
Overall, these examples illustrate how applying a comparative approach can improve public administration research by providing a way to assess the robustness of empirical findings and by generating discrepancies that spur additional analysis. In this case, the result was a clearer and more sophisticated understanding of the causal factors and relationships that affect government performance.
This study incorporated three key elements: (1) a management quality measure based upon criteria that clearly define good management practice, (2) a performance outcome measure that accurately captured policy goals, and (3) a comparative analytical approach based upon two different outcome measures that should be impacted by the same causal factors. Together, these elements shed new light on the management-performance connection and the causal factors affecting policy outcomes. For example, the comparative analysis illustrated the importance of accounting for the interaction of management and spending when modeling government performance outcomes.
The study also examined the benefits that can be gained by including these elements in public management and government performance research. For example, using management measures that are based upon assessments of clearly defined management practices should produce more accurate and useful research results. Additionally, developing measures based upon transparent and concrete criteria will facilitate constructive dialogue about what constitutes good management practice and inform academics' and practitioners' endeavors.
In short, future research could benefit from building upon the types of measures and methods used in this study because such research would generate more accurate and practical insights into good management practice and its impact on policy implementation outcomes.
Trevor Brown at the Ohio State University Glenn School of Public Affairs.
Amendments enacted in 1990 to Titles I, V, and VII of the CAA enhanced the responsibility and authority of states to reduce air pollution from industrial point sources (McCarthy, et al. 2007). Changes most relevant for this analysis include:
1. All states had to develop implementation plans for meeting air pollution standards both in areas in nonattainment with the USEPA standards and for preventing significant deterioration of air quality in areas that were in attainment. New classifications were created for metropolitan areas deemed to be in "nonattainment" with one or more air quality standards, and states' implementation plans were required to outline strategies for bringing such areas into attainment within specific time frames established by the law.
2. In nonattainment areas, new facilities and major modifications to existing facilities were only to be approved by states if offsetting emission reductions could be achieved.
3. All major air pollution sources, and certain nonmajor sources in nonattainment areas, were required to obtain annual operating permits that specified the level and type of pollutants a source may emit. States were charged with implementing the permit program including monitoring and enforcing compliance with permit requirements.
4. Penalties for purposeful violations of the CAA were increased, a grace period for ceasing violations without penalties was removed, and environmental agencies were authorized to assess administrative penalties.
The basic formula used by the USEPA to forecast the impact of the 1990 CAA Amendments on pollution levels was 2000 NOX Emissions = 1990 Industrial Point Source NOX Emissions x Economic Growth Factor x Pollution Reduction Factor.
Both scenarios employed an economic growth factor that is based upon federal government projections of the growth in production from each industrial facility and growth in state populations. The projections assumed that the geographic distribution of population and economic activity would be the same under both scenarios. The USEPA also estimated total national NOX emissions in a counterfactual scenario in which all national and state policies in existence prior to the 1990 CAA Amendments remained unchanged (the non-1990 CAA Amendments projection).
An example calculation of the ERNE for Pennsylvania is presented below to illustrate how this variable was derived for each state:
1. Pennsylvania's Estimated 2000 Emissions without the 1990 CAA Amendments:
A. USEPA national NOX emissions estimate for year 2000 without CAAA = 3,173,300 tons
B. Actual 1990 national industrial NOX emissions = 3,157,647 tons
C. Pennsylvania's share of actual 1990 industrial NOX emissions = 188,599 tons or 6.0%
D. Estimated emissions in year 2000 without CAAA: 3,173,300 x .06 = 190,398
2. State's Estimated 2000 Emissions with 1990 CAA Amendments:
A. USEPA national NOX emissions estimate for year 2000 with CAAA = 2,060,400 tons
B. Actual 2001 national industrial NOX emissions = 2,959,829 tons
C. Pennsylvania's share of actual 2001 emissions = 114,888 tons or 3.8%
D. Estimated emissions in year 2000 with CAAA: 2,060,400 x .038 = 78,295
3. Pennsylvania's ERNE: 78,295--190,398 = - 112,103
The total air quality spending figure includes spending within a state by both state and local agencies using revenue from all sources (i.e., national, state, and local). Local agencies in 27 states have significant CAA enforcement responsibilities that have been delegated to them by the state or directly by the USEPA.
I collected data for the air quality spending measure from different sources depending upon the state and local air agency. Initially, state and local Web sites were reviewed for annual reports from the state environmental agency and relevant budget and financial documents. When these documents did not provide the necessary data or the data did not clearly break out air pollution control spending, the author, with some assistance from NACAA, contacted officials in the state or local environmental agency and relevant budget agencies to obtain spending data or clarifications regarding published data.
For a few states, I also relied on spending data from the Environmental Council of the States after assessing its likely accuracy. In some cases, data for a particular year had to be estimated, based upon historical data and historical rates of growth in air quality spending, because I could not obtain actual figures.
Ultimately, the total state and local spending by state were determined for 47 states. Illinois, Montana, and New York are excluded from the analysis because data to calculate accurate air pollution control spending figures could not be obtained.
Ammar, Salwa, Ronald Wright, and Sally Selden. 2000. Rating state financial management: A multilevel fuzzy-based system. Decision Sciences 31:449-81.
Berry, William D., Evan J. Ringquist, Richard C. Fording, and Russell L. Hanson. 1998. Measuring citizen and government ideology in the American states, 1960-93. American Journal of Political Science 42:327-48.
Borins, Sandford. 2005. Grading the graders. [Review of the book Government performance: Why management matters]. Journal of Public Administration Research and Theory 15 (3):463-66.
Boyne, George A. 2003. Sources of public service improvement: A critical review and research agenda. Journal of Public Administration Research and Theory 13:367-94.
--. 2004. Explaining public service performance: Does management matter? Public Policy and Administration 19:100-17.
Boyne, George A., Kenneth J. Meier, Laurence J. O'Toole, and Richard M. Walker. 2005. Where next? Research directions on performance in public organizations. Journal of Public Administration Research and Theory 15:633-39.
Brown, R. Steven, and Valerie Green. 2001. State environmental agency contributions to enforcement and compliance. Environmental Council of the States.
Brudney, Jeffrey L., Laurence J. O'Toole, and Hal G. Rainey. 2000. Chapter 14: Concluding perspectives. In Advancing public management. new developments in theory, methods, and practice, ed. Jeffrey L. Brudney, Laurence J. O'Toole Jr., and Hal G. Rainey, 235-52. Washington, DC: Georgetown Univ. Press.
Bullock, Charles S. 1981. Implementation of the equal education opportunity act: A comparative analysis. In Effective policy implementation, ed. Daniel Mazmanian and Paul Sabatier. Lexington, MA: Lexington Books.
Burke, Brendan F., and Deil S. Wright. 2002. Reassessing and reconciling reinvention in the American states: exploring state administrative performance. State and Local Government Review 34:7-19.
Carson, Richard T., Yongil Jeon, and Donald R. McCubbin. 1997. The relationship between air pollution emissions and income: US data. Environment and Development Economics 2:433-50.
Clifton, Diane B. 2000. AU outreach report challenges state government's poor grade. Auburn University News, October 24. http://