growing public concern over the escalating cost and a perception of declining quality of undergraduate education has caused many state legislatures to focus attention on university performance and productivity Once seemingly immune from public scrutiny, because they compete directly for scarce public funding with other state agencies, public universities are being called upon to demonstrate that they are sound fiscal investments. Increasingly, states are promulgating policies linking university funding levels to institutional productivity. As each state seeks to establish performance standards for institutional accountability, the Florida experience points to the complexity in the planning and implementing process. University accountability is a many-sided issue which, if not carefully planned and implemented, can negatively affect institutional funding and frustrate or impede proper evaluation and fiscal assessment. The lessons learned from attempting to develop an accountability system for public universities will be of interest to governmental accountants and auditors.
In 1991, the Florida Legislature enacted Section 240.214, Florida Statutes, which required the State University System (SUS) to develop and implement an accountability process that would provide for the systematic, ongoing evaluation of the quality and effectiveness of the system. The intent was to establish an accountability process that would enable the monitoring and evaluation of universities' performance in the areas of instruction, research and service, while taking into consideration the differing missions of each of the state's nine public universities.2 While the statute does not define accountability, the inference can be drawn from the statutory requirements that accountability refers to the systematic evaluation of universities to determine whether they are producing the results for which they are funded, and whether they are doing so in an efficient and effective manner. As a guide to the accountability planning process, the statute directed the SUS to address a minimum of nine prescribed performance standards (see Table 1).
As a first step toward establishing a system of accountability, the SUS administrators, directed by statute, formed an accountability committee to assist with the development of an accountability plan Constituting the committee were representatives from the SUS, the Florida Legislature, the Postsecondary Education Planning Commission (PEPC), the public universities, the Governor's Office and the Office of the Auditor General.
Although the statute stipulated the accountability performance measures to be addressed, these measures did not immediately foster consensus among committee members. In the initial stages of the planning process, committee members' divergent perceptions of educational accountability and its interrelated issues began to unfold. At times, disparate perceptions of accountability in general and the statutory requirements in particular led to circular discussions that minimized consensus and slowed the planning process. After five years of planning, implementing, monitoring and assessing accountability, several important lessons were learned. This article attempts to recapitulate some of these lessons.
Before we turn to the lessons learned, it may be pertinent to point out that planning and implementing a system of accountability for public universities is a challenging process. The Florida case has shown that while the SUS was able to develop a systemwide accountability plan within one year, implementation of planned policies had to be phased in over a five-year period. Phased implementation was done so as to minimize disruptions in institutional functions and reduce undue financial strain that could have resulted from drastic operational changes.
Overall, the reporting of institutional productivity data proved to be the most difficult part of the implementation process. Prior to the 1991 accountability legislation, institutions adhered to standardized reporting formats developed by the Board of Regents (BOR) for statutorily mandated reports. These formats had to be restructured since universities collect large quantities of productivity data for administrative purposes that were not standardized across institutions. In the first year of reporting productivity data, some institutions experienced difficulties with the reporting format. The general consensus is that it will take another year or two before the format is made more uniform and provides reliable and valid data across all the institutions.
As institutions worked to improve their accountability reporting, the Florida Legislature took steps to refine the performance measures stipulated in the 1991 law. In 1994, the legislature deleted from the list of prescribed performance standards (see Table 1) the "Total number of degrees awarded, by institution and by discipline" (item 2), and "Classroom usage" (item 9). It then added a new standard that calls for "an analysis of administrative and support functions."
Lesson 1: Statutory provisions by themselves are not sufficient to guide the accountability planning process. Statutorily established accountability measures do not ensure consensus among individuals seeking to develop an accountability system for public universities. Frequently, statutory provisions are vague and lend themselves to varying interpretations that could converge or diverge depending on the professional locations or interests of the individuals who inside and outside the university system tend to interpret the statutes differently, in terms of how these affect or relate to them. Similarly, those individuals involved in the promulgation of the statutes and those who must comply with them also tend to impute their own meanings to the provisions. Within a given university, individuals located at various levels of the organizational strata tend to interpret statutory provisions differently, in terms of the mandatory responsibilities placed on their strata or units.
Given the diversity of the accountability planning body, and the disparate interpretations statutory provisions can generate, it is advisable to pull together a preliminary outline of an accountability proposal or plan in the initial stages of developing the accountability system. Differing perceptions of educational accountability tend to create dissension rather than promoting consensus. Therefore, in striving for consensus, a skeletal framework of an accountability proposal or plan could be pulled together from the reporting system already in place, existing literature, or discussions generated at the first or second meeting of the accountability planning group. This skeletal framework would help focus and guide discussions, demarcate areas of agreement and disagreement and assist in narrowing the disparate perceptions of accountability. At a minimum, a preliminary outline of an accountability proposal or plan should contain the legislative intent, a summary of the statutory requirements, clear statements or definition(s) of accountability and some indicators or measures of university performance, if these are not specified in legislation.
Lesson 2: Key terms relating to accountability must be clearly defined early in the planning process to foster meaningful discussion and discourse.
Individuals participating in the development of an accountability program for universities may be knowledgeable about the system of higher education. This knowledge, however, does not guarantee common interpretations of key accountability terms. While terms such as goals and objectives have become a part of government accountability parlance, people use them to convey different meanings. Individuals familiar with Webster's dictionary definitions of goals and objectives tend to use these terms interchangeably, as something to strive for. Others, familiar with program evaluation literature, commonly use goals to mean general statements of desired or intended outcomes, and objectives as clear, measurable statements relating to the accomplishment of goals. Goals in this latter usage are long-term while objectives are short-term.
Similar to goals and objectives, output (units, or services produced) and outcome (effects of the units or services) are terms relating to accountability which, if not clearly defined, contribute to circular arguments that impede the planning process. For example, it is not uncommon for individuals to refer to outcomes incorrectly as units or services produced. Sometimes too, a particular unit may be considered an output or outcome, as in the case of degrees awarded. University personnel consider degrees awarded as both output (units produced) and outcome (the effects resulting from teaching and administrative services provided). Given the differences in meanings that can be attached to key accountability terms, one would likely have difficulty in determining the exact usages in free-floating discussions.
Clear definitions of key terms help to guide the accountability planning process and minimize circular discussions and dissension. In open committee forums, people customarily do not ask each other to define their terms. Shared meanings are therefore crucial for effective communication, especially when individuals participating in the accountability planning process are selected from varied public and private entities that predispose them to diverse interpretations of terminologies relating to accountability. It is therefore necessary to define key accountability terms early in the planning process in order to foster meaningful discussion and discourse that leads to the successful development of an accountability system.
Lesson 3: Educators could benefit by being attentive to the political concerns inherent in an accountability system for universities.
As part of their legislative responsibilities pertaining to education, policy makers and legislative staff seek to address the competing concerns of university educators and the general public. For the public, the principal concern surrounding state university accountability is whether these postsecondary institutions are providing the desirable outcomes or results for which they are funded. This concern is rooted in several issues which include:
1. Whether tax dollars are being spent efficiently on academic programs that will take citizens into the 21st century;
2. Whether universities provide students with the best education possible to enable them to compete in the labor market and the global economy;
3. Whether universities have shifted the academic emphasis away from providing quality undergraduate instruction to that of faculty research activities; and
4. Whether funding for faculty research serves to benefit individual faculty primarily, or the students and institution as a whole.
Developing a system of university accountability often necessitates changes in current decision-making processes and organizational practices. However, change that requires a shift in the customary way of doing business is likely to create discomfort and may even be resisted. For university administrators and faculty, the preservation of autonomy and flexibility in their work is of utmost importance. Their principal apprehensions regarding accountability tend to be whether it is an effort to restrict academic freedom; undermine the faculty research that has become the cornerstone of faculty promotions and institutional prestige; reduce university funding; micro-manage institutions; or to compromise the degree of decisionmaking autonomy institutions enjoy. Given these concerns, faculty members are inclined to view with suspicion accountability policies that appear to shift to the legislature whatever management flexibility institutions currently have. In short, university educators considered to be the vanguards of educational change are not predisposed to readily accept policy changes that call for accountability within the academic domain.
In seeking to address competing educational concerns, legislators and legislative staff customarily rely on university administrators and personnel to provide them with relevant information on which to base policy and funding decisions. Since the current political climate influences legislative funding decisions, requests for information could create uneasiness among university educators. The common fear is that the information could be used to highlight institutional weaknesses, or even reduce funding. Educators could minimize this fear by being attentive to the public's call for universities to be accountable, by offering input to legislative proposals that seek to address the public's concerns and by knowing how the information provided by institutions generally guides or fails to influence accountability policies.
Moreover, to lessen the anxiety over accountability, educators would benefit by becoming knowledgeable about the legislative process of funding universities, the factors commonly influencing and impacting on this process, the institutional information necessary to make funding decisions, how the information is used and the results (outcomes) expected from the provision of funds. It is erroneous to assume that because educators are accustomed to studying societal problems and issues, that they are attentive to public concerns regarding university accountability and how these are played out in the politically charged policy-making arena. Educators must therefore become sensitized to the seriousness of the public's discontent with university education and the legislature's role in resolving this discontent.
Lesson 4: Policy makers need to be familiarized with the educational concerns relating to university accountability.
The dynamics of accountability are such that while university educators try to preserve control over daily institutional operations, legislators seek to preserve the proper use of taxpayers' money appropriated for such operations. Despite this dynamic process, few educators are apt to see themselves as being responsible for familiarizing legislators and legislative staff on the critical importance of academic autonomy and decision-making flexibility, the linkage between teaching and research, and how all of these factors impact on institutional performance and productivity. Given the enormous amount of policy issues addressed each year by the legislature, not many policy makers have adequate time to master the complex functions of university administration, or to develop expertise in university accountability.
Educators may take note that failure to simplify the complex functions of the university education system does not insulate them from the promulgation of legislative policy they deem unacceptable. The more effective they are in familiarizing policy makers about the complexities of academe in general, and institutional performance in particular, the greater their chances of influencing and guiding policy decisions. When called upon to make presentations before legislative committees, educators should seek to provide explanations of the relevance of their academic works to, and the relationship of institutional performances, on the policy-making processes. The need for institutional autonomy, the negative consequences of inadequate state funding, the pitfalls of micro-management and the overburdening demands of statutory reporting requirements need to be explained in relation to institutional performance and accountability. And, since policy makers are regularly reassigned membership to the various legislative committees, educators should constantly seek to inform policy makers on the multiplicity of factors influencing education production.
Lesson 5: The sensitivity inherent in assessing faculty productivity could be minimized if emphasis is placed on how this productivity would be operationalized.
Accounting for faculty productivity can generate prolonged debate and create discomfort among individuals participating in the development of an accountability system for universities. For universities, faculty productivity is a triadic function of teaching, research and service that is sometimes not clearly understood by people outside academe. Faculty research, the academic function most often criticized, has not only fostered institutional prestige and brought new information into the classroom, it has contributed to major advances in the physical, biological and social sciences as well as the humanities. These advances have helped to better peoples' lives, and have aided the nation's progress as a whole. Research activities supported by state funding allocations have not only contributed to discoveries in particular academic fields, but have frequently received federally and privately matched grants that helped defray the costs of many institutional programs. Whereas the public seem to proclaim good teaching as rewarding, it is internationally acclaimed research that gets rewarded, at times with a Nobel Prize.
For the public, the principal function of faculty is to teach and to produce competent graduates. The prevailing perception is that faculty members conduct research and work on publications to foster their own prestige, marketability and advancement, while limiting the time spent on classroom instruction. Though this view may appear to be extreme, it is difficult to deny that over the last several decades, the "publish or perish" syndrome has evolved to become well-entrenched in academe. Many faculty promotion and tenure decisions in public universities today tend to be heavily influenced by productivity in research and publication. Consequently, any accountability system that fails to take into account how important research and publications are for a faculty member's growth and advancement is simply unrealistic.
The contrasting views of faculty productivity discussed above could become most frustrating for accountability planners who must find objective ways to assess faculty productivity. It is therefore important to recognize incentive structures in academe and develop accountability systems accordingly. In this regard, faculty input into the accountability process is crucial.
The sensitivity inherent in assessing faculty productivity could be minimized if faculty members seek ways to provide to accountability planners input on how to operationalize faculty performance through the use of consistent measures or indicators. By failing to participate in the accountability planning process, faculty members would be denying themselves the opportunity to inform decision-makers on the establishment of performance indicator(s) that underplay their actual productivity, e.g., establishing the number of undergraduate credit hours produced and classroom usage as the principal indicators of faculty productivity. Furthermore, faculty members' failure to participate in the establishment of objective measures to evaluate their performance is not likely to abate public outcry and deter policy makers from promulgating legislation to account for faculty productivity.
By actively participating in the efforts to operationalize assessment of their productivity, faculty members would be better able to inform accountability planners and policy makers about the linkages between teaching, research and servicefunctions. They would also be able to contribute to the development of reliable and valid measures to assess their performance, and ensure the implementation of a uniform system of performance assessment that would provide comparative information and valuable feedback. Let us take sponsored research and course offerings as examples of faculty productivity. It may be that two comparable departments or colleges with the same number of faculty reported the same amount of research projects and offered the same number of courses. Does this mean that faculty productivity in these two departments or colleges is the same? Maybe, or maybe not. A comparative analysis may reveal that faculty in one department taught larger classes and produced a greater number of credit hours than faculty in the comparable department. This points to the question of equity in faculty productivity which, in the absence of a proper comparative analysis, would be left unanswered, thereby limiting administrators from focusing on area(s) of faculty productivity in need of attention.
Lesson 6: In developing an accountability system for universities, the importance of qualitative performance data should not be underplayed.
For both university administrators and policy makers, quantitative data significantly aid the decision-making processes. Despite this importance, quantitative data often frustrate the accountability planning process, especially since it masks quality. In considering the reporting of quantitative data for accountability purposes, close attention must be paid to the qualitative aspects of performance and productivity not revealed through quantification. As evidenced in Table 1, of the nine prescribed performance standards, only one specifically seeks to address institutional quality.3 Performance standard (5) requires the reporting of institutional quality as assessed by followup surveys of alumni, parents, clients and employers. Another source of qualitative data that could be included in the list of performance standards is student surveys that focus information gathering on such topics as the quality of classroom instruction, satisfaction with instructional materials, guidance and counseling, campus social climate, etc. These factors are known to influence graduation rates and institutional success.
With states focusing their accountability efforts on performance-based budgeting, it is easy to lose sight of the importance of qualitative data as the decision is made to shift funding from input to outcome measures. For example, quantitative data may suffice when a state funds its institutions on the basis of input such as student enrollment. However, quantitative data are often lacking when used to fund institutions on the outcome measure of graduation rates. Graduation rates depend on a number of qualitative factors such as institutional quality, students' learning abilities, motivation, indecision and self concepts. Students can be encouraged to complete their undergraduate education in four years, but there is no assurance what proportion would do so. Increasing graduation rates may be as simple as reducing the number of credit hours needed for graduation, but does such action improve the quality of undergraduate education? Again, some institutions may be able to increase their graduation rates by simply increasing faculty instructional hours; but, what of the quality of instruction? Increasing a professor's instructional hours almost ensures that a greater number of students would be taught. However, in instances where the quality of instruction is not very good, requiring more of it would certainly not make it any better. Furthermore, increasing a professor's instructional hours will most likely impact on faculty-student ratio as well as the quality of instruction. University administrators, faculty and staff could benefit by informing policy makers of the qualitative factors affecting institutional performance and the initiatives undertaken to address and minimize them. They may specify the steps taken to improve the quality of instruction, outline the procedures used to motivate students and identify the steps taken to counsel students about the consequences of being indecisive about committing to a major course of study.
Discussion and Conclusion
The development of a system of university accountability is an ongoing effort that could take several years to fully implement with reliable and valid results. Initially, most university accountability policies tend to generate discomfort, and, some may even be resisted. Discomfort and resistance seem to emerge out of one's perception of accountability. Individuals inside the university system tend to view postsecondary education as extremely complex, as something not easily comprehensible or explainable to people outside academe. From their perspective, accountability is usually seen as a set of externally imposed policies that seek to erode whatever autonomy and administrative decision-making flexibility institutions enjoy Individuals outside the university system seem to view accountability as a system of oversight to ensure that universities effectively and efficiently provide the services for which they are funded.
Whatever the perception, developing a system of accountability for universities is not a simple one-shot affair. To design a system that is acceptable by or representative of all institutions seems almost impossible. An approach that appears to have some degree of validity is to focus the accountability design on the most complex institutions within the university system, then tailor this design to fit, or use it as a guide to develop accountability for other institutions. It is perhaps easier to develop an accountability system in parts, linking each part to the other. As the parts are operationalized or tested over time, adjustments can be made to accommodate the uniqueness of each institution.
It should be emphasized that rejection of an accountability system does not cause it to wither away In addition, rejection or expression of displeasure over some accountability policy requirements without attempts to operationalize them does not invalidate them. Frequently, it is easier to make convincing arguments about the validity or invalidity of accountability requirements using empirical evidence (qualitative and quantitative data) rather than relying on logical discourse or presentations. Documented attempts at operationalization of accountability requirements, and empirical data on institutional effects or impact, almost always offer better support for desired policy changes. The accountability storm currently rolling across nearly every state in the nation is unlikely to abate unless the concerned constituencies take appropriate action to address the key issues involved.4
1 This article is in no way representative of the views of the Florida Legislature or the State University System (SUS). The author was employed as a legislative analyst and served on the SUS accountability committee from 1991 to 1995.
2 Florida's nine public universities are: University of West Florida (UWE:), University of Florida (UF), Florida State University (FSU), Florida Agricultural and Mechanical University (FAMU), University of North Florida (UNF), University of Central Florida (UCF), University of South Florida (USF), Florida Atlantic University (FAU) and Florida International University (FlU). These universities emphasize differing missions.
3 It is difficult to determine whether success rates on these exams can be directly attributed to the quality of instruction students receive at universities or some other source. Many students frequently seek out additional schooling before taking professional exams.
4 Iwould like to thank the Government Accountants Journal reviewer who offered many valuable suggestions to revise this article.…