uation must, therefore, be "just-in-time," providing smaller packages of evaluation findings for use in assessing innovations. TQM practices, such as statistical process control and the Plan-Do-Check-Act cycle, can be integrated into traditional evaluation practices.
Rapid change increases the requirement to demonstrate that an agency's program is accomplishing something ( Behn 1991). Small-scale, timely evaluation can help others understand innovations and, if appropriate, replicate them ( Levin and Sanger 1994). Formative evaluations, or evaluations that are process-oriented, that answer the questions of primary stakeholders, and that include qualitative methods that allow a "holistic assessment of relevant phenomena" ( Thomas 1995), will be more useful to public managers and policyrnakers. The application of technological tools such as automated information systems, telephone surveys, and computer-assisted analysis and reporting can increase the possibility of evaluation being timely and useful to public managers. Benchmarking, or the establishing of standards or "best practices," leads not only to evaluation of current programs but also to goal-setting for program improvement.
The problems with evaluations fall into three categories: political, measurement, and usefulness. Political problems result from the tension between those being evaluated and the evaluators, especially when there is a probability of program termination or reduction. In public management, political problems also emerge when the priorities of a legislative of funding body are different from those of the agency. Measurement problems occur because many governmental programs or services are difficult to measure, and evaluators must develop measures for selected objectives. Stakeholders in the evaluation process may not agree as to the appropriateness or accuracy of those measurements. The usefulness of evaluations is compromised if the study is ad hoc or ex post facto rather than an integral and inherent part of the program from its inception. Then, necessary data may be incomplete or absent.
Usefulness is also limited when evaluators issue reports filled with technical terms and complex statistical analyses that confuse and mislead. Evaluation reports that are simple and easy to understand, however, may miss important, more complex elements of the program. Practitioners of the "new" evaluation must be more involved in the program planning and development process, building alliances with public managers so that these problems are avoided or minimized and evaluation findings have an impact on how social services are delivered.
The usefulness, and therefore the utilization of program evaluation, can be improved in a number of ways: (1) Be clear about premises underlying a program and conduct the evaluation in such a way that those premises are addressed; (2) identify objectives and evaluation criteria that are people-oriented; (3) explicitly consider potential unintended consequences of programs, especially negative effects; (4) specify processes inherent in the program that the evaluation ought to investigate; (5) identify potential users of evaluation results early in the process; (6) analyze alternative approaches within the program; (7) consider more than one objective and multiple evaluation criteria; (8) do not reject evaluation criteria because they are difficult to measure; (9) err on the side of too many objectives or criteria, rather than two few; (10) specify client groups on which the analysis should attempt to estimate program impacts; (11) always include dollar costs as one criterion; (12) involve administrators and program practitioners at every step of the evaluation, from planning to the writing of the draft report; (13) involve potential users of the evaluation where possible; (14) complete the evaluation on time and release the results as soon as possible; and (15) use effective teaching and marketing approaches in presenting and disseminating findings. ( Chelimsky 1985; Hatry 1987; Weiss 1982).
Evaluation is always a political process, since it involves identifying objectives, selecting measurement criteria, accessing a variety of information sources, analyzing data within a specified environment, and reporting those data in ways that are understandable and useful. The impact of politics on evaluation in the public management setting is even more profound. Evaluations can be used as a political tactic; evaluators who are aware of implicit or explicit politics can minimize this practice. Alternatively, evaluation can be used as a guide in shaping policy or program changes.
The very act of conducting an evaluation may be important, if it encourages members of the organization to examine their work and the structure that supports it ( Weiss 1977). Evaluation may help agency administrators and staff, as well as legislators and other important parties, to review program goals and renew their commitment to program outcomes. This review may lead to behavioral and policy changes at a number of levels, regardless of the findings of the evaluation study. Thus effective evaluation may be even more important in today's environment of frequent and rapid changes and increasing fiscal constraints in assuring the success of public management programs and services.
SUSAN C PADDOCK
Behn, Robert D., 1991. Leadership Counts: Lessons for Public Managers from the Massachusetts Welfare, Training and Employment Program. Cambridge: Harvard University Press.
Campbell, David T., and Julian C. Stanley, 1966. Experimental and Quasi-ex experimental Designs for Research. Chicago: RandMcNally.