Toward Developing Measures of the Impact of Library and Information Services
Durrance, Joan C., Fisher-Pettigrew, Karen E., Reference & User Services Quarterly
A convergence of factors, both within and outside of librarianship, has created an environment conducive to the development of what has in the past seemed too difficult--measures that will be able to determine the impact of library services. These factors include: advances in research that improve evaluation approaches; demands for public sector accountability; and governmental activities aimed at determining service outcomes. They will influence the development of a new generation of evaluation tools for librarians and other professionals. This article examines these factors within the framework of today's key evaluation questions, "What differences do public services make?" Using data from a recently completed IMLS-funded study, the authors identify and discuss impacts of library community information services as well as implications for the development of context-centered evaluation tools.
Factors Influencing Changes in Approaches to Public Sector Evaluation
Advances in Evaluation Scholarship
Researchers are making contributions to a paradigm shift by turning the evaluation lens away from institutions and toward people's activities. In the process they are beginning to provide public sector professionals with the knowledge and skills they need to turn the evaluation lens away from the institution and toward users of services.
Patton's definitive book on evaluation is a landmark contribution to understanding utilization-focused approaches to evaluation. It not only traces the development of these approaches, it also synthesizes many social service evaluations and presents approaches to focusing a particular evaluation, evaluation questions, methods, data analysis, and presentation of findings. (1) Patton puts measuring impact, or end results, at the top of his hierarchy of evaluation. Table 1 is based on this hierarchy.
The Aspen Institute's Roundtable on Comprehensive Community Initiatives produced a rich array of evaluation research, including two major reports on evaluation research and a database of measures for community research (www.aspenroundtable. org). These reports, developed by some of the evaluation field's finest researchers, provide a rich store of relevant theoretical and methodological approaches as well as a cogent discussion of the challenges and complexities associated with "fundamental questions about how to ascertain the ways in which an investment of resources has paid off." (2) This work clearly details the incredible difficulties faced by evaluators who seek to determine outcomes of community services. These researchers stress the need for evaluators to ground their work in theory, in particular the theory of change, and to incorporate contextual factors. Finally, they recognize that "the practice of evaluation is itself a profoundly political and value-laden process, involving judgments about the validity of program objectives and choices about how progress can be measured." (3) The work of these researchers is vitally important to developing an understanding of the complexities of this type of evaluation.
In Local Places, Global Connections: Libraries in the Digital Age, Schement also articulated the importance of understanding the context in which citizens seek information and adopt new technologies. He warned that libraries "lag in [their] understanding of the evolving social context--a context in which libraries will have to justify themselves," and suggested that libraries consider "how Americans [will] live their lives as citizens, as economic actors, and as social beings." (4)
Digital library researchers have begun to examine the social aspects of the design, use, and impact of information systems. (5) Bishop and her colleagues argue that combining these approaches with participatory action research
... focuses digital library design and evaluation directly on the digital divide. Participatory action research demands relevant outcomes for marginalized members of society. …