Best Methods for Evaluating Educational Impact: A Comparison of the Efficacy of Commonly Used Measures of Library Instruction
Schilling, Katherine, Applegate, Rachel, Journal of the Medical Library Association
Objectives and Background: Libraries are increasingly called upon to demonstrate student learning outcomes and the tangible benefits of library educational programs. This study reviewed and compared the efficacy of traditionally used measures for assessing library instruction, examining the benefits and drawbacks of assessment measures and exploring the extent to which knowledge, attitudes, and behaviors actually paralleled demonstrated skill levels.
Methods: An overview of recent literature on the evaluation of information literacy education addressed these questions: (1) What evaluation measures are commonly used for evaluating library instruction? (2) What are the pros and cons of popular evaluation measures? (3) What are the relationships between measures of skills versus measures of attitudes and behavior? Research outcomes were used to identify relationships between measures of attitudes, behaviors, and skills, which are typically gathered via attitudinal surveys, written skills tests, or graded exercises.
Results and Conclusions: Results provide useful information about the efficacy of instructional evaluation methods, including showing significant disparities between attitudes, skills, and information usage behaviors. This information can be used by librarians to implement the most appropriate evaluation methods for measuring important variables that accurately demonstrate students' attitudes, behaviors, or skills.
In academic and professional environments that are increasingly evidence based and outcomes driven, librarians are likewise called upon to provide tangible evidence that instruction in information literacy (IL) skills is valid and legitimate. Legitimacy is usually quantified through a hierarchy of expected educational standards and outcomes. These include the impact of instruction on students' information retrieval skills, course grades, IL skills, and achievement of program and national standards. The assessment of student learning is important for demonstrating academic achievement and program success, particularly in the context of increasing tuition costs. Measuring student learning outcomes is a process that necessitates sequential and systematic evaluation of library training, workshops, or courses.
Quality educational program evaluation includes both quantitative and qualitative measures. Many commonly used evaluation tools such as selfreported perception surveys focus solely on students' perceptions of their own skills, their knowledge, or the library. These are important areas of investigation, for user satisfaction and self-confidence are significant factors in understanding students' attitudes about libraries and information. At the same time, a balanced approach is ideal, requiring librarians to distinguish between and appropriately apply effective evaluation measures to provide evidence of both attitudes and actual learning outcomes.
The authors surveyed the academic library literature on student learning assessment from 2007 to 2012 to uncover meaningful relationships between various measures of learning for assessing knowledge, skills, attitudes, or behaviors. Several questions related to the evaluation of library instruction were asked:
1. What evaluation measures are commonly used?
2. What are the pros and cons of popular evaluation measures?
3. What are the relationships between measures of skills versus measures of attitudes and behavior?
This review includes the articles that specifically address the use of evaluation measurements and instruments to assess IL skills or the impact of IL training on student learning. The review is not limited to the biomedical journal literature, but also includes literature from the arts, humanities, and social sciences. The resulting sample includes articles published since 2007 that report on the use of measures of performance (tests, products, portfolios), attitude (surveys), and behavior (in relation to the use of library resources). …