The unanticipated death of a healthy research study volunteer at a major university highlighted the importance of a comprehensive and exhaustive literature review in conducting responsible research [I]. When this unfortunate event was investigated, it became evident that many researchers who perform literature searches have little guidance about what constitutes an appropriate or sufficient search to support human subject research. The researcher in that study had completed a basic search of MEDLINE and recent texts but had not conducted an extensive search, thereby missing relevant published research [2, 3]. After many of its members conducted literature searches themselves, the committee assigned to investigate this event found it was divided on what constitutes a sufficient literature search to support human subject research .
The Medical Library Association (MLA) recommended development of standards for literature searching to help institutional review boards assess the quality and comprehensiveness of literature searches in studies under review, and the association developed a policy statement on the role of expert searching in the health sciences [5-7]. Ln an article published long before this unfortunate event, Cullen and Mason decried the lack of standards for health sciences literature searching:
medical researchers are not required to account for the rigor of the literature review in the same way as they are required to account for experimental design, methodology and accurate reporting of findings in the research report which it precedes. They are trained in science but not in the bibliographical structure of their own discipline. 
To address this educational need, the Health Sciences Library System (HSLS) at the University of Pittsburgh developed "Responsible Literature Searching for Research: A Self-Paced Interactive Educational Program (RLS)" in 2003, with funding from the Association of Academic Health Sciences Libraries (AAHSL), through a cooperative agreement between the Association of American Medical Colleges (AAMC) and the Office of Research Integrity (ORI), US Department of Health and Human Services . The project was proposed in response to a request for applications from ORI and AAMC for initiatives to promote the responsible conduct of research, including development of an online curriculum module.
RLS is an instructional tool aimed at teaching clinical researchers the fundamentals of effective literature searching for research practice. The program is self-paced, allowing users to work through the program with no restrictions on pace or order. The program consists of nine chapters with didactic information, practice guidelines, examples, and supplementary materials, and it includes tests and quizzes to measure knowledge gained. The RLS program objectives are to teach and describe generally accepted practices and principles associated with the biomedical literature search process. These practices and principles include appropriate use and limitations of major information resources, guidelines for responsible literature searching, HSLS resources and services available to support the literature search process, role of reference librarians in this process, and best approaches for specialized topics such as drug safety and identification of adverse events.
Because the module is based on principles applicable in most research-oriented settings, the cooperative agreement with AAMC/ORI specified that the content be freely available to other academic health sciences libraries and medical centers to adapt to their local settings and resources.
An advisory committee of experienced medical reference librarians · from major institutions (Yale University, University of North Carolina at Chapel Hill, George Washington University, University of Wisconsin-Madison, and the Medical College of Georgia libraries), as well as eight HSLS reference librarians, evaluated the RLS content and test questions before deployment. The feedback received from these experienced librarians indicated they felt the program was a thorough review of searching philosophy and technique, and they approved of the depth and breadth of content and the logical flow of materials. ORI invited HSLS to demonstrate the program during the "Responsible Conduct in Research Exposition" at the annual meeting of the Society of Research Administrators International, held in Pittsburgh, October 2003 . More than fifty attendees, including professional staff from ORI's Division of Education and Integrity, reviewed the draft content and provided positive feedback about the usefulness of this type of program. Ten clinical researchers from the University of Pittsburgh also reviewed the program and completed the quizzes on a pilot basis. They provided detailed feedback on content applicability, clarity, comprehensiveness, usefulness, and ease of use. RLS was formally introduced at the AAMC annual meeting and the 2005 MLA annual meeting [11, 12].
RLS was first made available in January 2004 through the University of Pittsburgh's web-based institutionwide responsible conduct of research (RCR) framing program, "Research and Practice Fundamentals (RPF)" . In May 2006, the RPF site was relaunched as Internet-based Studies in Education and Research, University of Pittsburgh Health Sciences eLearning Environment
Since implementation in 2004, 1,653 unique users have successfully completed the RLS module and received certification by scoring 80% or higher on its test or quizzes. Between implementation of the module in 2004 and its move to another system in 2006, 478 users completed the module. Between May 2006 and February 2009, 1,175 users completed the program. As with other modules in the RCR, users are required to complete an evaluation at the end of the module before they receive certification. When the module was moved in 2006, the evaluation form was significantly changed, and thus the data between the 2 evaluation periods cannot be compared. The results reported here are for the 1,175 users completing the module between May 2006 and February 2009.
The majority of the users (1,055) were affiliated with the University of Pittsburgh or were UPMC users, and 80 were from other institutions. Users had the option to obtain certification by taking the module test without reviewing the module's content. Of the 1,175 users, 925 (79%) users attempted the module test without reviewing the module's content. They either felt that they already had enough knowledge to pass the test, or they decided to take a chance on passing. Of those, 300 (25%) were certified through this "test out" feature.
The evaluations at the end of the module included 7 items designed to assess users' perceived outcomes as a result of completing the module. These questions were modeled after those used in the other RCR components. The responses from users were very positive (Table 1). Approximately 80% of users responded that the degree to which they expected to change the way they conducted their professional responsibilities as a result of completing the module was moderate, high, or very high. The same was true about the way users thought about their professional responsibilities. Eighty percent responded that reviewing the module moderately, highly, or very highly changed how they thought about their professional responsibilities. Approximately 90% of users responded that the module moderately, highly, or very highly enhanced their knowledge of the subject area, and 92% responded that the content was relevant to their professional responsibilities. Ninetytwo percent responded with "moderate," "high," or "very high" that the objectives of the module had been achieved. Only 10% responded that there was high or very high evidence of commercial bias in the presented materials.
The 7th item in the evaluation, not included in Table 1, was an open-ended question asking users if they could make any changes to the module what would they be? Only 15% (174) of the users completing the evaluations answered this question. Of those, 57% (99) recommended no changes to the module. The other comments included shortening the module, including fewer questions, mcluding more readings, and adding video or color.
Ten health sciences libraries requested RLS content to adapt to their local settings. Only two, Weill Cornell Medical Library and the Greenblatt Library of the Medical College of Georgia, have completed the adaptation at the time of this writing and made it available to their users. The module design made it easy to adapt to other institutional settings. The RLS content is given to requesting institutions in Microsoft Word files, with the quiz questions and answers in a Microsoft Excel file. The only change required is to match the resources and names to that of the individual institutions.
In April 2006, RLS was accepted for inclusion in the AAMC Providing Online Resources to Advance Learning in Medical Education (MedEdPORTAL). The MedEdPORTAL is a web-based resource designed to serve as a prestigious pubMshing venue, through which faculty and medical schools can disseminate peer-reviewed educational materials . Submissions go through peer review using a standardized instrument developed by the AAMC Peer Review Task Force. The instrument evaluates numerous educational dirnensions of the submitted resource, based on accepted standards of educational scholarship.
RLS was developed to teach clinical researchers the fundamentals of effective literature searching for research practice. The evaluations from users clearly show that those who completed the module felt it enhanced their knowledge of the subject area. They also indicated the content was relevant to their professional responsibilities and that they expected to change the way they thought about and conducted these responsibilities as a result of their gained knowledge.
One limitation to the results should be noted. Participants were required to complete the evaluation to obtain certification. This mandatory evaluation system was employed because it matched the evaluations in the other modules in RCR and would be familiar to users of this system. There are limitations in using mandatory evaluations as they are prone to bias in reporting by the response group [16, 17].
Because the module was written with a broadbased approach to literature searching, minor content revisions occur yearly. These revisions are tracked by the author throughout the year, requiring a small amount of effort in updating the RLS content.
The positive feedback from users of the RLS module suggests that this type of instructional tool can serve as a model for similar educational programs at HSLS and in other institutions. Future research should continue to identify opportunities to measure the impact of web-based learning formats for instruction.
The authors thank Ratna Jain for her assistance with the evaluations.
* Funded by the Association of Academic Health Sciences Libraries through a cooperative agreement between the Association of American Medical Colleges and the Office of Research Integrity of the US Department of Health and Human Services.
* This article has been approved for the Medical Library Association's Independent Reading Program
1. Steinbrook R. Protecting research subjects: the crisis at Johns Hopkins. N Engl J Med. 2002 Feb 28;346(9):716-20.
2. Perkins E. Johns Hopkins tragedy: could librarians have prevented death? Inform Today [Internet]. 2001;7 Aug [cited 24 Feb 2009].
3. Savulescu J, Spriggs M. The hexamethonium asthma study and the death of a normal volunteer in research. J Med Ethics. 2002;28(l):3-4.
4. Dang CV, SchafferGregory F. (Johns Hopkins University). Letter to Michael A. Carome (Office for Human Research Protections) 13 Jul 2007 [Internet], [cited 17 Jun 2009].
5. McLellan F. 1966 and all that - when is a literature search done? Lancet. 2001 Aug 25;358(9282):646.
6. Medical Library Association. Recent tragedy underscores vital role of medical librarians [Internet]. Chicago, IL: The Association; Aug 2001 [cited 17 Jun 2009].
7. Medical Library Association. Medical Library Association policy statement: role of expert searching in health sciences libraries [Internet]. Chicago, IL: The Association; 4 Sep 2003 [cited 17 Jun 2009].
8. Cullen R, Mason D. Quality assurance in health sciences literature searching: applying the ISO 9000 quality standard. Health Libr Rev. 1995 Sep;12(3):173-89.
9. Office of Research Integrity, US Department of Health and Human Services. AAMC funded projects for RCR 2003 awards [Internet]. Rockville, MD: The Department; 30 Jan 2007 [cited 10 Sep 20091-
10. 19 exhibitors set for RCR Expo in Pittsburgh. Office Res Integrity Newsl [Internet]. 2003 Sep;ll(4):5 [cited 17 Jun 2009].
11. Wessel CB, Tannery NH. Responsible literature searching for research: a self-paced interactive education program. In: Association of Academic Health Sciences Libraries, Innovations in Medical Education Exhibits, 29th Annual Session, 2004 Annual Meeting Program. Sponsored by the Group on Educational Affairs, Association of American Medical Colleges, at the 115th Annual Meeting of the Association of American Medical Colleges, p. 36.
12. Wessel CB, Tannery NH, Epstein BA. Responsible literature searching for research: a self-paced interactive education program. Poster presented at MLA '05, the 105th Annual Meeting of the Medical Library Association; San Antonio, TX; 17 May 2005.
13. Barnes BE, Friedman CP, Rosenberg JL, Russell J, Beedle A, Levine AS. Creating an infrastructure for training in the responsible conduct of research: the University of Pittsburgh's experience. Acad Med. 2006 Feb;81(2):119-27.
14. Health Sciences Library System, University of Pittsburgh. Responsible literature searching for research [Internet]. Pittsburgh, PA: The University; 1996-2009 [cited 17 Jun 2009].
15. Wessel CB. Responsible literature searching for research: a self-paced interactive program [Internet]. MedEdPORTAL, Association of American Medical Colleges; 2005 [cited 17 Jun 2009]
16. Howard GS. Response-shift bias: a problem in evaluating interventions with pre/post self-reports. Eval Rev. 1980 Feb;4(1):93-106.
17. Donaldson SI, Grant-Vallone EJ. Understanding selfreport bias in organizational research. J Bus Psychol. 2002 Winter;17(2):245-60.
Charles B. Wessel, MLS; Nancy Hrinya Tannery, MLS; Barbara A. Epstein, MSLS, AHIP
See end of article for authors' affiliations.
Charles B. Wessel, MLS, email@example.com, Head of Hospital Services; Nancy Hrinya Tannery, MLS, firstname.lastname@example.org, Associate Director for User Services; Barbara A. Epstein, MSLS, AHIP, bepstein@pitt .edu, Director; Health Sciences Library System, University of Pittsburgh, 200 Scaife Hall, 3550 Terrace Street, Pittsburgh, PA 15261
Received March 2009; accepted July 2009…