Academic journal article Reference & User Services Quarterly

Measuring the Completeness of Reference Transactions in Online Chats: Results of an Unobtrusive Study

Academic journal article Reference & User Services Quarterly

Measuring the Completeness of Reference Transactions in Online Chats: Results of an Unobtrusive Study

Article excerpt

This article reports on the results of an unobtrusive study used to measure whether online chat reference can effectively answer short, subject-based research questions. The study develops measures for analyzing how completely these questions are answered and uses them in combination with other factors in determining the overall effectiveness of the service examined. A methodology for developing these new measures is explained and the results are studied in relation to other traditional methods of reference evaluation.


One of the issues librarians wrestle with as they establish and develop chat reference services is deciding what level of service they want to provide to their users. Should chat reference answer everything that in-person reference does? Should it be restricted to certain types of questions, such as ready reference or library policy-related issues? This last question is key when determining policies for virtual reference services, and then defining staff guidelines and parameters for answering questions.

Once these policies have been put into practice, as Meola and Stormont suggest, their impact and success should be measured through the various means of reference evaluation. (1) This analysis can take many forms, from looking at overall question accuracy, to user satisfaction, to individual components of the service such as length of chats, time asked, and so on. The various types of questions that patrons may ask necessitate different measures when looking at the quality of response given by library staff.

In the academic library environment, questions regarding how to begin term paper or subject based research topics have long been standard fare at the reference desk. In the virtual environment, Diamond and Pease defined eleven separate categories of questions for their study of user questions types, and found that "Starting-points for term papers and assignments" was the second most frequent type of question asked. (2) In discussing user service expectations, they go on to note that "Academic libraries should be prepared to answer a full range of questions ..." and more specifically, "Limiting digital reference service to 'ready reference' questions alone does not adequately meet users' needs and may not even be understood by them." (3) Knowing that users are asking these types of questions necessitates devising means and measures for analyzing how well librarians are providing answers to them.

One of the main concerns when the University of Illinois library began its virtual reference service was whether the limitations of the chat medium would make answering research questions for papers and speeches too awkward or unwieldy to be useful to the user. In practice at the traditional reference desk, answering these questions often involves explanations of the way information is organized, the appropriate tools to use (for example, online catalogs vs. article databases), as well as analysis and explanation of the results from searches. When the library's chat service was established, it was decided to try to answer this type of question, with an eye toward later evaluation. The study presented in this article used unobtrusive observation methods to analyze how completely short, subject-based research questions were being answered in the chat environment and whether the service was measuring up to initial policies and expectations.

Research Questions and Goals

For the purposes of this study, a "subject-based research question" refers to a question in which the student wishes to find a particular kind and number of sources (either books or journal/magazine articles) on a specific topic. The primary goal of the study was to determine if subject-based research questions could be answered online with an equivalent level of detail and completeness as is expected of similar in-person reference transactions.

Research Questions

Research questions for this study included:

* How completely are users' subject-based research questions being answered? …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.