Academic journal article Information Technology and Libraries

The Evaluation of a User-Oriented System: The MELVYL System's Many Designers

Academic journal article Information Technology and Libraries

The Evaluation of a User-Oriented System: The MELVYL System's Many Designers

Article excerpt

As the MELVYL system has matured, its designers have employed numerous ways of discerning users evaluation of the system, ranging from online user comments to formal advisory groups. This article describe how these mechanisms have evolved and how they are used by the designers, and discusses the rising expectations of users.

As the recent winter Olympics drew to a close and the commentators reminded us that the summer Olympics were a few months away, they frequently quoted the simple but powerful motto of the games: "faster, higher, stronger." This resounding phrase seemed to echo and eventually shift to "faster, bigger, farther" as I reviewed user comments, answered phone calls, and responded to electronic mail about the MELVYL system recently. The MELVYL catalog has grown from a database of approximately 750,000 records to a system of nine databases and a gateway to almost forty other systems, and the expectations of its users have grown along with it. Each new feature or addition elicits an initial euphoria followed shortly by the "why can'ts"--why can't it be faster, bigger, and go farther. As other authors in this issue have pointed out, the system's biggest fans are usually its most demanding critics.


As we, the designers, plan and discuss each change or new capability, we try to second guess the needs, preferences, and potential sources of confusion of the user. These discussions are often intense, sometimes even emotional, since everyone believes he or she knows what the user wants. After all, everyone is a user and therefore an expert. Ultimately, each controversial topic is pushed, shaped, wheedled, tweaked, tuned, and adjusted into an amalgam of the best ideas or at least the best compromises. These diverse viewpoints are a necessary part of the process, reflecting in part the diversity of the real user population. The original design team recognized the need for a broad view when they stated their first guiding principle for the development of the catalog: "The catalog must accommodate the entire user population ranging from experienced searchers to first-time library patrons." (1)

The designers went on to state specific guidelines for designing the MELVYL system's user-friendly patron interface. (2) These guidelines are still being followed and are the foundation for new guidelines to expand the user interface to take advantage of graphical capabilities in the future.

The real definition of what is user-friendly comes not from a set of guidelines, but from the users themselves, of course. Again, the early designers anticipated the need to record and evaluate not only what users say but what they do. They employed the one-way mirror of the behavioral scientist in the form of transaction logs to observe users actions. Users remain anonymous in the logs, but identification numbers are assigned to terminal locations so that the sequential activity of a user's session can be recorded. The logs were part of the initial evaluation of the system that was funded by the Council on Library Resources. (3) The logs also have been used to answer specific questions, such as whether users were searching for corporate authors when employing the original Author index (which combined personal and corporate author indexes), as well as to capture entire sessions for analysis of unexpected difficulties.

Two additional tools that were part of the . original plan are more similar to those used by market researchers. Online questionnaires, akin to market surveys, were used to evaluate user reactions to the prototype, (4) and later to compare users of the MELVYL MEDLINE and CURRENT CONTENTS databases. (5) Users are not forced to take these surveys and are allowed to stop at any time. Response rates are not as high as those for questionnaires administered in person, but they are substantial enough to provide a snapshot of user characteristics and stated reasons for using the databases. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.