Kicking the Tires: The Fine Art of CD-ROM Product Evaluation
King, Alan, Online
How many of us buy a new car without first taking it for a thorough test drive? How many of us buy a new television set without consulting Consumer Reports to find the most highly rated model? The answer to both these questions is - very few. As a nation, we pride ourselves on being educated and informed consumers. We scrutinize advertisements, comparison shop, and solicit the advice of a variety of publications and experts. Yet in the marketplace for library products, these very same skills often atrophy or disappear.
Buying CD-ROMs is certainly no exception. Since I began to write this column for ONLINE, I have been collecting my colleagues' comments on various CD-ROM products. Among the most common complaints is a general disillusionment with a product's performance: "It doesn't really do what I thought it was going to do when I bought it." This comment, rather than being an indictment of the product is actually the sign of a poorly-made purchasing decision. What is it about the library market that can cause rational (more or less) individuals to bypass common sense and buy without adequately reviewing a product? Part of the problem may be our unfamiliarity with the product we want to purchase. Very few of us would consider our-selves to be computer or CDROM experts. The jargon alone is enough to send a sane person over the edge. However, we often find our-selves in the position of not only making a decision about which CD-ROM database to purchase, but also choosing the hardware on which that product will run.
Compounding this problem is, the fact that the tools we have come to rely on in the general consumer market do not adequately exist for library products. Despite the variety of excellent CDROM reviews available in our professional press, there is no Consumer Reports for CD-ROM - no single source for CD-ROM product information.
Without adequate knowledge and with limited access to consumer information, we must rely on our own judgement to make the right choice. In that light, the need for a comprehensive and systematic method for evaluating potential products is essential. In the remainder of this column I will outline the basic foundation for planning and executing a CD-ROM product evaluation. WHY TEST?
To summarize the soapbox rhetoric above, the obvious reason to test a product is to maximize consumer satisfaction - to make sure that the buyer is happy with his/her purchasing decision. However, the benefits of evaluation can go far beyond a simple checklist of product feat characteristics. An effective test 1 shed light on the need for and extent of staff and patron training; on consumer (patron) acceptance of the product; and on the potential impact of the product on other information center resources (interlibrary loan, circulation, and collection development). Testing can also offer insight into patron preference for information access and delivery, and can point out directions for marketing the product within your institution.
WHAT TO TEST?
Product: Testing the actual product involves many steps. The first is to compare the advertising claims with reality. Take the checklist of features supprisied in the manufacturer's literature and compare them against the actual product. Do they match up?
Now look at the other characteristics of the database. Examine the search interface. Is it easy to understand and use? Does it allow for Boolean searching? Is the interface menu-driven, command-driven, or both?
Look at the quality of the information. Are abstracts well-written and informative? Are bibliographic citations complete and uniform? Are there numerous typographical errors? Is the indexing adequate and consistent?
Examine the system help files. Are they adequate and do they really help? Does the product manual offer clear explanations about features? Does the manual supplement or complement help files?
Print some search results to test that feature of the product. …