Twelve Point Plan for Quantitative and Qualitative Evaluation of Databases
Miller, Ron, Computers in Libraries
The assessment of a database's value to your library is not a simple matter. A thorough evaluation must account for many variables in order to arrive at a true measure of database value. Many vendors promote a simplified quantitative approach: calculate the cost per full text title and find out who gives you more full text titles for your money. Simple enough, but a bit too simple. There is much more to factor into the equation. We wouldn't choose our print reference purchases on a "cost-per-page" basis, so why should the database purchase be reduced to such a simplistic calculation? The quality of the library patron's user experience and the number of truly relevant results retrieved must weigh significantly in any evaluation of database value to a library.
Twelve points to consider:
Quality of Content
Quality of content must enter the calculation of database value.
Ask how are journals selected for inclusion in the database. Are journals selected by recommendation from a panel of subject experts and experienced librarians, or is there an "include everything but the kitchen sink" approach to create inflated full text numbers?
Peer-review numbers can be deceiving. Look for a database vendor that sets an unimpeachable standard when deciding which publications qualify to be designated as peer-reviewed. The database publisher should guarantee that the peer-review label means literally that an independent scholar has reviewed and recommended the article for publication. Peer-review must not be applied simply to any journal that is not a popular or trade title.
Titles Appropriate to Database Subject
Are the titles selected for inclusion in the database really relevant to the subject area? The presence of extra titles in a full text list does not provide a better user experience, rather it will tend to provide the highly unsatisfactory user experience of searching for journal articles and receiving a results set filled with inappropriate articles or even government produced pamphlets masquerading as full-text journals. Ask yourself if your library patron will be satisfied when a search of what is advertised as a complete education resource brings back articles from the likes of National Tax Journal, or Omega: Journal of Death and Dying.
Is there a controlled vocabulary based on the literary warrant of the actual collection of items in the database, or has a database aggregator simply adapted a "one-size-fits-all" vocabulary specific to nothing and of little true value in bringing users all of the available items on a topic?
Relevance Ranking That Works
Is there a relevancy ranking option available, and does the relevance-ranking algorithm understand that words found in fields like subject and abstract are more indicative of the "aboutness" of an article than random occurrences elsewhere in the article?
Search and System Efficiency
Does the database require executing a second search in order to sort the articles by relevance? This provides impressive usage statistics that make the database appear to be your most popular database, but does your patron really want to execute two searches when one could bring the best results? Ask yourself if those impressive statistics are truly a measure of database value, or more likely an indication of database inefficiency.
Ease of Access to Best Results
Does the database give library patrons quick and easy access to the information that they seek whether they are novice searchers or experienced professionals? Does the results set come back sorted only by date of publication in reverse chronological order, or are they sorted with the goal of giving users the best articles available, automatically placed at the top of the results set in one search? …