"Linkrot" and the Usefulness of Web Site Bibliographies

Article excerpt

This article examines whether outdated URLs are still a problem and to what extent they decrease the usefulness of paper bibliographies and reviews of Web sites. The authors make specific recommendations that bibliographers and Web site authors can use to increase the probability of users finding Web sites that may have relocated, thus improving the long term utility of Web site bibliographies.

A traditional role of librarians has been to collect, evaluate, and annotate information on a given topic and to produce bibliographies and reviews. With the birth of the Internet and World Wide Web, librarians have expanded this role to include collecting and evaluating Web sites. Annotations of Web sites now appear in professional journals for librarians, in traditional librarian-oriented review sources (such as Choice and Library Journal), in such electronic formats as the Internet Scout Project and Librarians' Index to the Internet, and in virtual libraries, including INFOMINE and Argus Clearinghouse.(1) Librarians also publish subject bibliographies that appear in professional journals of other disciplines.

Paper-based subject bibliographies of Web sites have several benefits, including their use as "ready-reference tools for librarians and information specialists," even though these lists, like traditional bibliographies, are usually out of date by the time they are published.(2) An additional problem exists for Web site lists: while traditional bibliographies rarely give the precise location of the resource, the addresses of resources on the World Wide Web (URLs) are included and can change. When Web pages are removed or moved by their creators without a "forwarding address," the URLs given in the bibliography are no longer correct, a phenomenon known as "linkrot." When a link expires, users have to wait until search engines update their databases, unless the new URL is posted at the old address or the user is automatically sent to the new site.

How pervasive is linkrot in this environment? In 1998, Benbow examined two of her previously published Web resource journal articles and found that 50 percent of the URLs selected in mid-1995 and 19 percent of the URLs selected in mid-1996 were no longer accurate.(3) Kitchens and Mosley examined URLs in selected Internet guide books and found that "on average, 29 [percent] of URLs were inactive within two years of publication."(4)

Is linkrot still a significant problem in paper bibliographies of Web resources? To determine this, the authors examined Web site annotations that appeared in the "Internet Resources" column in College & Research Library News over a twelve-month period. In addition to reporting the results of the inquiry, we will offer suggestions and strategies for both bibliographers and Web site authors that may make bibliographies more useful on a long-term basis and decrease linkrot.

Method

The authors examined the URLs and annotations of Web sites (those beginning with "http://") in "Internet Resources" columns from October 1997 through October 1998. The analysis had two main goals: to determine how many URLs were outdated and to determine if the annotations provided enough information to retrieve the URLs using a meta search engine. The original sample included 510 URLs. After we excluded obvious typographical errors (for example, "ww." instead of "www.") and duplicates, the final sample contained 482 URLs. These sites were tested during the month of October 1998 to determine whether the URLs were still active. URLs that were active in October 1998 were retested in May 1999 to see if they were still in use.

If the URLs had expired, the authors documented if and how users were directed to the new locations, including instantly taking users to the new location, listing the new URL on an error page (for example, "404 File Not Found"), or providing site maps or site search functions on the error page. …