Technology Usability Issues in a State Government Voice Mail Evaluation Survey

Beginning of article

This paper summarizes results of a user evaluation survey of computer-based telephone messaging ("voice mail") technologies. A mail survey of the population of voice mail users in state government agencies approximately 3000 users) yielded 1072 user responses. Survey analyses tested hypotheses regarding how voice mail evaluations are influenced by user characteristics, system use and experience, task and situation factors, and differences in organizational context and operations. Highlights of the results include findings relating both endogenous (age and gender) and exogenous (reported length of experience and average weekly use) user characteristics to ratings of the voice mail interface. Participating organizations reported mean differences in ratings of the interface and situation-based voice mail appropriateness. The findings emphasize the importance of a systems approach to integrate user-centered design and functional, situational, and organizational contexts in order to enhance the success of implementing voice mail and other telecommunications technologies.

INTRODUCTION

What is the meaning of a successful application of a new technology? The mere existence of the technology does not ensure its usefulness or appropriateness for improving the quality, efficiency, or range of tasks that a user may wish to accomplish. For instance, consider a state agency trying to increase the computerization and exchange of agency records by purchasing and distributing a large number of computers. From a purely administrative standpoint, the technology diffusion is complete when the boxes containing the computers arrive and are unpacked. However, if the computers are not used as planned and users instead use the computers only to perform word processing, is this a successful technology implementation? This issue of organizational integration and diffusion of new technologies lies at the heart of the study of implementations of new communications technology presented in this paper.

Some information technology applications and implementations may be particularly suitable for wide diffusion and acceptance in a variety of organizations, based on their easy integration with existing tasks and potential for fulfilling the unmet needs of an organization. One example is electronic voice messaging, or "voice mail." Although voice messaging has been available for decades in the form of answering machines, it now provides enhanced capabilities for sending multiple messages to a number of users simultaneously; for simplified recording, retrieval, and message storage; and for increased variety of sending and receiving options. Unlike prior answering machine systems, current generations of voice mail systems may provide numerous menu options available through the telephone interface.

The continued use of the telephone interface, already established in most organizations, may be one reason that modern voice mail systems are so easily integrated into organizations. Previous research has demonstrated that the acceptability and success of any information technology system (or, indeed, any innovation) is dependent in part on situational, organizational, and technological variables (Adrianson and Hjelmquist, 1991; Allen and Hauptman, 1987; Lea, 1991; Miller, Davis, and Elio, 1990; Rice, Grant, Schmitz, and Torobin, 1990; Rice and Shook, 1988; Rogers, 1983). Therefore, these variables should be included in the discussion of the ability of voice mail in general, or of a particular voice mail product, to meet the needs and task demands of an organization.

Usability, Organizational Context, and

System Design

A number of researchers have remarked on the low success rate of organizational implementations of new information technology systems. Some estimates suggest that only 2001o of new systems show real improvements in productivity and organizational functioning (Eason, 1988). Several reasons are given for this low success rate, including a lack of technology usability, failure to match the technology with the existing organizational context, or a failure to appreciate the needs of the potential users in system design and application (Davis, 1989; Heller, 1989; Markus and Connolly, 1990; Rouse, 1990; Schein, 1990; Sproull and Kiesler, 1991). Factors other than initial technology development can interfere with effective application of the technology and limit the cost-effectiveness of the technology (Eason, 1991; Karat, 1990, 1992).

Even an issue such as technology usability has several levels of meaning and importance in the implementation of new technological systems. Many systems designers consider usability issues primarily in terms of software interface designs. However, evidence suggests that this type of usability is insufficient for the broad diffusion of a new technology. Several researchers have identified cases in which otherwise usable information technologies were not fully implemented because of a lack of limited organizational utility (see, for example, Davis, 1989; Maryniak [Nelson] and Caldwell, 1992).

The differences in mission, culture, and patterns of activity in organizations lead to tremendous differences in attitudes and approaches to technology (Daft and Lengel, 1986; Eason, 1988; Kraut, 1987; Schein, 1990; Sproull and Kiesler, 1991; Turnage, 1990). Some of these differences are based on differences in situation demands from one organization to another even in the same industry, or on the organization's local environmental conditions (Emery and Trist, 1965; Lea, 199 1; Malone and Rockart, 199 1; Rice et al., 1990; Rogers, 1983; Sage, 1992).

Organizational validity -- the fit between a technical system and its organizational context of use -- can affect users' attitudes toward system changes and the ease of technology implementation (Agervold, 1987; Markus and Robey, 1983; Miller et al., 1990; Rouse, 1990). The requirements for an information technology system design to match the demands of the organization focus on identifying the existing and future needs of the information technology users in the organization. This approach, often described as user-centered or human-centered technology design, has been shown to be effective in a number of manufacturing, industrial, and other settings (Crow, 1992; Eason, 1991; Karat, 1992; Rosenbrock, 1989; Rouse, 1990).

METHOD

Responding to complaints about a lack of responsiveness to organization and user needs, the Bureau of Information Technology Management (BITM) of a Midwestern state government began a test of the electronic voice mail systems being offered to state government offices. The research project began when BITM consulted with the first author regarding a user-centered survey evaluation of the voice mail systems. These discussions, which began in February 1992 and continued through the summer and fall of 1992, resulted in a survey set of software usability and organizational utility …