The Past, Present, and Future of "IS Success"
Petter, Stacie, DeLone, William, McLean, Ephraim R., Journal of the Association for Information Systems
Since the introduction of information systems more than 60 years ago, organizations want to ensure that their systems are effective or "successful". Much has changed in the evaluation of information systems success during this period. The role of information systems in organizations has changed dramatically, as have the key stakeholders and the expected benefits of the investments in IS. During this period, IS research has evolved to keep pace with the changing expectations regarding the success of information systems, yet practice tends to lag behind. In this commentary, we discuss five eras of information systems evolution and explain how the perceptions and measures of successful information systems have changed across these eras. By looking at the past and present, we are able to comment on how our understanding of success has evolved over time in research and practice. We discuss the inadequacy of IS success evaluation in practice. Finally, we offer four themes as calls for future action related to the research of information systems success.
Keywords: IS Success, IS Effectiveness, Eras of IT, IS History
"Success" is achieving the goals that have been established for an undertaking (Anonymous).
Regardless of whether the economy is booming or busting, organizations want to ensure that their investments in information systems (IS) are successful. Managers make these investments to address a business need or opportunity, so it is important to identify whether the systems meet the organization.s goals. Keen (1987) described the mission of IS as:
The effective design, delivery, use and impact of information technologies in organizations and society. The term "effective" seems key. Surely the IS community is explicitly concerned with improving the craft of design and the practice of management in the widest sense of both those terms. Similarly, it looks at information technologies in their context of real people in real organizations in a real society (p. 3).
Based on Keen.s view of information systems, we believe the evaluation of the "effectiveness" or "success" of information systems is an important aspect of the information systems field in both research and practice. However, the manner in which we evaluate the success of an information system has changed over time as the context, purpose, and impact of IT has evolved. It is, therefore, essential to understand what these changes have been and what they mean for the future.
Information systems success research evaluates the effective creation, distribution, and use of information via technology. As information technology has developed since the mid-1950s, information has become more voluminous, more ubiquitous, and more accessible by all. If we believe that information is power, this progress in information availability has changed the power dynamics of relationships between corporations and consumers, between buyers and suppliers, between small business and large business, and between citizens and their governments. Thus, the measurement of IS success has become ever more complex while, at its core, still simple.
The complexity arises because the uses and users of information systems are ever expanding. Therefore, the context has infinite possibilities in terms of the purpose of an IS and the definition of its stakeholders. Yet the measurement of information systems success at its core is still simple because there are consistent key elements in the measurement of success, such as information quality, system quality, use, and outcomes. The challenge that researchers and practitioners face today is, as the sophistication of information systems and their users increases, we can lose sight of the basics. Relevance, timeliness, and accuracy of information are still key to IS success, even as our information systems, and the measures of these systems, grow increasingly more complex.
This commentary explores the history of our understanding of information systems success. We examine how the concept of effective or successful information systems has progressed as information technology and its use has changed over the past 60 years. We identify five eras or periods of information systems; and for each of these eras, we consider the types of information systems used in firms, the stakeholders impacted by these systems, the relevant research about information systems evaluation, and the measurement of IS success in practice during each of these periods. Based on the evolution of the field.s understanding of IS success, we offer suggestions on what this means for on- going research and practice. We share our insights on the current state of IS success measurement in practice and present four themes or "calls for action" on the future of IS success research.
2. The Changing Nature of IS Success
In reflecting on our understanding of information systems success as a research community, we sought to identify a framework to explore our past and gain insights for our future. In prior discussions of the history or foundations of information systems, many researchers have proposed various approaches or frameworks to describe the history of information systems. Some frameworks identify eras by looking at shifts in research (Hirschheim & Klein, 2011). Other frameworks focus on the information technology artifact by identifying changing features of the hardware, such as the mainframe and minicomputer era, the personal computer era, the client/server era, the internetworking era, and the cloud computing era (Dahlbom, 1996; Jessup & Valacich, 1999; Laudon & Laudon, 2010). Still other approaches focus on the applications or types of information systems that dominated a period of time, such as data processing, functional systems, management reporting, decision support systems, and enterprise systems (Kroenke, 2007; O'Brien, 1999).
In this work, we take the last approach and organize our discussion of the history of IS success using eras that characterize changes in the applications and use of information systems over time, similar to Kroenke (2007) and O.Brien (1999) cited above. By considering how information systems have evolved over time, we can see how the definitions and views of success have changed as well.
For purposes of this commentary, we propose five eras of information systems, based on frameworks developed by Dahlbom (1996), Kroenke (2007), and O.Brien (1999). In particular, we rely heavily on the framework developed by O.Brien, but make our own adaptations based on our insights and interpretation of the history of information systems success. Our five eras of information systems include:
* The Data Processing Era (1950-1960)
* The Management Reporting and Decision Support Era (1960-1980)
* The Strategic and Personal Computing Era (1980-1990)
* The Enterprise System and Networking Era (1990-2000), and
* The Customer-Focused Era (2000 and beyond).
It is neither possible nor reasonable to definitively attribute exact dates to each of these eras; however, the inclusion of date ranges can be useful to those trying to attribute the advances in technology to the progression of information systems. Generally, the date ranges we attribute to these eras are consistent with the dates identified by Kroenke (2007) and O.Brien (1999). These eras are neither mutually exclusive nor all inclusive, but they help us consider how information systems success has changed over time as our use of information systems has become more sophisticated, universal, and complex.
In the following sections, for each of the eras of information systems, we describe the context of the era to offer insight about how information systems were used during that time period as well as to suggest who was using the information systems at that time. With an understanding of this context, we then describe how success was viewed during the era, from both an academic and practical standpoint, based on a review of the literature for each era. We conclude each era description with a summary of how information systems success evolved by describing the profile of IS success measurement in research and practice.
2.1. Data Processing Era (1950s - 1960s)
In the initial days of computing, computers were essentially sophisticated calculators used exclusively by those in the military or in the financial industry (Dahlbom, 1996). In the military, mainframe computers were used to perform complex ballistic calculations. In the financial industry, a few banks and insurance companies used computing technology to handle the volume of calculations required by these firms and to produce reports based on these calculations. Computing technology was only used in a small segment of the industry initially, and an even smaller subset of personnel actually used the machines as part of their daily work. The number of stakeholders directly impacted by information systems was relatively small, and these individuals were highly trained and knowledgeable in computing technologies (Hirschheim & Klein, 2011). As this era progressed, information systems existed at the "periphery of management," as a tool to complete certain subtasks within work processes. For example, the development of transaction processing systems automated work processes to improve efficiency (O'Brien, 1999). However, given that the focus in this era was to automate processes, information systems did not affect management practices (Leavitt & Whisler, 1958).
During this era, research on evaluating the success of information systems was scant at best (Hirschheim & Klein, 2011). In practice, the challenge in this beginning stage of information systems was that the development process was low quality; thus, there were problems with the technical quality of resultant systems (Lyytinen, 1987). Therefore, the measure of success would primarily be based on whether or not the system met specific technical needs such as speed and accuracy. Measurement of IS success was ad hoc in practice. Some organizations used cost/benefit analysis to evaluate information systems, but this analysis was typically performed ex ante to justify committing resources to a project rather than ex post to determine if a system yielded projected benefits to the organization (Farbey, Land, & Targett, 1999).
Figure 1 summarizes the measures of IS success used during the data processing era in both research and practice. Given that information systems was a budding research domain during this area, researchers. understanding of success measures was nascent. Within practice, there was more of a concern about evaluation, but the view of success by practice was technocratic, in that the focus of success was on whether the technical requirements of the system were met. As research and practice learned more about the potential of information systems within organizations, the interest in measuring IS success began to evolve.
2.2. Management Reporting and Decision Support Era (1960s-1980s)
As computing technology advanced throughout the 1960s and 1970s, workers increasingly used computers for monitoring and controlling production and for automating administrative work (Dahlbom, 1996). Early information systems could produce structured information for routine decisions (Mintzberg, 1972), yet some managers struggled to use the information provided by these systems in their decision-making tasks (Daniel, 1961). Other managers began to see the power of information systems to make routine decisions, which reduced the need for, and the skill level of, employees (Leavitt & Whisler, 1958; Zuboff, 1988). Over time, managers and researchers began to realize that it was not a lack of information or data that prevented good decision making, but the inability of a manager to process the volume of information provided (Ackoff, 1967). Managers needed additional techniques and models to understand the information upon which they based their decisions, particularly for non-routine, more complex decisions (Gorry & Scott Morton, 1971; Mintzberg, 1972). More individuals within firms became exposed to information systems during this era, but users of these systems tended to be limited to those trained to deal with these difficult-to-use, highly technical systems (Dahlbom, 1996).
In an attempt to evaluate IS success in the workplace, researchers realized that factors beyond the technical qualities of the system were important. It became increasingly evident through research that human factors, in addition to technical factors, should be considered when evaluating the success of information systems (Bostrom & Heinen, 1977; Langefors, 1974; Mumford, 1974). The ability of individuals to use the information provided by management information systems became important to managers, since the systems produced reports needed to make key business decisions. In the Minnesota Experiments, the ability of information systems to support decision effectiveness was the ultimate evaluation of the success of an information system, considering both the characteristics of the information system and of the individual (Dickson, Senn, & Chervany, 1977).
Researchers also began to consider use as a measure of success, suggesting that any other measures of success (such as profitability, improved decision making, and so forth) ultimately relate back to system use (Ein-Dor & Segev, 1987). However, use alone was not enough, argued Ginzberg (1978), but whether this use generated change in the organization through management action or continued use of the system. Others considered additional benefits of information systems, such as reducing costs, improving accuracy, advancing management and control, or increasing the speed of operations; yet researchers acknowledge that many of these benefits of information systems were difficult to quantify (King & Schrems, 1978).
In practice, some managers were skeptical about the potential benefits of information systems in organizations. Some argued that information systems should only support data processing and should not create change in the organization.s structure, goals, or culture (Mandell, 1975), suggesting that minimal organizational impact would be one way to assess the success of an information system. Others struggled with the large amounts of paper and data that seemed to reduce rather than improve productivity; yet some managers did realize the potential of information systems to reduce headcount and lower costs (Taylor & Dean, 1966).
2.3. Strategic and Personal Computing Era (1980s - 1990s)
During this era, organizations and managers began to realize the potential of information systems to help organizations achieve their strategic goals. The concept that an information system should be strategically aligned with the organization became a strong focus for both research and practice. Furthermore, in the 1980s, the introduction of the personal computer changed the landscape of information systems use within firms. As the development and perceived value of the personal computer increased, an increased focus on personal productivity marked a profound shift in how computing was perceived and conducted. Organizations began to realize that individuals could create benefits for their companies by using information systems (Dahlbom, 1996). Also, as user-friendly interfaces became more commonplace, the introduction of personal computing became less daunting in the workplace. The use of information systems moved from the "back office" to the "front office," and information systems were more involved in the work processes of increasing numbers of employees (Rockart & Flannery, 1983). More employees were able to use information systems, and they needed less training to do so (Dahlbom, 1996).
In the quest to use information systems to support strategic goals, an important aspect of IS success is whether information systems are aligned with the strategic goals of the organization. Chief Information Officers (CIOs) identified the issue of strategic alignment as one of their major concerns about information systems in their organizations (Brancheau, Janz, & Wetherbe, 1996), and researchers developed frameworks to help organizations identify whether their information systems were aligned with the overall goals of the organization (Henderson & Venkatraman, 1999; Hirschheim & Sabherwal, 2001; Sabherwal & Chan, 2001). Therefore, during this era, alignment between the IS and the organization became increasingly important as a measure of information systems success.
Furthermore, at an organizational level, the success of information systems within an organization was considered based on the systems. ability to influence productivity, profitability, and customer surplus (Hitt & Brynjolfsson, 1995). Research measuring the organizational impacts of technology focused on the "productivity paradox," the concern that investments in information systems were not resulting in improvements in organizational productivity. Extensive research examined the question of whether IT investments had an impact on the organization (Chan, 2000). Some research used secondary data provided at the organizational and national levels to measure whether technology had produced a positive impact on the firm or nation (Chan, 2000). The use of productivity as a measure of IS success had mixed results. In response, researchers cautioned that the measure of success of the information systems being evaluated should be at the level of analysis at which the value was being realized (Chan, 2000; Rai, Patnayakuni, & Patnayakuni, 1997); meaning that if the system was created to provide benefits to a department, then the value of the system should be evaluated at the departmental level of the organization. While some researchers focused on quantifiable analyses to examine the success of information systems, others recognized the need to look at the overall value provided by the system to the firm beyond just accounting measures (Chan, 2000; Huerta & Sanchez, 1999).
Hamilton and Chervany (1981a) discussed two different ways that the success of information systems could be determined by organizations during this era. One approach to evaluating IS success was a goal-centered approach in which researchers examined the information system to see if it met the objectives specified by management. The second approach was a system-resource approach, where researchers considered the users and whether the system was serving their needs to facilitate communication, improve job satisfaction, or fulfill other needs beyond the primary organizational objectives. This approach highlighted the importance of evaluating information systems from a variety of perspectives including management, users, IS staff, and internal auditors (Hamilton & Chervany, 1981b).
Also in this era, more individuals who were not highly trained in computing found themselves interacting with information systems, through the use of personal computers. Therefore, measures considering the individual.s view of an information system became prominent within the literature. Another widely-cited method of evaluating IS success during this period was the measurement of technology acceptance. Davis (1989) developed the technology acceptance model (TAM) using the theory of reasoned action and the theory of planned behavior (Fishbein & Ajzen, 1975) to explain why users accept some information systems more readily than others. The dependent variable, or measure of success, was whether or not the user is willing to use the information system in the future. Other studies considered the individual.s use of the information system and his/her improved performance (e.g., Goodhue & Thompson, 1995).
As more employees began using information systems in companies, research related to the success of systems increasingly focused on user perceptions by asking, "Do you like it?". During this era, the user information satisfaction concept became increasingly prevalent in the literature (Bailey & Pearson, 1983; Ives, Olson, & Baroudi, 1983). The user information satisfaction (UIS) instrument that was developed at this time examined multiple facets of information systems ranging from system capabilities, to ease in understanding information provided by the system, to support from management, vendors, and IS staff. Another instrument to capture user satisfaction, the end user computing satisfaction instrument, was developed during this same period to measure IS success (Doll & Torkzadeh, 1988).
In an attempt to bring multiple dimensions of success together in a comprehensive framework, DeLone and McLean (1992) offered their IS success model, which brought together views of success from earlier eras and organized them into six dimensions of success. Previous measures of success about accuracy, processing speed, and flexibility of a system were grouped together as measures of system quality. The quality of the information provided to individuals and teams used to make decisions in terms of format, relevance, and usefulness were viewed as dimensions of information quality. The use of the information system was viewed as an important determination of success as well as the user.s satisfaction with the system. Finally, the impacts of the system to both the individual and the organization were considered the fifth and sixth dimensions of IS success. This model of IS success synthesized different measures of effectiveness that had been used in past eras and explained how these measures related to one another in evaluating an information system.
In practice, however, and in contrast to the research emphasis on impacts of IS success on organizational performance, CIOs continued to focus on measures relating to IS operations performance and development performance. Operational performance measures included system availability, user problem reports, and IS operations costs, while development performance focused on project management factors such as on-time completion, user acceptance, and post implementation audits (Betts, 1992; DeLone & McLean, 1993). Users were increasingly adopting strategic and end-user-based systems, but the measures being used by practitioners did not capture this phenomenon (DeLone & McLean, 1993). Multiple surveys of CIOs during this period found that measuring information systems effectiveness or success was one of their major concerns: however, although this was an important issue, it was elusive to capture in practice (Brancheau et al., 1996; Brancheau & Wetherbe, 1987; Dickson, Leitheiser, Nechis, & Wetherbe, 1984).
Overall, this era was marked by the addition of newer measures of information systems success, which are summarized in Figure 3.
In addition to the organizational benefits provided by the system, this era had an increasing focus on benefits to the individual. The success measures in this era used by researchers reflected a sociotechnical perspective. The idea of use of the information system became linked to the idea of user acceptance of the information system. It became important to measure the user.s satisfaction with the information system in addition to other technical or operational components of the system. This era began to embrace a more holistic view of success, in which both organizational impacts and how the individual user was affected by the information system became important. Yet, in practice, the view of success was detached from the user and organization in that the measures of success seemed to focus on project management and the system itself, rather than the users and the full impact of the IS on the organization.
2.4. Enterprise System and Networking Era (1990s - 2000s)
Networks and client-server computing changed how individuals and organizations used information systems in this new era; data could now be shared among applications and managers rather than remain isolated on a single machine or with a single individual (Dahlbom, 1996). The emphasis shifted to improving collaboration among individuals, groups, and organizations in an effort to harness the power of networked information. Transaction processing systems, decision support systems, and management information systems that originally worked in isolation became more sophisticated as enterprise systems connected these disparate information systems across departments and organizations. The number of users of information systems increased further as these systems were used enterprise-wide and across organizations.
During this period, the proliferation of enterprise systems in organizations, such as enterprise resource planning (ERP) systems, promoted research to evaluate the success of these systems. Some researchers developed measures for evaluating IS success for ERP systems by using constructs from the D&M IS success model (DeLone & McLean, 1992, 2003), such as system quality, information quality, use, and impacts (Sedera, Gable, & Chan, 2004). Other researchers advocated a process in which the evaluation of the success of the ERP system (i.e., a balanced scorecard approach by Kaplan & Norton, 1992) is a feedback loop, so that specific actions can be taken to improve the ERP to meet the goals of the organization (Uwizeyemungu & Raymond, 2009). Shang and Seddon (2002) developed a framework to identify the different types of benefits that can be realized after the adoption of enterprise systems, which included operational, managerial, strategic, IT infrastructure, and organizational benefits. However, organizations continued to wonder if their large investments in enterprise information systems and technology were actually producing tangible benefits. Research identified that many of the benefits derived from enterprise systems may be intangible and more difficult to measure; these intangible benefits would measure success by considering the improvements in business processes, customer service, or organizational flexibility (Murphy & Simon, 2002). The proliferation of enterprise systems kept a strong focus on the overall impacts that information systems could offer to the firm.
Many firms in this era chose to outsource important IT and operational functions to improve efficiency (Hirschheim & Dibbern, 2009). For interorganizational information systems (IOS) that connect organizations with their suppliers and partners, both internal and external, metrics are important. For example, the adoption of an IOS can affect an organization internally, through changes in governance, as well as externally, by having impacts on strategy, operations, and social interactions across the organization (Robey, Im, & Wareham, 2009).
While the ability for information systems to improve individual decision making continued to be important during this era, collaboration became increasingly valuable to firms, as reflected in the research published during this period. Information systems research introduced a new stakeholder beyond the organization and the individual: the team. Research demonstrated that team performance could also be improved by systems, such as group decision support systems (Zigurs & Buckland, 1998). Other perspectives when evaluating success were also considered. Seddon, Staples, & Patnayakuni (1999) identified different stakeholder groups such as observers, individuals, groups, managers, and societies that were concerned with the evaluation of information systems, along with different types of measures that could be used to evaluate systems. Using a matrix, the authors demonstrated that there should be different measures of success, depending on the stakeholder involved and the context of the system.
In this era of distributed use of information systems, researchers suggested the need to consider system support staff as another dimension of success. Pitt, Watson, & Kavan (1995) discussed the need to consider the IS function as a whole as it develops, maintains, and supports information systems throughout the enterprise. In their research, the service quality of the IS function was proposed as an important component of measuring the success of information systems. This component of service quality had been captured in early measures of user satisfaction (Bailey & Pearson, 1983; Doll & Torkzadeh, 1988), but this study suggested that the IS function itself should be more prominent when evaluating the success of an information system, particularly since this support role becomes increasingly important as more individuals within the firm use enterprise-wise systems.
Research in this era recognized that benefits of an IS may exist at many levels, including at the international, national, industrial sector, firm, application, or stakeholder level (Smithson & Hirschheim, 1998). Realizing that an information system can have impacts beyond just the individual and the organization, DeLone and McLean (2003) reexamined the literature and updated their IS success model to expand these two impacts into the more comprehensive variable of net benefits. In this update, they acknowledged that benefits provided by an information system could occur at multiple levels of analysis, ranging from benefits to the individual up to benefits to society. This change, along with the introduction of IS service quality, productivity, and team performance as measures of IS success, were some of the key changes in our IS success measures during the Enterprise System and Networking Era.
In practice, as organizations continued to increase the sophistication of the information systems they used within and across firms, there was continued reflection on the value that IT can provide to organizations. Just like the "productivity paradox" debate in the 1990s in which there was a concern that investments in IT were not producing a tangible benefit to firms, concerns were raised about the ability of information systems to provide a strategic value to firms (Carr, 2003). The issue raised was whether systems had become commoditized and were no longer a source of competitive advantage in most firms, particularly in terms of achieving strategic goals.
To determine whether IS research was relevant and was impacting practice, Rosemann and Vessey (2008) examined the measures of IS success as proposed by the D&M success model to see if they resonated with IS practice. In their focus groups, practitioners revealed that they did not consider or use many of the measures of IS success identified by researchers, e.g., use, intention to use, or information quality. Furthermore, these practitioners discussed their emphasis on evaluating IS project quality as a measure of IS success, such as whether the project was on time, within budget, and met functionality. The findings of this study suggest that practitioners focused more on evaluating the process of developing systems (i.e., IS project quality) than on the benefits provided by the finished product (i.e., the resultant information system). These results from Rossmann and Vessey are consistent with other studies of practice during this era. Jones and Hughes (2001) found that if firms performed a formal evaluation of an IS, they tended to rely on cost/benefit analysis or an assessment of "does the technology work" rather than considering the social impacts or the value the system provided to the firm.
Figure 4 chronicles the progression of IS success measures through the enterprise system and networking era. The measures of success in the research literature attempted to be comprehensive in order to identify all of the benefits to the organization and stakeholders. Yet, in practice, most of the measures of IS success reflected a more project-oriented perspective in which success was based on the success of the project management effort rather than the success of the system itself within the organization.
2.5. Customer-Focused Era (2000s and Beyond)
The current customer-focused era reflects the continuing sophistication of information systems. In this era, individuals have the potential to receive customized experiences based on their interests, preferences, or roles. Amazon.com used personalization to recommend products to customers based on the user.s prior purchases and viewing habits on their website. On 9 December 2009, Google announced that it would begin to customize search results based on information gleaned from the user, such as location, browser used, and other user account settings. Thus, two different individuals, using the same keywords in Google, would receive different outcomes for their search results. Other entities followed suit, such as Facebook, Microsoft Bing, and Yahoo! News (Pariser, 2011).
As this push to make technology more customizable to the characteristics of individual users progresses, new measures of success arise. Increasingly, in the Customer-Focused Era, we find that customers interact with information systems directly rather than through employees of the organization. A user can order products, track shipments, and receive customer service without ever interacting with an individual within the firm. In the customer-focused era, social media, social networking, and peer-to-peer computing show that information systems are no longer for business and productivity alone, but also for hedonic uses such as entertainment. Governments are using information systems to cut costs, improve accuracy, improve service, and enable citizens to engage in policy making. In certain municipalities, citizens can obtain government services online and file tax returns via cell phones. As the user, whether consumer or citizen, gains increasing access to information and its capabilities, power is shifting from the organization to the consumer.
As information systems are used, not only by managers and employees of the firm but also by customers and suppliers, external measures of IS success become more important. For customer- facing systems, information systems success measures should consider if systems used by the customer, or to support the customer, are enjoyable, easy to use, and useful, and at the same time provide benefits to the firm (Wang, 2008). In this era, measurement becomes more complex; that is, systems must create value (success) for the customer and the firm concurrently. The commercialization of the Internet and the growth of e-commerce have forced managers to consider their organizations. website as a powerful tool to generate interest in their products and to create new customers. Therefore, measures of success for e-commerce and customer-facing systems have risen to prominence in the customer-focused era (DeLone & McLean, 2004; Liao, Palvia, & Lin, 2006).
Information systems and technology now connect us globally, impacting not only businesses, but also our personal lives (Friedman, 2007). The advent of Web 2.0 and the increasing power of smart phones have allowed people to be connected anytime and anywhere. We are becoming a collaborative population that works together to generate knowledge and share information for projects, such as the Human Genome Project, and for more personal interests, such as Facebook and YouTube (Tapscott & Williams, 2006). The changes in our information systems can stimulate economic development and create opportunities for social and economic change in both the developed and developing world.
Prior methods of evaluating IT success - system and information quality, use, user satisfaction, individual benefits, and organizational impacts - are all still relevant in the customer-focused era; but the context and metrics related to these factors have been changing over time. Given the increasing importance of the user.s interaction with the system, the context of the information system adds to the complexity of measuring its success. For example, the evaluation of information systems success used in e-government to serve citizens has been on the rise (Gil-Garcia, Chengalur-Smith, & Duchessi, 2007), and the measurement of these systems varies. Murray, DeLone, and Golden (2011) developed and tested a public value measure of net benefits for citizens of government websites. Others researchers have focused on more operational impacts of government projects, such as cost savings and operating efficiency (Hackney, Jones, & Lösch, 2007). Another example is healthcare information systems, in which measures of success are based on factors such as mortality and healthcare outcomes, in addition to profitability (Devaraj & Kohli, 2003) or productivity (Menon, Lee, & Eldenburg, 2000).
Information systems are no longer purely utilitarian in their function, but have shifted to become systems that have enjoyment for the user as their purpose. For instance, van der Heijden (2004) examined users. acceptance of hedonic information systems as a measure of IS success. This shift demonstrates two different expansions of the study of information systems success: the purpose of the information system studied (i.e., those for work versus those for pleasure) and the stakeholders affected (i.e., customers in addition to employees). This suggests the need for additional measures of IS success when the information system provides enjoyment as opposed to a system that supports a specific work task. Some hedonic information systems may also contribute to the organization by providing a medium for advertising (e.g., Google, Facebook) or providing subscription content (e.g., Wall Street Journal, Consumer Reports). However, in these hedonic systems, it is important to measure success by considering if the systems are important and appropriate for the users, not just usable (Palmer, 2002). This will provide feedback to the organization regarding whether these systems are contributing to the organization.s performance.
As information systems become more personal and customized, we must recognize that the perception an individual has about an information system may vary among user groups and even among individuals. One stakeholder group may believe a system is a success, while others may view it as a failure (e.g., Bartis & Mitev, 2008).
When measuring the success of an information system, how the information system supports the user (e.g., for data entry, to obtain information, for pleasure) should also be considered (Alter, 2008, p. 458). By considering the larger work system in which the information system is embedded, the organization is able to see whether the information system is the cause of positive - or negative - impacts to individuals or the organization, or if these net impacts are due to the environment or the work process being supported (Alter, 2008, p. 458).
A study by Seddon, Graeser, & Willcocks (2002) examined how managers actually evaluate the success of information systems in their firms. They found that many managers use techniques such as the balanced scorecard method (Kaplan & Norton, 1992) to evaluate the success of information systems, but other firms still struggle with evaluating their systems. The quintessential question for most managers is: "Am I receiving the value from IT that I need or want?". The challenge of developing measures for evaluating the success of information systems is still prevalent today. Many organizations struggle to evaluate the success of their information systems because there are so many factors can confound their assessment (Caldeira & Ward, 2003). While some research has produced instruments to evaluate success in some contexts, such as enterprise resource planning systems (Sedera et al., 2004), the measurement and conceptualization of IS success in a practical context remains complex (Gable, Chan, & Sedera, 2008).
Figure 5 shows the progression of past and present measures of IS success in both research and practice. Research has acknowledged the need for customized measures of IS success based on the context of the system, the work process supported, and the stakeholders considered. However, practice is still struggling with evaluating IS success. When success is evaluated, the approach is rigid with a focus on using formal frameworks like the Balanced Scorecard approach. While this is a step forward for practice, without adapting these frameworks to today.s complex systems, firms may overlook other important benefits of their information systems.
3. The State of IS Measurement in Practice
IS practice has lagged IS academic research in terms of the measurement of IS success. Table 1 shows the differences between research and practice across the five eras of information systems in terms of the measures of IS success, the levels of analysis, and the stakeholders considered. Everyday challenges and demands on the IS profession continue to relegate success measurement to a back seat (Gwillim, Dovey, & Wieder, 2005). This is unfortunate because credible success measurement is key to demonstrating the value of information systems. CIOs continue to be frustrated by their inability to justify the value of what they do for, or contribute to, their organizations. To date, the dominant focus of IS success measurement in practice is on inputs and processes, rather than on outcomes. It also tends to be project-based rather than asset-based.
The primary measures are cost-benefit assessments before the system is built, project quality during the building of the system, and post-implementation audits after the system is implemented to determine whether the system is working from a technical perspective (DeLone & McLean, 1993; Ward, Taylor, & Bond, 1996). More care seems to be placed on evaluating whether the project should be funded, rather than on evaluating whether the system provides intended benefits (DeLone & McLean, 1993; Ward et al., 1996).
Unfortunately, stakeholders perceive the evaluation of information systems success as something that is costly and difficult; so nothing seems to happen as a result of evaluation effort (Jones & Beatty, 2001). Evaluations of information system success after implementation tend to focus on the ability of the project to meet time, scope, and functionality requirements. A survey of 60 UK organizations found little evidence that organizations examine if the benefits expected from the project have been obtained once the project is completed (Ward et al., 1996). Smithson and Hirschheim (1998) have further argued that most IS evaluation undertaken by IS professionals concentrates on technical aspects such as, "Does it work?" rather than social aspects such as "Is it used successfully?",or business aspects such as "Does it deliver value?". The true value of an information system often comes months and even years after its implementation. The value comes when the system is being used effectively by management to make key decisions or by customers to make buying decisions or by suppliers to improve product and delivery. At best, only 20 percent of a system.s value is created during the system development process, while 80 percent of its value is realized during its use (Marchand, Kettinger, & Rollins, 2000); therefore, use and outcomes should be the real focus of IS success measurement within organizations.
The ability to measure IS success based on impacts or outcomes is tied to an organization.s ability to measure its organizational outcomes. Therefore, IS managers need to be proactive in the development of organizational performance measurement systems in order to provide a framework for measuring IS impacts. The goal should be that evaluating the success of an information system "exists across the complete IS lifecycle, and involves assessment against strategic goals and objectives, the use of an appropriate measurement technique, and the capacity to feed back results into the enterprise (hence signifying a level of organizational learning)" (Irani, Sharif, & Love, 2005). Unfortunately, this is not the practice in many organizations. Practitioners are often forced to rely on their "gut feel" or cost/benefits analysis rather than more comprehensive evaluation techniques (Bannister & Remenyi, 2000). The downside of the use of cost/benefit approaches to measure success is that many studies have found it difficult to prove a relationship between information technology investments and quantifiable organizational benefits (Caldeira & Ward, 2003).
A final observation is that organizations tend to neglect the role of the user or individual when measuring the success of an information system. Consistently, practice focuses on specific aspects of the system, project, or organization with little attention to how the system is used and whether the users are pleased with the outcome. While research has argued for the importance of evaluating success from the user.s perspective (Bostrom & Heinen, 1977; DeLone & McLean, 1992, 2003; Hamilton & Chervany, 1981a; Langefors, 1974; Mumford, 1974), practice has failed to heed this advice. Researchers have evolved their understanding of success in terms of measures, unit of analysis, and stakeholders examined (Grover, Jeong, & Segars, 1996); but there has been a time lag in these ideas appearing in practice. This lack of concern for user impacts has led to dramatic consequences for some firms, such as Telstra.s failed IS implementation that generated a union complaint about harsh working conditions (Tindal, 2008). The unwillingness or naivety of firms to consider the users. point of view about an information system means that organizations are lacking a critical input to determine whether their systems are, indeed, successful.
In the Customer-Focused Era, the key to understanding information success is that the most important class of user is the customer. Information is power, and ubiquitous and cheap access to information for customers and citizens has changed the power dynamic between organizations and their customers. Value creation via information systems must be measured from the customer.s perspective and not only from the organization.s perspective. Companies must measure system quality, information quality, service quality, use, satisfaction, and net benefits from the customer.s experience. Few existing IS measurement systems are built to measure these customer or citizen perspectives on success.
4. IS Success Research: Where to Go From Here
As a discipline, there is much for us to do in our future as we continue to study the evaluation of the success and effectiveness of information systems. In this section, we highlight several themes for researchers to consider about the future of IS success: recognizing the importance of "information" in the information system, developing adaptive research processes and models to measure IS success, disseminating practical approaches for measuring IS success, and avoiding silos in IS success research. We offer these themes as calls for action in the future of IS success research.
4.1. Recognizing the Importance of "Information" in the Information System
As technological and economic forces have created an "information age," information distribution and use have expanded to the far ends of the globe, serving a diverse population of users for a wide range of purposes. On the surface, this fact would appear to complicate exponentially the evaluation of information systems success. However, at its core, the fundamental purpose of information systems has not changed. Information systems communicate processed data into a relevant form for "users" who need the information to make decisions or take action. These users may be using the information to make a decision to improve an organization, to take action to positively impact society, or for entertainment.
The flow in today.s information breaks down traditional organizational boundaries to create temporary collaborations or collectives. IS researchers have an unprecedented opportunity to lead practice by developing success metrics that consider the information needs for stakeholders beyond the organization, such as customers, suppliers, and partners. For example, by considering this expanded view of information systems success beyond the boundaries of the organization, we recognize that the customer drives value. Therefore, we can consider the potential of information systems to provide information for value creation, rather than just systems efficiency.
This view of IS success should help the organization identify how well information systems support and positively influence the business strategy. As products become commoditized at an increasing pace, value can be created through information when a commoditized product incorporates information to offer a better customer experience. The key to information value is relevance, and relevance differs across users and their circumstances. While often understudied, the importance of information quality remains important as a key component of the information system (Petter, DeLone, & McLean, 2008). Recent history demonstrates that customer values change rapidly, so systems that track and predict customer value must change rapidly as well. In the future, information systems must be increasingly organic, real time, and predictive. To accomplish this goal, it is not just the technical quality of the system that will drive benefits to the organization or society, but the information that is produced by the system.
4.2. Developing Adaptive Research Processes and Models to Measure IS Success
Success measures in the early eras of information systems were evaluated quantitatively and relatively objectively. Current research recognizes that many measures of information systems success are more subjective and include intangible benefits. The wide range of stakeholders and purposes of the systems has further complicated the measurement of IS success.
In the most recent era of information systems, there is an increasing focus on the design and measurement of information systems based on customer value and social value. How do business systems improve the customer experience, build loyalty, and generate revenue? How do systems create social value for geographic and virtual communities? How do government information systems increase public value as determined by citizens individually and collectively? Information systems have the potential to solve complex social challenges. Terrorism, natural disasters, and economic trauma are ripe for more intelligent information systems that can create what IBM calls a "smarter world."
These challenges imply large-scale, mega decision support systems that require masses of location- based data with flexible real-time processing. Success will be measured by shared social value of the outcomes resulting from information use and intelligent decision making. As always, "good" outcomes should be the focus of IS measurement. The challenge is that the definition of "good" is fleet