Assessing the validity of IS success model: An empirical investigation ...

Proceedings of the Second Workshop on Knowledge Economy and Electronic Commerce

Assessing the validity of IS success model: An empirical investigation on ERP systems

Wen Lung Shiau Department of Business Administration, National Central University in Chung-Li, Taiwan, R.O.C [email protected]

Wen-Hsien Tsai Department of Business Administration, National Central University in Chung-Li, Taiwan, R.O.C [email protected]

Ping-Yu Hsu Department of Business Administration, National Central University in Chung-Li, Taiwan, R.O.C [email protected]

Ming-Sung Cheng Department of Business Administration, National Central University in Chung-Li, Taiwan, R.O.C [email protected],

Jun-Der Leu Department of Business Administration, National Central University in Chung-Li, Taiwan, R.O.C [email protected]

Yi-Wen Fan Department of Information Management, National Central University [email protected]

Abstract The purpose of the present study is to empirically assess the updated DeLone and McLean (2003) IS success model on enterprise resources planning (ERP) context. Structural modeling techniques were applied to data collected by questionnaire from 270 firms, which used ERP systems. The updated DeLone and McLean structure model contained five variables (information quality, system quality, service quality, use, user satisfaction, and net benefits). Our results support the updated DeLone and McLean (2003) IS success model and contributed by considering the relationship between system quality and net benefits. Keywords: ERP, IS success, quality, IS use, user satisfaction, net benefits

1. Introduction The impact of information system (IS) is widely recognized by academics and 430


practitioners that information system success can increase its affection potentially (Wixom and Watson 2001; DeLone and McLean 1992, 2003). The wide spread use of information technology attracts lots of IS researchers to adopt some theorems, develop measurements, and do empirical investigation. DeLone and McLean (1992) reviewed 180 conceptual and empirical studies and organized as six major dimension or categories – System Quality, Information Quality, Use, User Satisfaction, Individual Impact, an Organizational Impact. From 1993 to 2002, more than 285 articles in referred journals have referred to it. The measurement of IS success progressed continuously during last decade. In 2003, DeLone and McLean reviewed more than 100 articles and provided an updated DeLone and McLean (D&M) IS success model. The updated D&M is success model is consisted of information quality, system quality, service quality, use, user satisfaction, and net benefits. DeLone and McLean strongly recommend that the updated D&M IS success model should continue to be tested and challenged (DeLone and McLean 2003). Motivated by Delone and McLean’s call for validation of their updated D&M model, we did an empirical investigation of updated D&M model on Enterprise resource planning (ERP) systems. Enterprise resource planning software systems are large and complex integrated software packages that support standard business activities and provide lots of benefits to enterprises. That is why most Fortune 500 companies have enterprise resource planning software installed (Oliver and Romm 2002; Kumar and Hillegersberg 2000). Firms that spend millions of dollars on ERP are primarily concerned about how their investment on information technology will influence firms’ performance. ERP systems are programs that aim to provide integrated software to support day-to-day business operations a decision-making. In order to support key business operation and management, ERP System need to integrate multiple functions including human resource, manufacturing, sales support, financial and cost accounting, and almost any other data-oriented management process (Hitt et al 2002). In order to support correct decision – making, ERP Systems, provide real time data which comes from different function areas. The adoption of ERP System requires large investments and organizational resource. Thus, ERP systems compared with traditional and simple management information system characterize a completely different class of information technology application (Amoako-Gyampah and Salam 2004). The benefits of ERP can be classified into 4 categories: (1) Standardized and integrated information from different function area. (2) Centralized administration activity (3) Increase the ability to add new function and reduce maintain cost (4) Enable general business processes toward best of practice process (Gattiker and 431


Goodhue 2000). Because of those benefits, ERP systems have become increasing widespread over the last decade (Hitt et al 2002). There is little statistical evidence on validation whether ERP System is success or not. This study modified the updated DeLone and McLean IS success model with the context of ERP System. The results support the updated DeLone and McLean IS success model with another relationship between system quality and net benefits. The rest of this paper is literature review, research model and hypothesis, methodology, result, discussion, conclusion and limitation.

2. Literature review A large number of studies have been conducted to identify factors that contribute to information systems success during the last decade. The dimensions of information system success are broad and multifaceted providing significant research opportunities and challenges. There has been a growing interest on the part of IS researchers to study the multidimensional of information system success ： on information quality, system quality, service quality, system use, user satisfaction, individual impacts, and organizational impacts. Information quality is used to measure the quality of the information that the system produces. It focuses on understandability, relevance, and variety (DeLone and McLean 1992, 2003). System quality is used to measure the information processing system itself. It focuses on system integration, flexibility, reliability, and response time (DeLone and McLean 1992, 2003).System use is a measure on using the system. It focuses on the help to make decision, the frequency use of the report, and the frequency use of the system (Bailey and Pearson 1983). User satisfaction is a measure on users pleasant-unpleasant continuum (Seddon 1997). Net benefits are composted by individual impacts and organizational impacts. It focuses on a measure of the sum of profits and costs to the use of an information technology application. (Seddon 1997; DeLone and McLean 2003) Some studies apply multiple measures at the same time and some endeavors have been made to again insight in the mutual relations between measures and organizational performance. Wixom and Watson (2001) investigate the factors affecting IS success. They proposed 36 items for measuring data warehousing success. There were 111 valid respondents and 91% of them were actively involved in the data warehousing system. They find that both information quality and system quality are significantly related to net benefits. Gelderman (1998) investigated the relation between user satisfaction, usage of information system and performance. There were 170 valid respondents and 23.2% of them were either information manage or 432


controller. The results indicate that user satisfaction is significantly related to performance and system usage is not significantly related to performance. Bailey and Pearson (1983) proposed 39 factors for measuring computer satisfaction. They found five most important factors are accuracy, reliability, timeliness, and relevancy. Etezadi-Amoli and Farhoomand (1996) developed 31 questionnaire items to measure end user computing satisfaction and user performance (net benefits). Their respondents were 341 employers of 22 organizations and 60% of the respondents were in managerial or professional positions. They find strong relations, such as easy of use and user performance, functionality of system and user performance, and quality of output and user performance. Igbaria and Tan (1997) investigated the influence on system use and individual impacts (net benefit). They used 371 valid data from a large organization is Singapore and found a significant relation between system use and individual impacts. Teng and Calhoum (1996) examined the IT decision-making relationship with the emerging organizational computing environment. Their respondents were 139 individual decision maker and 44% of the respondents were top managers or supervisory managers. They find that managers recognize the value of information technology usage in decision-making, the recognition is highly associated with how intensively these information technologies are used, and the IT usage can facilitate and improve decision-making. Teo and Wong (1998) investigated the performance impact of computerization in the retail industry. They adopted 14 items and 4 of them were used to measure user satisfaction, such as over all hardware satisfaction, software satisfaction, consultant satisfaction, and satisfaction with software vendor support. A total of 1455 companies responded. The results indicate that information quality is positively related to managerial satisfaction and organizational impact. Kremers and Dissel (2000 ERP) investigate the ERP System migration. They interview 24 Bean customers worldwide by telephone. They find that the value of an ERP System lies not so much in the product itself, but in its effective and efficient usage. Hitt et al (2002) investigated the business impact between investment in ERP and productivity measures. There were 350 unique firms responded. The result indicates that ERP usage is significantly related to financial performance.

3. Research model In 1992, DeLone and McLean’s comprehensive review of different information system success measures concluded with a D&M IS success Model in Figure 1. Figure 1. D&M IS success Model

433


System Quality

Use Individual Impact

Information Quality

Organizational Impact

User Satisfaction

In this model, system quality and information quality directly affect use and user satisfaction. Both use and user satisfaction interrelate and directly affect individual impact, which in tern directly affect organizational impact. The conclusions of this paper were: Is success is a multidimensional and interdependent construct that needs to define and measure interrelationships among those dimension. Researchers should be contingent on their objectives and context when they select successful dimensions and measures. Results can be compared and findings validated. The IS success model should investigate and incorporate organizational impact in more field and clearly need further development and validation. Based on D&M IS success model, a lot of empirical studies examined the associations among the measure (Etezadi-Amoli and Farhoomand 1996; Goodhue and Thompson 1995; Guimaraes and Igbaria 1997; Igbaria and Tan 1997; Teo and Wong 1998). Other empirical studies implicitly examined the multiple success dimensions and interrelation of this model (Wixom and Watson 2001; Gelderman 1998; Igbaria et al. 1997; Teng and Calhoun 1996; Torkzadeh and Doll 1999; Weill and Vitale 1999; Yoon et al. 1998; Yuthas and Young 1998). These IS researchers suggested many improvements to the D&M IS success model. In 2003, DeLone and McLean reviewed more than 100 articles. After comprehensive analysis and discuss, they proposed an updated D&M IS success Model in Figure 2. Figure 2. An updated D&M IS success Model Information Quality

System Quality

Intention to use

Use

NET Benefits

Service Quality User Satisfaction 434


In updated D&M IS success model, DeLone and McLean add a third dimension “service quality” to quality characteristics and combine “individual” and “organizational impacts” into a single variable, “net benefits.” The conclusion of this paper were: The updated D&M IS Success Model is a useful model for developing e-commerce success measures and should continue to be rested and challenged. Each dimension of this dependent variable should be carefully defined, selected, and measured because of complex, multidimensional and interdependent nature of IS success. It is better to use validated measures rather than the development of new measures. Given the importance of IS support service quality should be considered as and important dimension of IS success. After good development and testing of “Net Benefits measures on the individual, organization, firm, industry, and national levels, researchers should investigate and incorporate ”Net Benefits” measures on their study. The selection of IS success dimension and measure should be contingent on the context and objectives of their study. The focus of this study is on use of ERP systems. In this paper, we modified the updated D&M IS success model by considering it in the context of enterprise resource planning (ERP) system. To measure the success of the IS department “Service quality” may be the most important variable. For measuring the success of a single system, as opposed to the IS department, “information quality” and “System quality ” may be the most important quality component. (DeLone and McLean 2003). Therefore, we do not consider service quality because ERP is a single large system. Our investigation is on the firms that already implemented and used a ERP System. Thus, it is not necessary to consider the intention to use component. Wixom and Watson (2001) identified significant relationships between the system quality and perceived net benefit in data warehousing systems. Both ERP and data warehousing are a single system; we should take the relation between system quality and net benefit into consideration. Our model is shown in Figure 3. Figure 3. Research model ERP Information Quality

ERP System Quality

Use ERP

ERP NET Benefits

ERP User Satisfaction

435


Information quality, use, and use satisfaction A good information quality represents, more accurate, valuable, useful, timely, an relevant information for decision-making. If the out-put from the system is inaccurate, valueless, usefulness, and irrelevant, it had detrimental effect the usage attitude of the users, (Teo and Wong 1998;Teng and Calhoun 1996), In other words, if the information quality is good, it is more likely that the users would like to use it. In addition, the purpose of computerization efforts is to provide better information for users. Information quality can be belated to the user satisfaction (Seddon and Kiew 1994). System quality, use, user satisfaction, and net benefits. System quality is concerned with user interface, easy of use, usefulness, performance, and quality of document (Seddon 1997). If a system is not easy to use, such as slower response time, incorrectness and incompleteness system output, and system crash, it has detrimental effect the usage attitude of the users. In other words, if the system quality is good, it is more likely that the users would like to use it. Users use systems in order to get information they need whether users are pleasant or not when they use a system are relying on system functions and status. For example, users spent much time on system and got nothing, they will feel more distress. A system with high system quality can lead to individual and organizational benefits (Seddon 1997). Wixom and Watson (2001) proposed a system with flexibility and integration can lead to perceived net benefits. In other words, if the system quality is good, it is more likely to have benefits to firms. Use, use satisfaction and Net benefits A system will be used intensively only if users feel that it meets their need and facilitates their own goals (Gelderman 1998; Srinivasan 1985). After users make use of a system, users will able to evaluate the attitude to an information system. If an information system do well for users, it is more likely to have better sensation of users. A rationale for system use as an IS success measure is the idea that it will contribute to performance when it is used (and will not contribute to performance when it is not used) (Gelderman 1998). User Satisfaction and net benefits Users are able to assess the value of information system when they use the system pleasant or unpleasant (Wixom and Watson 2001; Bailey and Pearson 1983). The value of IS for organization to conclude that the benefits (rewords) will out weigh the costs (efforts). Users are pleasant with system because system can help them to facilitate their jobs. They used and provided more accurate date to system. The system can give users a better understanding context, increase decision-making productivity, reduce the cost, and increase net benefit (Wixom and Watson 2001 ). 436


4. Methodologies 4.1 Measures The items used in the constructs of this study were adopted from relevant prior research. The adopted items were validated and wording changes were modified to fit the ERP context. The items were measured based on a seven-point Likert scale ranging from (1) “strongly disagree” to (7) “strongly agree” and listed in Table 1 with their source. Table 1. Question items used in the study Construct

IQ3 Information Quality

Measure

Itemª

IQ4 IQ5 SQ1

Source

The outputs contain information in Delone and McLean the sequence that I find to be useful. 1992,2003 The outputs are easy to understand.

Developed and validated by

The outputs provided by the system Srinivasan 1985, Etezadi-Amoli are relevant to the decisions I make. and Farhoomand 1996 The system contains accurate data. Delone and McLean 1992,2003

SQ2 System Quality SQ3 SQ5

USE1

The system contains all needed data for related business processes. The data in the system reflects current process statuses.

USE2

Pearson 1983, Etezadi-Amoli and Farhoomand 1996

The system is used to help making decisions.

procedures for assessing users on a pro rata basis for the system that the utilized is reasonable. The frequency of use of reports and

USE3

Srinivasan 1985 and Bailey and

The system response time is short.

The schedule of charges and the System USE


documents generated by the system is high.

USE4

Frequency of use of the system

USE5

Average connect time per access

437

Delone and McLean 1992,2003 Developed and validated by Srinivasan 1985, Bailey and Pearson 1983


The degree of congruence between US1

what the user wants or requires and what the information products and services provided is high.

US2

The software provides complete Delone features.

and

McLean

1992,2003

The description of the functions/ Developed and validated by

User Satisfaction US3

commands displayed on screen is Bailey

and

Pearson

1983,

Etezadi-Amoli and Farhoomand

clear.

The users have positive feelings of 1996, and Teo and Wong 1998 US4

assurance or certainty about the systems provided.

US5 NET benefit

Amount of support provided by vendor or other sources is sufficient.

NB1

To reduce cost of stocks

Delone McLean 2003

NB2

To reduce cost of purchase


NB3

Improve inventory turnover

Martinsons 1999

ªIQ represents information quality, SQ is system quality, USE is using ERP, US is user satisfaction, and NB is net benefits.

4.2 Data collection A mail survey was distributed to 2000 companies which is randomly selected from China Credit Information Service with 5000 leading companies established in Taiwan. A total of 657 responses were received and 376 have used ERP systems. There were 270 valid respondents among companies used ERP systems. The respondents represent a diverse sample with regard to industrial fields. There are 2.22 % in food products, 4.44 % in textiles, 4.07 % in plastics, 2.96 % in chemical products, 10.37 % in information technology products, 19.26 % in electric and electronic machines, 10 % in steels, 6.66 % in transportations, 2.59 % in finance, 5.92 % in trades, 2.96 % in cement and construction industries, 10.74 % in other manufactures, 17.8 % in others. About ERP implementation, 42.2 % of respondents developed ERP system by themselves or outsourcing and 57.8 % of respondents purchased ERP package software to employ ERP systems.

5. Results Reliability and validity analysis The reliability and validity of the measurement instrument was carried out using 438


reliability analysis (Tables 2), factor analysis (Table 3) and confirmatory factor analysis. The reliability was assessed by the Cronbach alpha (Cronbach 1951). The table 2 shows that the reliabilities of the items ranged from 0.83 for ERP use to 0.95 for net benefits. The reliabilities for each construct are above the conventional level of 0.7 (Nunnally 1978). Convergent validity, which is the items load high on their respective constructs, was accessed with factor analysis. As show in table 3, all items had high loading on their respective constructs and support measurement convergent validity. Discriminate validity was tested with confirmatory factor analysis. The test was applied to every combination of latent variables (Anderson and Gerging 1988). Confidence intervals between latent variables don’t contain the value of 1.00 The range of results is from –0.0336 to 0.1076 computed with Lisrel. These results support the discriminate validity of the multiple-item scales. Table2. Reliability analysis Summary of measurement scales (N=270) Constructª

Mean

Standard

Reliability

deviation

(alpha) 0.88

IQ1

5.81

0.92

IQ2

5.59

1.00

IQ3

5.42

0.94

SQ1

5.86

1.05

SQ2

5.91

0.89

SQ3

5.95

0.92

SQ4

5.77

1.11

USE1

5.31

0.95

USE2

4.89

1.11

USE3

5.7

0.97

USE4

5.71

0.93

USE5

5.63

1.09

US1

5.40

1.01

US2

5.15

1.13

US3

5.16

10.5

US4

5.27

1.07

US5

5.22

0.98

NB1

5.41

1.06

NB2

5.34

1.05

NB3

5.31

1.01 Total

0.87

0.83

0.94

0.95

0.95

439


ªIQ represents information quality, SQ is system quality, USE is using ERP, US is user satisfaction, and NB is net benefits.

Table 3. Factor analysis Results of factor analyses Factorª 1 IQ1

0.690

IQ2

0.705

IQ3

0.646

2

SQ1

0.735

SQ2

0.801

SQ3

0.664

SQ4

0.646

3

USE1

0.531

USE2

0.582

USE3

0.703

USE4

0.768

USE5

0.687

4

US1

0.728

US2

0.796

US3

0.792

US4

0.834

US5

0.796

5

NB1

0.860

NB2

0.849

NB3

0.825

Only loading of 0.5 or above are shown ªFactor 1 represents information quality, factor 2 is system quality, factor 3 is using ERP, factor 4 is user satisfaction, and factor 5 is net benefits.

Modeling analysis The LISREL 8.5 was conducted to test the nested model. The fit of the overall nested model was estimated by various indices provided by LISREL. The modeling analysis allows researchers to assess and modify theoretical modes among other theoretical specifications and competitive models (MacCallum 1995). Anderson and Gerbing (1988) recommended estimating a series of fire nested structural models to assess the theoretical model. These five models are defined as followings: The 440


saturated model (Ms) is defined as one in which all parameters linking the constructs to one another are estimated. The unconstrained model (Mu) is defined as one in which one or more parameters contained in Mt are estimated. The theoretical model (Mt) is defined as a researcher’s model based on substantive or theoretical model of interest. The constrained model (Mc) is defined as one in which one or less parameter contained in Mt is estimated. The null model (Mn) is defined as one in which all parameters relating the constructs to one another are fixed at zero. A model, M2, has one or less freely estimated parameters than M1’s, is said to be nested within model M1 and noted as M2