Evaluating the impact of information communication technologies on ...

25 downloads 4044 Views 191KB Size Report
Investment in information and communications technologies. (ICTs) across all ... Importantly, the models needed for the analysis of these com- ... care clinical systems (PoCCS) (specifically, order entry and ..... Level 2 Samuels Building.
MEDINFO 2004 M. Fieschi et al. (Eds) Amsterdam: IOS Press © 2004 IMIA. All rights reserved

Evaluating the impact of information communication technologies on complex organizational systems: a multi-disciplinary, multi-method framework Johanna I Westbrooka, Jeffrey Braithwaiteb, Rick Iedemab, Enrico W Coieraa a

Centre for Health Informatics, University of New South Wales, Sydney, Australia, bCentre for Clinical Governance Research, University of New South Wales, Sydney, Australia Johanna I Westbrook, Jeffrey Braithwaite, Rick Iedema, Enrico W Coiera

Abstract

serves, “The fact that many serious and competent scholars can conclude that there has been little net productivity gain attributable to this technology seems enough proof that something is wrong”[1] (p14).

The health informatics research community has been undertaking work at the organization-technology intersection for some time now. However there is no one place in the literature which presents a rigorously-defined evaluation framework for use by practitioners and researchers seeking to assess the impact of information and communication technologies on organizational processes and outcomes. There are two main challenges. One is to conceptualize the design features of such an evaluation framework. The second is to specify what data will be gathered and how. This paper aims to address each of these problems.

ICTs in Health Care Organizations

Introduction

Evaluation of the impact of ICTs has been hampered by the complexity of organizations and the consequent difficulty in deriving simple models that predict success or failure. Single method approaches may measure the behaviour of individual organizational components, but the complexity of organizational structures means that many of the behaviours we are interested in measuring emerge out of the interaction of multiple variables. Organizations are complex systems and the development of predictive models to support ICT deployment clearly requires a multimethod approach to first determine which variables are relevant and then to determine the ways in which the variables interrelate.

We have not done well at predicting the consequences of IT investments aimed at productivity. The way out is to improve our ability to know what works. …. We want not deep truths about physical nature but merely practical truths about what helps people and organizations do better work faster. [1] (p343)

So far, evaluation of technical factors has identified variables important to interface design and functionality. However, technical features are estimated to explain less than 20% of IT failures[2]. Technical evaluations have failed to determine why useful and useable systems are often unsuccessful[3].

Investment in information and communications technologies (ICTs) across all industries is rising substantially. In the US for example “..since 1960, something over $4 trillion … has been spent on computing and total current expenditures for the United States amount to around 10 percent of GNP”[1]. (p15) ICTs are generally promoted as resulting in more efficient and effective work processes, reducing error and improving safety and quality. ICTs are credited with freeing work from the limitations of time and space, potentially affording greater interdependence and faster exchanges among workers and organizational units within and between organizations. Further, it is argued that ICTs enhance transparency of work practice and allow greater accountability. The role of ICTs is viewed as of particular importance with the increasing complexity of work and workplaces where the probability of human error is increased, and the consequences associated with such error are heightened.

In contrast, studies of how humans interact with computers, such as Computer-Supported Cooperative Work (CSCW) studies and workplace studies, have investigated the ways in which individuals and teams interact with and around technologies such as flight simulators, bridges of large ships, and subway control rooms[4-5]. A drawback associated with these socio-technology interaction studies is that in the cockpit, and on a ship’s bridge, a small team conducts a sequence of tasks in accordance with rules and regulations that are in many cases both explicit and easily followable. Put differently, their primary workplace is clearly defined and relatively isolated. Studies focused on such simple organizational interactions and structures, however, are of questionable relevance to more complex and fragmented work settings such as academic medical centers.

Keywords: health informatics, evaluation model, information and communication technologies, socio-technical

The health care sector, with its many professional subgroups, complex work processes and power structures, represents a much more fluid and dynamic context with fewer formalised control mechanisms[6]. Our research focuses on the academic medical centre as a complex system[7] of loosely-coupled, intri-

Despite continued investment and high expectations that ICTs will produce significant organizational benefits there is little empirical evidence to substantiate claims. As Landauer[1] ob-

1323

JI. Westbrook et al.

Another body of literature, the work on the diffusion of technological innovation[21], has identified the processes by which innovation is communicated though certain channels over time among members of a social system. Here however little attention has been paid to the consequences for workers and organizations of adopting technological innovations.

cate human conduct, with a large number of interlocking organizational and technical components, as well as a varied range of processes and outputs. In that sense, a hospital with its multivariate services, professional expertises and service sites is perhaps better comparable to the complex networks that link organizations across sectors such as banking, the media, professional services, and telecommunications.

There seems, in summary, to be a considerable gap. The evaluation of the consequences of ICTs requires a multi-method approach conducted over a period of several years as consequences unfold. The aim should not merely be to measure global organizational indicators, but to classify the nature of the outcomes in ways that are more variegated, spelling out which are desirable versus undesirable, direct versus indirect, anticipated versus unanticipated, contextual versus decontextual, and so on. Further, consequences should be assessed in terms of the extent to which benefits have resulted, as well as the extent of, and change in, the distribution of benefits across the organization[21]. For example, benefits may accrue to some organizational areas while impacting negatively on others.

The Need for a Comprehensive Evaluation Framework Little is known of the role or impact of ICTs in these complex, dynamic domains where individuals must juggle multiple interleaving tasks and sift through uncertain and changing evidence. Importantly, the models needed for the analysis of these complex socio-cultural systems are very different from more traditional ones used to analyse control, procedural and implementation strategies. More comprehensive evaluation approaches are required which include techniques that are able to account for not merely the technical and routine aspects of work, but also the social, organizational, cultural and cognitive dimensions of work [8-12].

We propose an evaluation framework that seeks to address the complexity[22] and political fragmentation of human practice[23] and their roles in determining the use and outcomes of ICT implementation. We have devised a four year (2003-2006) research program to empirically validate this framework allowing further refinement and the development of a generic multimethod evaluation model (MEM) to assess the impact of information and communication technologies on organizational processes and outcomes.

Research examining organizational and socio-cultural factors has provided insight into variables which diminish or create barriers to the use of ICTs within health organizations. Recent work has paid attention to the interactive complexity of group work[5] demonstrating the need to move outside the traditional individual decision-maker assessment and take account of the context of practice, team integration, and related socio-cultural issues. Our recent work has shown for example, that a team’s perception of their working relationships is positively associated with effective use of ICTs[13]. Research undertaken on group dynamics[14], the link between community and practice[15] and the co-organization of professional culture[16] has identified some of the effects of initiatives which attempt to standardise work practices across professional groups. Given that ICTs impose standardised work practices, their reception is likely to be highly dependent on the extent to which professional sub-cultural differences and tensions can be addressed, negotiated and if not overcome, then at least put in abeyance in the form of a ‘contingent inter-polar balance’[17].

The framework has a multidisciplinary wholistic design as well as deploying methods of ICT evaluation[24]. The framework incorporates individual novel evaluation techniques rarely applied to ICT evaluation such as results mapping. The project’s sociocultural focus combines more standard qualitative research methodologies with video-ethnography[25].

Evaluation Framework The focus of the research program is the evaluation of point of care clinical systems (PoCCS) (specifically, order entry and electronic prescribing systems) in major academic medical institutions. The framework is based upon an extensive review of previous evaluation studies of PoCCS and analyses of the limitations and strengths of prior approaches[26]. The framework will be tested using a prospective study of the implementation of PoCCS at three academic medical centers in Australia. A repeated measures comparative design is being employed. Organizations chosen will act as their own controls, as variables will be compared prior to and after system implementation over three time points: Stage 1 - Pre system implementation; Stage 2 - Six months post-implementation; Stage 3 – 18 months-two years post-implementation. In addition, data from other academic medical centers in the state will be used to make comparisons between those hospitals with and without PoCCS at each stage. In summary, the framework incorporates a range of methods including: surveys, interviews, focus groups, ethnography, worktask analysis, work sampling, web-log analysis, critical incident technique, results mapping, and clinical outcome indicator data analysis. Each of these methods is briefly elaborated below.

In addition, the measurement of organizational performance outcomes related to the use of ICTs are rare and have tended to originate from management consultancy, operations research, and econometrics. Several studies emerging in these areas have demonstrated a negative relationship between IT investment and organizational productivity[1]. Studies of reasons for ICT failure are sparse and those that do exist are inconclusive[3]. Most of these are retrospective study designs and thus are limited in their ability to hypothesize about causal factors[18]. Overall it appears that approaches to comprehensive evaluation remain rare. Few studies incorporate a multi-disciplinary approach or use a multi-method design[19-20]. No studies, for example, have traced the links between improvements in organizational outcome indicators and a diversity of predictive variables (technical, organizational and socio-cultural factors) within organizations as ICT were implemented.

1324

JI. Westbrook et al.

a random reminder method, such as that developed by Shu[31] staff will carry a beeper that activates at random intervals over a specified time period. Participants will record the activity they are engaged in each time the beeper beeps using a handheld micro-recorder. Activities will be coded into categories of work practices and changes in proportion of time spent on specific work practices over time and between professional groups will be examined. The video-ethnography will involve filming relevant aspects of work practice as identified through non-participant observation. Video data will be analyzed with input from organizational professional members, as well as using micro-behavioral analytic methods as proposed by Birdwhistell[32]. Thus, the field notes and film data serve as both reflexive and analytical materials, enabling subjective and objective explorations of work flow and task analysis.

Organizational Profiling The organizational profiling process will capture extant organizational and systems-wide data including budgets, staffing profiles and skill-mix, service profile, organizational structure, existing process indicators and current information technologies. Interviews and Focus Groups Following the collection and analysis of data from the organizational profiling, interviews and focus groups will be conducted with staff. The interviews will be structured to determine views and attitudes toward ICT and the impact of these on various aspects of work. Central issues to be explored include work standardization and systems change. An adaptation of the Critical Incident Technique[27] will be applied to obtain detailed narrative accounts of the impact of PoCCS on work practices, decision-making and outcomes. The technique provides a systematic approach to analyzing substantial numbers of detailed reports of behaviors leading to successful or unsuccessful outcomes on a task or process.

Computer log analysis All transactions on the PoCCS will be stored as web-server logs and a sample will be analyzed using a modified version of transaction log analysis. Transaction log analysis uses log data to examine variables such as the time, content, and type of transactions made by the user at the terminal[33] Patterns of information exchanges will be described. This will include descriptive statistics regarding, for example, the type of information exchange, time and frequency of use and location of access. Our previous research has shown this to be an effective method[34]. The results will also provide outcome indicator data such as a measure of error rates. Changes in performance indicators, for example a reduction in the number of decision alerts generated when incorrect drug combinations or duplicate tests have been ordered between stages 2 and 3, will be assessed.

To provide a quantitative measure of the extent to which PoCCS are impacting upon organizational processes and outcomes the data gathered using the Critical Incident Technique will be analyzed using Results Mapping[28]. Results mapping builds on the critical incident technique by providing a standardized comparative methodology to translate narrative accounts into codes, levels and scores of outcome-focused program performance. Results mapping will generate data regarding the extent to which ICTs are producing improvements in organizational processes and outcomes and provides a quantitative method for monitoring these changes over time. Organizational Culture Survey A validated survey tool for measuring the organizational safety culture will be administered to a random selection of hospitals across the state of New South Wales at each stage. These data will allow comparison between those hospitals with and without PoCCS. Further surveys will be administered at the 3 study sites. Survey 1 will examine change capacity during the pre-implementation stage; Survey 2 will focus on change engagement in stage 2, and Survey 3 will determine change accomplishment in stage 3. Each of the surveys will contain open and closed items relating to: professional subcultural allegiances, organizational-managerial modality (bureaucratic versus organic), change capacity and inclination, IT readiness and IT skills of staff, and expectations of technology.

Discussion We have analyzed some of the problematics of ICT evaluation from a socio-technical standpoint and explained how the academic medical center is a particularly complex environment in which ICTs are increasingly being implemented. There is no extant evaluation framework available by which to assess the impact of these ICTs on clinical work and organizational outcomes. We propose a rigorously-defined evaluation framework for use by practitioners and researchers seeking to assess the impact of ICT on health care organizational processes and outcomes. The two main challenges we have tried to address in this paper are to, conceptualize the design features of an evaluation framework and to specify what data will be gathered and how.

Team Climate Survey Support for innovation within each work team selected will be measured as a predictive variable. The Team Climate Inventory (TCI)[29] is a validated 44-item questionnaire that measures team members’ perceptions of support for innovation in the team. It is hypothesized that the effective use of the new ICT will be associated with high team climate scores. Our previous work examining the effective uptake of IT indicates that team climate is a predictive factor[13].

We now have a testable framework. We have embarked on a four year program to use the framework, gather the necessary data and analyze the robustness of the framework within the setting of academic medical centers. In essence, we are testing the hypothesis that the design of our evaluation framework is reliable, valid and sustainable across time. We aim for the final model to be generic and generalisable for use beyond the health sector.

Ethnography Within each of the sites, detailed observational work will take place on two selected departments. The ethnography component consists of three elements: a non-participant observation, task analysis, and video-ethnography[9]. The task analysis[30] involves mapping the work-flows and dynamics that currently contextualise order-entry and medication prescribing, and the changes which these flows and dynamics undergo following the introduction of the information technology. Using

Conclusion Implementation of large ICT remains a high-risk activity for most organizations. In the main, the short and long-term conse-

1325

JI. Westbrook et al.

of a point-of-care online evidence system. JAMIA 10, 246253.

quences of introduction are unknown. This situation will continue without the development of a comprehensive and validated ICT evaluation model to provide the means of identifying factors which predict the consequences of ICT for organisational processes and outcomes. Identification of these predictive factors will enable organizations to manage more effectively the business risks associated with ICT, and to make informed decisions regarding ICT impact.

[14]Engeström Y. Expansive Visibilization of Work: An Activity-Theoretical Perspective. J Collab Comput 1999;8:63-93. [15]Star S. Working together: symbolic interactionism, action theory and information systems. In: Engeström Y, Middleton D, eds. Cognition and communication at work. Cambridge: Cambridge University Press, 1998:296-340.

The national and international cost of failing to address this issue can be measured in the millions of dollars consumed by failed ICT and the excessive over-expenditure of IT budgets which have become the rule rather than the exception. This multi-method, multi-disciplinary study seeks to provide comprehensive answers to these vexing questions.

[16]Degeling P, et al. The Organization of Hospital Care and its Effects. Sydney: University of New South Wales, 2000. [17]Degeling P, et al. Professional sub-cultures and hospital reform. Sydney: University of NSW, 1998:321. [18]Massaro T. Introducing physician order entry at a major academic medical center: I. Impact on Organizational culture and behaviour. Acad Med 1989; 64:20-25.

References

[19]Santahanam R, et al. An empirical investigation of ODSS impact on individuals and organizations. Dec Supp Sys 2000; 30:51-72.

[1] Landauer T. The trouble with computers: Usefulness, usability, and productivity. Massachusettes: MIT, 1995. [2] Aarts J, Peel V. Using a descriptive model of change when implying large scale clinical information systems to identify priorities for further research. Int J Med Inform 1999; 56:4350.

[20]Kaplan B. Evaluating informatics applications - some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform 2001; 64:39-55. [21]Rogers E. Diffusion Of Innovations. New York: Free Press, 1995.

[3] Lehoux P, et al. Assessment of a Computerized MR System: Disclosing Scripts of Use. Eval Plan 1999; 22:439-453. [4] Heath C, Luff P. Convergent Activities: Line Control and Passenger Information on the London Underground. In: Engeström Y, Middleton D, eds. Cognition and Communication at Work. Cambridge: CUP, 1998.

[22]Kast F, Rosenzweig J. Contingency views of organization and management. Chicago: SRA, 1973.

[5] Goodwin C, Goodwin M. Seeing as Situated Activity: Formulating Planes. In: Engeström Y, Middleton D, eds. Cognition and Communication at Work. Cambridge: CUP, 1998:96-129.

[24]Klein K, Kozlowski S. Multilevel theory, research and methods in organizations: foundations, extensions and new directions. San Francisco: Jossey-Bass, 2000.

[23]Pfeffer J. Power in organizations. Marshfield: Pitman, 1981.

[25]Iedema R, Forsyth R, Delaney G, Jacob S, Westbrook J, Braithwaite J, Barton M.Video ethnography as a methodology for studying the technologization of practice. Submitted to MedInfo 2004.

[6] Helmreich R, Merritt A. Culture at Work in Aviation and Medicine: National, Organizational and Professional Influences. Aldershot: Ashgate, 1998. [7] Kauffman S. The origins of order: Self-organization and selection in evolution. Oxford: OUP, 1993.

[26]Westbrook JI and Gosling AS (2002) The Impact of Point of Care Clinical Systems on Health Care: A Review of the Evidence and A Framework for Evaluation. Kensington: Centre for Health Informatics, University of NSW.

[8] Kaplan B, Shaw N. People, organizational and social issues: Evaluation as an exemplar. In: Haux R, Kulikowski C, eds. Yearbook of Medical Informatics 2002. Stuttgart: Schattauer, 2002:91-99.

[27]Lindberg D, et al. Use of MEDLINE by physicians for clinical problem solving. J Am Med Ass 1993; 269:3124-3129.

[9] Kaplan B, Duchon D. Combining qualitative and quantitative methods in information systems research: a case study. Management Information Systems Quarterly 1988;12:571586.

[28]Kibel B. Success stories as hard data: An introduction to Results Mapping. In: Gullotta T, ed. Prevention in Practice Library. New York: Kluwer Academic/Plenum Publishers, 1999.

[10]Kaplan B. Addressing organizational issues into the evaluation of medical systems. Journal of the American Medical Informatics Association 1997;4:94-101.

[29]Anderson N, West M. Measuring climate for work group innovation: development and validation of the team climate inventory. J Org Beh 1998; 19:235-258.

[11]Martin J. Organisational culture: mapping the terrain. California: Sage, 2002.

[30]Annett J. Theoretical and pragmatic influences on task analysis methods. In: Chipman S, Shalin V, eds. Cognitive task analysis. New Jersey: Lawrence Erbaum Associates, 2000:25-41.

[12]Hofstede G. Identifying organizational culture: Disentangling the concepts. Organ Stud 1998; 19:1. [13]Gosling AS, Westbrook JI, Braithwaite J. (2003) Clinical team functioning and IT innovation: A study of the diffusion

1326

JI. Westbrook et al.

[31]Shu K, et al. Comparison of time spent writing orders on paper with computerized physician order entry, Medinfo, London, 2001. IOS Press. [32]Birdwhistell R. Kinesics and context: essays on bodymotion communication. Harmondsworth: Penguin, 1970. [33]Wyly B. From access points to materials: A transaction log analysis of access point value for online catalogue users. Lib Res Tech Serv 1996; 40:211-236. [34]Westbrook J, Gosling S. Patterns of utilisation of the Clinical Information Access Program (CIAP) by clinicians in NSW: An analysis of web server logs, Ninth National Health Informatics Conference, Canberra, 2001. Health Informatics Society of Australia (HISA). Address for correspondence A/Professor Johanna I Westbrook Deputy Director, Clinical Centre for Health Informatics Level 2 Samuels Building University of NSW Kensington 2052 AUSTRALIA

1327