Facilitating Organisational Learning through Causal ... - CiteSeerX

4 downloads 626 Views 172KB Size Report
IS/IT Project Risk Management. Abdullah J. ... Abstract. Information System and Information Technology (IS/IT) .... Fifteen MSc students were asked to use causal.
Facilitating Organisational Learning through Causal Mapping Techniques in IS/IT Project Risk Management Abdullah J. Al-Shehab, Robert T. Hughes, Graham Winstanley School of Computing, Mathematical and Information Sciences University of Brighton Watts Building, Moulsecoomb, Brighton, BN2 4GJ, UK {A.A.Shehab, R.T.Hughes, G.Winstanley}@brighton.ac.uk Abstract. Information System and Information Technology (IS/IT) development and implementation have become more difficult with the rapid introduction of new technology and the increasing complexity of the marketplace. IS/IT projects often encounter a range of problems that can be described as failure. Thus, learning from an analysis of past projects and from the issues contributing to failure is becoming a major stage in the risk management process. In IS/IT projects, it is common for groups of stakeholders to participate in planning and management. One important element in these activities is risk assessment, that is, the identification of potential risks and their interrelationships throughout the project lifecycle. The ability to visualise cause and effect risk networks and the capability for interactive network building and modification have the potential for individual and group risk identification, justification and prediction. In this paper we introduce Causal Mapping as a method of accomplishing this, and describe two experiments: one carried out with a group of masters-level students and a second with practitioners from a government organization who had experienced an IS/IT project failure. These two exploratory experiments have demonstrated the potential (and also some of the problems) of the approach in identifying problem areas in past projects, through the collaborative construction of cause and effect maps that allow project participants to visualise their perceptions.

1. Introduction One survey of over 13,000 IT Projects [1] estimated that US corporations spent more than $255 billion per year on software development projects of which $55 billion was wasted on failed projects. The project success rate was just 34%, while the project failure rate was 15%, with 51% of projects suffering from cost overruns, time overruns, or a reduction in the features and functions delivered. For some time IT projects have been notorious for their proneness to fail (see, for example, [9]). In the United Kingdom, recently reported problems include delays to an online hospital booking application in September 2004 [10], to the national firearms database in October 2004 [12] and to the implementation of a secure national radio system for ambulance and fire services in November 2004 [11]. A very well known

supermarket chain reported a write-off of £260 millions associated with IT and supply chain systems [13]. The United Kingdom is not alone in suffering from these setbacks. One reason that has been put forward for the prevalence of these failures is that information systems/information technology (IS/IT) applications are not constrained by physical laws [3]. “Software is largely free of constraints and its potential is therefore unlimited”, according to [2]. Thus it is easy to embark on over-ambitious and illadvised projects. This is perhaps exemplified by the supermarket project mentioned earlier: in that report [13] an analyst is quoted as suggesting that the organization “was too ambitious on the business side, trying to over-segment its customer base which meant that it, in turn, created an overly complex IT system that wouldn’t scale.” This paper discusses issues of differing perceptions among project stakeholders during the post-evaluation process, and techniques of identifying and rationalising them into a combined and agreed model. Within this process, a method is introduced which is capable of presenting, in a visual diagrammatic fashion, the factors that have a bearing on project failure and their interrelationships. This allows the different stakeholders in a project to use the diagram to collaborate in the creation of risk models that can simulate the propagation and evolution of risks throughout the project life cycle.

2. Learning in the Risk Management Process An effective organizational learning cycle can increase the capability and maturity levels of the team, project and organisation. However, although risk management is focussed on identifying future problems, it is usually difficult for people to foresee future events and problems [14]. The study of past projects, however, can help to ‘sensitise’ project participants to the potential obstacles to a new project’s success. Pitagorsky [15] sums this up neatly: “The most important step to improve the quality of decision making is the Post-Implementation Review.” The processes by which an organisation can learn from past experiences are core elements of the concept of the ‘learning organization’ [16]. As Argyris [16] shows in his survey of the literature on learning organizations, some writers have identified obstacles that prevent organizations using past lessons as a basis for improving future performance. Leavitt and March [17] for example point out that organizations often adopt strategies that have worked in the past but which do not work in new situations. The lessons may be based on a small number of cases that might not in fact be typical. The links between cause and effect in past projects may not in fact be obvious or can be subject to controversy. What exactly happened in a past project may not be clear, and judgements about the relative success or failure of a project may depend on the viewpoint of individual stakeholders. These obstacles to effective organizational learning do not in themselves invalidate the argument that organizations need to learn from past experience. In fact it is argued that they underline the need to make explicit the nature of the lessons learnt from past experience. The research described below attempts to address some of these issues through the collaborative creation of causal mapping. This has been influenced by the constructivist paradigm that focuses on the way in which human actors use their experiences to construct mental models of the likely outcomes of future actions. Where actions are collaborative in nature, as in IS/IT projects, shared common cognitive models need to

be constructed as the basis for future effective action. Causal mapping is examined as one possible tool to assist in the construction of such models. One issue to be considered is the degree to which individuals have common or differing perceptions of the same situation.

3. Causal Mapping Our research is focussed on the use of the diagrammatic technique of Causal Mapping (CM) as a method of documenting past experience using IS/IT case studies. It involves assessing the ability of stakeholders to identify the relationships among problem areas in a case study by creating a visible map. According to [5] “A causal map is a word-and-arrow diagram in which ideas and actions are causally linked with one another through the use of arrows. The arrows indicate how one idea or action leads to another.” This approach is well established one early use [6] was to analyse how diplomats and government officials decided and applied policies, particularly in the field of foreign policy. In the field of project management and operational research, they have been used extensively [5],[6]. It should be noted that causal maps are often referred to as “cognitive maps”, which generally represent people’s perceptions of the relationships between cause and effect in a situation. Huff [4] suggests that an advantage of causal maps is that they can portray information about a system more succinctly than a corresponding textual description. An example of a causal map is shown in Figure. 1.

Fig. 1. Example of a causal map drawn in the student experiment

Each oval shape represents a problem area as a concept (variable) and the whole map represents a situation. The example above represents problem areas which have led to a delayed product.

4. Exploratory Experiment using Causal Mapping with Students There is a question about the degree of similarity in the perceptions that different observers have of the same situation. To explore this question, causal mapping was used to support the learning of students enrolled in the Master of Science (MSc) Information

Systems programme within the School of Computing, Mathematical and Information Science at the University of Brighton. Fifteen MSc students were asked to use causal mapping to examine a case study of project failure, identifying different problems which led to the failure of an IT system. They were also required to identify the interrelations between the different variables they had identified. One of the objectives of the experiment was to familiarise students with post-mortem or post-evaluation processes. The students were given a one-hour session to introduce Causal Mapping and explain how it could be used in such cases. They were divided into four groups to encourage discussion and were asked to draw a cognitive map of the case study. the one-hour tutorial group session explored the following: -

The case study being investigated How to identify problem areas in terms of cause and effects Methods of drawing CM and applying it to the data extracted from case study Analysis of resulting CM via group session.

The result was four different Causal Map diagrams representing four perceptions of the case study. All four of the original groups could be said to have similar backgrounds as they were all masters students on the same programme. They were all detached from the case study in the sense that they had not been involved in the original project and should not therefore have any personal bias. All groups were provided with the same textual description of the case study and so were working from the same evidence. Differences in how concepts were described by each group were observed. For example, while “contractors are – are not monitored” could be deemed to have the same declarative content as “contractors more – less monitored.”. Other examples could not easily be grouped into the same general concept class, but the debate about subtle, but important, differences could be illuminating. One of the authors drew a map of the same case study before handing the case study to the students. The concepts addressed in the five maps were compared to find similarities and dissimilarities. The total number of concepts in each map were 12, 11, 9, 11 and 14. In order to identify the equivalent concepts in the five maps, each one of the three authors (identified as raters A, B and C below) examined all of the concepts to identify the common ones that had been found by more than one group. The outcome is listed in Table 1. Table 1. illustrates the equivalent concepts common to more than one map Rater A B C

one group 25 18 26

Identified concepts common to more that one map two groups three groups four groups 3 2 5 5 3 5 7 3

five groups 1

All raters counted a large number of factors that were only identified by a single group. It is noticeable that in addition to the lack of agreement between the original groups, the raters themselves also often disagreed about which of the identified factors could be

regarded as the same. The differences between the three raters often arose from disagreements about what an identified concept really meant. For example ‘Need for Web presence’ concept was considered by one of the authors equivalent to ‘New Technology’ but was not by the other two authors. Even where groups agreed on the important concepts or factors, there could be disagreement over the relationships between the concepts. A particular problem was where two concepts were agreed to be related in terms of cause and effect, but some identified intermediate factors between the two while others did not – see Figure. 2 where the lower fragment has additional intermediate factors.

Figure. 2. Two examples of relationships from different maps

Note that in Figure 2 a slightly different notation is used to that in Figure 1. Two extremes, positive and negative, have been identified for each factor. A positive value at one node would normally lead to a positive value in any dependent nodes, while a negative value would lead to negative values in the dependent nodes. A minus sign by the arc indicates that this is reversed: that a positive value will lead to a negative value in the dependent nodes etc. Despite these differences, students found the mapping exercise useful. They were asked to complete an evaluation form on how useful the causal mapping method was and how easy it was to grasp. The majority of the students supported the use of causal mapping and also found that the method was easy to grasp and could be used on such cases in the IS/IT world. Furthermore, the use of CM as a focus for debate appeared to stimulate the learning process.

6. An Exploratory Experiment using IS/IT Practitioners It might be expected that different results might be found when dealing with real IS/IT practitioners and a real project. One of the authors has documented a longitudinal case study in Kuwait. This was accomplished during several field trips from 2003 to 2005. The problematic project started in 1998-1999 and raised many failure issues at the beginning of 2000, and suffered from various setbacks during the following two years. At one point the project was stopped for a period of time, and many stakeholders thought that it had failed and been abandoned. However it was reinitiated and went through much revision of the project design and management approaches, but problems and issues remain with the project up to the current time. For experimental purposes, the project team members were divided into two groups, managers (5 participants) and technical staff (8 participants). One of the objectives at this stage was to produce a combined diagram reflecting the perceptions of each group. It was considered that having both groups in one group session was undesirable because of the effect that managers might have on staff opinion in an open session. The possible causes of the problems with the project were explored with project participants in each group using causal mapping. Casual maps were drawn individually by project team members facilitated by one of the authors. Individual maps were then

combined to produce a single map for each of the two groups using the following protocol: -

Similar factors identified by different participants were combined No other factors were deleted and all factors were transcribed to a single map All the causal links between factors were transcribed to the single map The combined map was then reviewed and modified by all the individuals meeting together.

A large amount of data was collected, which is currently being analysed as part of a subsequent investigation into the application of quantitative model building. Currently a commercial tool, Decision Explorer by Banxia [20], is being assessed for use in this case study. The managers’ combined map consolidated 92 concepts from five individual maps, and was then discussed by the mangers in a group session. This led to a final version of the combined map with a total of 25 concepts. Figure 3 shows the combined managers’ map and illustrates how complex the combined map can be.

306 project delay 369 lack of standarization and quality control 547 work ethics of some of CC staff

370 new technology

546 depending on some of CC staff

545 CC lack of management skills and technical knowledge

411 lack of project control

405 unclear project plan

530 increase working efforts because of increased project size

309 bad communication between CC & Contractor 304 contractor poor performance + sub withdraw

365 poor documentation + incomplete feasibility study

501 cancelation of prototype stage + not following project plan

315 lack of top management support

506 budget allocated is less than project requirements 491 stakeholder orintation befor and after comencement of project

367 budget allocated not according to project requirements

498 proposal evaluation according to lowest not best

311 CC tech qualifiying program ... resources short + lack of tech qualifiying program

314 revesions to contract 301 unfrozen requirments

551 qualification should be done before the comencement of the project

308 long period of prject execution 303 lack of user/stakeholder particibation + support + trust

531 changes takes long time

Fig. 3. Managers combined map This process was repeated for the employees’ (analysts and programmers) group. The first version of the map produced a total number of 123 concepts and the final combined map had 24 concepts.

7. Similarity of Concepts in Management and Staff Maps The combining of individual maps clearly leads to complicated models, even after group review. Two issues arise from these complex models: -

Are the maps a good reflection of the situation being examined? It could be that the underlying pattern of causality really is complex. Are the maps effective guides to future action? Even if a model is a good representation of what has happened in a particular project, it may not be easy to apply to future projects.

These questions are the basis for ongoing research. An immediate task has been to measure the differences in perception between the two groups and also differences between members in each group. At group level, each of the authors acted as a rater and identified those factors that were shared by the two groups. There were some interesting differences – see Table 2 below. Table 2. Illustrating the equivalent concepts common to two maps.

Rater A B C

Identified concepts common to more that one map one group two groups 35 7 39 5 33 8

The small number of factors common to both groups is striking. There is also, once again, much variation between the raters’ evaluations. Taking the managers’ group alone, the three raters attempted to highlight factors that had been identified by all five group members on an individual level, four of the members, three members and so on. This was followed in the usual way by a group session for the raters in which apparent differences in interpretation were identified and debated. The combined results are shown in Table 3. Table 3 Concepts identified in group-session by more than one group Members

Rater A B C

Identified concepts common to more that one map one group two groups three groups four groups five groups 28 9 6 2 4 28 6 3 2 4 32 5 5 2 5

A major reason for the differences between raters was that it was only rarely that exactly the same wording was used by different participants. This led to problems of interpretation. In some cases it was simply that the wording used was ambiguous. In other cases, different raters might use different criteria for associating concepts – for example one rater might group together ‘little project management experience’ and

‘little development experience’ as both relate to inexperience, but other raters might take a different view. The exercise above did not look at cause and effect relationships, but was limited to the factors (that is, the nodes rather than the arcs). When a broader view was taken of both factors and the relationships between them, then many of the ambiguities were resolved. For example, in the case of ‘little project experience’ and ‘little development experience’, these clearly had different effects on the outcomes and should therefore have been distinguished. Another cause of contention was where similar but arguably distinct factors could be grouped under some collective title. An example of this was ‘changing requirements’ and ‘new requirements’. While both would have some outcomes in common, ‘changing requirements’ could have additional ones, such as the need to examine and modify existing code. Differences in outcomes was agreed to be a reason for not merging factors.

8. Discussion and Conclusion. Despite the differences in perceptions and the consequent complexity of the negotiated common maps, the participants still reported that the exercise was useful in terms of organisational learning. The series of experiments had the dual advantage of improving the participants’ knowledge base in risk management, and also served to illustrate how the method could be used to facilitate a collective continual learning process in project risk management. It may be that the value was in the process rather than the final product. As far as organisational learning is concerned, the participants reported the following advantages of causal mapping (CM): -

Group discussions guided by CM encourages participation. CM-facilitated group discussions tends to enhance communication between the project members. Using CM provides a clear picture of the project situation through its diagrammatic representation. The diagram enables identification of the interrelations between risks.

Ontological issues remain, as is the apparent problem of the disparity of perceptions between participants. However, the processes involved in producing causal maps describing risks within a project are useful as an elicitation method, and the differences that emerge are in themselves important in brainstorming (group) sessions. The ability to visualise cause and effect networks, plus facilities to dynamically modify them, has the potential, not only to create agreed and justified risk models, but also to facilitate documentation, model re-use and knowledge management and to reach a level of consensus. It should be recognised that the ontological issues that emerged were accentuated by having disparate groups of people, with different roles and with potentially different ways of expressing concepts. As Gruber [7] points out, “One problem is how to

accommodate the stylistic and organizational differences among representations while preserving declarative content.” We are involved in exploring the usefulness of CMs in identifying what has gone wrong in past projects. The concept of subsequently applying this to new projects is the motivation for our investigation, but this is not the focus of this paper. We have found that individuals tend to have different views of what has happened in a project, and this is the case whether all individuals are looking at the same textual description or considering a real project that they have been involved in. We have found in the experiments forming the basis of this paper, that managers and staff may have different views of what has gone wrong within a project. While there are apparent differences of opinion, the problem that we need to address for new projects is how to build a consensus that can form the basis for future co-ordinated action. Some of the problems are caused by ambiguity in terminology and some through differences in the granularity of defining concepts. This paper has reported some of our research which is concerned with seeking ways of building consensus, by making explicit through CM techniques the assumptions that people base their opinions on. Another line of research is the investigation of tools and methods that would allow participants to test out assumptions by allowing them to exercise the models they have created. This could be based on capturing the strength of beliefs and the likelihood of occurrence of concepts from the experiments’ participants, and translating these perceptions into values, which may then be used to simulate the map as a working model. One approach being considered is ‘Fuzzy Cognitive Maps’ [18], and what Montibeller, et al [20] refers to as ‘Reasoning Maps’.

Acknowledgment We would like to thank those of our colleagues who have contributed to this work, especially Tim Brady and Nick Marshal of CENTRIM at University of Brighton.

References 1. Standish Group International, Inc. “CHAOS Chronicles report”, Yarmouth, Massachusetts, USA, [online] < http://www.standishgroup.com/press/article.php?id=2>(2003) 2. The Royal Academy of Engineering : The Challenges of Complex IT Projects. (April 2004) ISBN 1-903496-15-2 3. Brooks, F.P: No silver bullet, essence and accidents of software engineering. IEEE Computer (1987) April 10-19 4. Huff, A.: Mapping Strategic Thought., Wiley and Sons, USA(1990) 5. Bryson, J., M., Ackermann, F., Eden, C., Finn, C., B.: Visible Thinking: Unlocking causal mapping for practical business results. John Wiley & Sons, Ltd. England (2004) 6. Axelrod, R. (ed.) Structure of decision: The cognitive maps of political elites. Princeton University Press, Princeton, NJ, USA (1976)

7. Gruber, T., R.: A Translational Approach to Portable Ontology Specifications. Knowledge Systems Laboratory technical Report KSL 92-71, Stanford University, CA. (1993) 8. Al-Shehab, A., Hughes, B., Winstanley, G.: Using Causal Mapping Methods to Identify and Analyse Risk in Information System Projects as a Post-Evaluation Process: ECITE 2004: The 11th European Conference on Information Technology Evaluation. Amsterdam (2004) 9. Flowers, S. Software Failure: Management Failure, John Wiley & Sons (1996) 10. Arnott, S. “Delays hit NHS bookings trial” Computing 2 September (2004a) p1 11. Arnott, S. “Emergency services face network delays” Computing 18 November (2004b) p1 12. Nash, E :“Firearms database delayed once again” Computing 28th October(2004) p1 13. Knights, M :“Sainsbury’s dumps inefficient systems” Computing 21st October(2004) p1 14. Wiegers, K. :Know Your Enemy: Software Risk Management. Software Development magazine, October 1998.[16] Pitagorsky, G. (2000) PM Network, Project Management Institute, March, (1998) pp 35-39 15. Pitagorsky, G :PM Network, Project Management Institute, March, (2000) pp 35-39 16. Argyris, C. :On organizational learning. (2nd Edition) Blackwell, London(1999) 17. Leavitt, B., and March, J.G. :“Organizational learning”, Annual Review of Sociology 14 (1988) 18. Kosko, B. :Fuzzy cognitive maps. International Journal of Man-Machine Studies. (1986) 24, 65-75 19. Decision Explorer is manufactured by Banxia Software,www.banxia.com 20. Montibeller G., Ackermann, F., Belton, V., Ensslin, L. “Reasoning maps for decision aiding: An integrated approach for problem structuring and multi-criteria evaluation”, the 46th Conference of the British Operational Research Society, 7-9 Sep, York, UK (2004)