Disaster Management Control: Decision Support ...

Preprints of the 18th IFAC World Congress Milano (Italy) August 28 - September 2, 2011

Disaster Management Control: Decision Support through Performance Assessment François Galasso*, **. Carine Rongier* Matthieu Lauras*. Didier Gourc* *Université de Toulouse, IUT de Rodez, Rodez, 12000 FRANCE (Tel: +33 (0)5-65-77-15-67; e-mail: [email protected]) **Université de Toulouse, Mines Albi, Centre Génie Industriel, Albi, 81000 FRANCE (Tel: +33 (0)5-63-49-30-00; e-mail: [email protected]). Abstract: Performance evaluation of complex processes requires the management of several dimensions focused on specific parts of the processes. In the frame of disaster management, non-governmental organizations have to consider the solving phase of the disaster lifecycle as a whole set of processes leading to a nearly industrial structure. Hitherto, performance assessment, providing them the base for ensuring the global performance of their actions, was done afterwards in order to improve their knowledge and set up best practices. According to interviews done with the French Red Cross, this approach is not sufficient anymore. Thus, this paper investigates a lack about the practices done in the context of disaster management which concerns the elaboration of performance measurement system for providing a decision support tool. After a state of the art, several aspects are covered: firstly, a process analysis is given, then the performance system is described and finally, a specific focus on indicators is done and applied to a case study. Keywords: Performance evaluation, Process models, Decision support systems, Disaster management, Performance indicators.

1. INTRODUCTION Disaster management lifecycle introduces four mains steps (Coppola, 2007, Ahmad, 2004, Beamon, 2004): prevention, preparation, response, and recovery. Prevention and preparation phases are before the occurrence of the event leading to the disaster. The recovery phase consists in middle or long term actions in order to restore all facilities in the damaged zone. Finally, the response phase is the one which requires short term actions in order to provide assistance to people in the disaster area. Obviously, the response phase can be considered as the most critical phase. Whatever the kind of crises, the need for fast actions introduces a lot of autonomy for the teams on the field. Their main goal is to complete each action they have to do. However, the completion of such action is hardly measurable due to a lack of time and efficient methods and tools. This statement leads to a practical problem consisting in an improvement of the practitioners’ methods for performance assessment. The literature has already pointed out the competition opened between non-governmental organisations in order to demonstrate their ability to make an improved use of their funds (Balcik, 2008). Thus, considering the complexity in terms of actors and internal processes of such an organisation, the performance assessment and the underlying global processes analyses introduce a mean to improve the control in the frame of our particular case that is the response phase (Hollnagel et al., 2006).

Copyright by the International Federation of Automatic Control (IFAC)

Taking into account these two statements, this paper introduces a new approach for decision based on performance assessment. This article is based on a method developed in order to position performance indicators on the key processes of the response phase (Rongier et al., 2010). However, the method given by Rongier et al., (2010) does not investigate aggregation mechanisms between several decisional levels (i.e. strategical, tactical and operational) and does not study the interactions between indicators. A lot of methods (SCOR, Balanced Scorecard,…), industry or project management oriented, are based on these statements in order to answer correctly at each decision level. They provide the necessary constructs to establish Performance Management Systems (PMS) focused on Key Performance Indicators (KPI). In this article, crisis management is firstly discussed on a practitioner point of view. Then, considering all processes involved in the response phase of a crisis, the decision support through performance evaluation is exposed as our scientific contribution. Finally, the application of the previous concepts through a web-based decision support tool is presented. 2. DISASTER MANAGEMENT: A PRACTITIONER POINT OF VIEW Obviously, disaster or crisis management, although some authors distinguish them respectively according to their human or natural origin (Kovoor-Misra, 1995), equally relies on the realisation of fast actions due to the emergency. So, in

10758


the whole article, we employ both disaster and crisis indifferently in a general meaning, as long as the response phase will require the same performance characteristics in each case. This section aims at describing the scientific statement in a practitioner’s point of view. Firstly, a global approach is used in order to consider crisis management. Then a more focused view on field activities lead us, in a third subsection, to a problem statement and modelling. 2.1 Global Context Competition between non-governmental humanitarian organisations (referred as NGO) (Balcik, 2008) is mainly traduced by an optimised used of the funds given by donors. Considering that these organisations are complex due to a lot of stakeholders (i.e. international committees, national agencies, local resources…), the decision process spreads over several decisional levels (see Figure 1).

2.2 Practical Conditions Both the problematic of positioning indicators and performance assessment is topical in the humanitarian sector and tends to develop. Indeed, according to the literature, this sector needs tools for performance assessment. Davidson (2006) argues that humanitarian organizations are practically unable to measure the performance of their supply chain, for example. The application of tactical decisions in the field requires a lot of autonomy as the situation is very unstable in the early beginning of a crisis. As an example, teams often have to deal with scarce resources for communication. Considering the operational level, the main activities are “physicals” and rely on the ability of teams to provide assistance in the damaged zone. There are no computational activities dedicated to performance evaluation. Communication is often informal. Reports are sent from the field to the headquarters once a week. 2.3 Problem Statement

Strategic level

On one hand, practitioners at tactical and strategic levels need to ensure the good use of the funds given by donors. On the other hand, the performance of the teams on the field is based on the right enhancement of several processes in order to give them the necessary means to face the bad conditions of a disaster.

Management Operations Support

Management

The answer developed by the authors in this article is to develop a Performance Management System (PMS) taking into account the particular conditions of a crisis. It shall tend to provide an easy mean to make sure that (i) the actions on the field are correctly done and (ii) decision makers have sufficient information to have a better view on the crisis evolution and reallocate funds if necessary.

Operations Support

Tactical level

Thus, the PMS shall have the following characteristics:

Fig. 1. Main common processes of a NGO considering strategic and tactical levels.

-

Figure 1 is obviously a simplified view as the tactical level introduces a breakdown for each macro-process. However, it is usual to see such kind of hierarchical decomposition in a lot of different contexts. The advantages and disadvantages of such a structure are well known currently. Thus, the global schema relies on a lot of interrelated processes dedicated to: -

-

Operations: are the processes which describe the preparation and the organisation of the action on the field, Management: groups the investigation, coordination, guidance and performance assessment of the activities Support: inventory, financial, human and, information resources and supplies.

-

-

-

10759

Web-based: it is an essential characteristic because the PMS must be constantly accessible to all the actors, who are located at various places of the world, Easy: the turnover is very important in the humanitarian sector thus, an easy to use tool is needed as the duration of the formation is very restricted in time of crisis, Several measures: tools developed can enable the storage of all the measurements of a crisis, Process oriented: the starting point of the tool is the modelling of the crisis response process at the strategic level. Users with different roles: stakeholders have several roles in crisis response as well as in the use of the PMS. Three roles have been identified: (i) administrator who manages the maintenance of the tool and the creation of a new crisis (i.e. create new cartography, processes and IP associated when occurs a crisis); (ii) decisionmakers, who are at the headquarters, have a limited access to the consultation of the measurements and a complete access to the performance assessment zone;


-

(iii) the team leaders who are on the field will have the free access to the measurements, Centralized: all the data (process, measurement, evaluation, document exchanged…) are stored at the same place, with the same tools: which saves time and avoid data loss. This characteristic is very important for the humanitarian sector. 3. PERFORMANCE EVALUATION FOR DECISION SUPPORT TO CRISIS MANAGEMENT

3.1 Literature Review Like projects, crises are specific. As long as the global organisation may remain the same, the actions are specifics (several kind of crisis found in the literature). However, the common processes show a multidimensional perspective. For example, Lauras et al., (2010) have detailed the performance cube (Fig. 2) describing how can a project be evaluated through a Triptych (meaning 3 categories of homogenous indicators), the project tasks and, categories of indicators.

Fig. 2. The Performance Cube (Lauras et. al., 2010) This Performance Cube is intended for project management. A similarity could easily be found between the activities dedicated to a project and activities in regular processes like industrial processes. A recent state of the art has been done by Gunasekaran and Kobu, (2007). From this state of the art several metrics are defined for the Supply Chain Management. These KPIs are based on nowadays well known methods (balanced scorecard, ABC, SCOR…) and can be partially applied to crisis management. Beyond the meaning associated to each indicator that permits to clearly measure an important part of the considered processes, attention is paid to the visual presentation of these indicators. In order to build efficient dashboards for the decision makers, two main concepts should be addressed: the first one is a position of the KPIs regarding their associated objectives; the second one is a tendency about the evolution of each process involved in the system. The latter balances the former showing that even if a process is not up to the

required standard, some improvement actions may have already started. 3.2 Focus on Crises Response Phase The crisis response requires specific KPIs. Although some KPIs could be similar to industrial (i.e. total length of the supply chain; purchasing or delivery costs...) or project management concerns (see Kim, 2009 for ERP implementation for instance). Thus, the problem of specific integration in order to provide information to upper decisional levels comes up. 3.3 KPI Definitions and Aggregation Considering the previous statements, the following model for describing processes and associated indicators is proposed: {p} is the set of processes. {k} is the set of Key Performance Indicators. {c} is the set of classes for the KPIs {c} = {Efficiency; Relevance; Effectiveness; Flexibility; Responsiveness}. τ is the period between two measurements. {d} is the set of dates on which the measurements are made, so, the elements of {d} are assumed to be a multiple of τ. Ok,d is the set of objectives associated to each KPI. It is obviously assumed that a KPI has only one objective. These objectives can be adjusted at each date. Ek,d is the set of measurement (evaluation) of each indicator at each d date, Ek,d is obtained by direct measurement done by the teams in the field. Mi,j ∀i,∀j ∈ {p} is the cartography of processes meaning that a process i is a subprocessus of j when Mi,j = 1. Otherwise, Mi,j = 0. Ai,j ∀i,∀j ∈ {p} is the adjacency matrix showing with process is adjacent to another. So, i is a predecessor of j when Pi,j = 1 otherwise, Pi,j = 0. Cc,k ∀c ∈ {c}; ∀k ∈ {p} is the association between one KPI and a class of KPI. It is assumed that a KPI belongs to only one class of indicators. Pk,d is the matrix indicating the position of each indicator regarding to its objective in Ok,d and its associated measurement in Ek,d. Pk,d is expressed in percentages. Tk,d is the matrix indicating the tendency of each indicator regarding to its deviation from the objective in Ok,d and its previous value at Tk,d-τ. Gc,d is the matrix indicating the position of each class in {c} of indicator. Gc,d is expressed in percentages. Sc,d is the matrix indicating the tendency of each class in {c} of indicator. wk is the value of the weight which is associated to each KPI. These weights are defined according to crisis practitioners. {p}, {k}, {c}, τ, M, A and, C are the main characteristics of the crisis response process. They are based on existing data and, according to practitioners, are established in the preparation phase in order to be deployed as soon as a crisis occurs. Thus, they are considered as constant in our model.

10760


O and E evolve with the crisis in order to take into account its evolution. As long as a crisis can get worse or better, the objectives associated to each indicator should be revised. Equations (1) and (2) are dedicated respectively to the calculation of the position and the tendency of each indicator or class of indicators.

Pk , d =

Tk , d =

Ek , d − Ok , d Ok , d

× 100 ∀k , ∀d

Pk , d − Pk , d −τ Pk , d −τ

× 100 ∀k , ∀d

(1)

(2)

Equations (3) and (4) calculate positions and tendencies for indicators classes.

∑C × P ∑C i,k

Gi , j =

k, j

k

∀i ∈ {c}, ∀j ∈ {d }, ∀k

i ,k

k, j

k

-

-

k

∑C ×T ∑C

gathering data of the crisis to enable Decision-Makers to identify where problems are, i.e. to help them to control the response process and to make easier the experience feedback; - assessment of the performance according to activities and processes. An aggregation of the measurements of KPI (first level aggregation) is thus necessary. - graphical presentation of the performance indicators at the crisis level in order to, for example, write reports. The tool takes into account these needs and embodies equations (1) to (4). Thus, thanks to the web-based system, several kinds of performance can be assessed:

(3)

i,k

Si , j =

-

∀i ∈ {c}, ∀j ∈ {d }, ∀k

(4)

i ,k

k

In order to construct the global system of indicators, a set G”p,c,d,k could be elaborated by a combination of the sets X, P and C. This leads to equation (5) which can be used to extract aggregations over, for example, each process or each class of indicators.

G ′p′, c , d , k = X p , k × wk Pk , d × Cc , k ∀p , ∀c, ∀d , ∀k

-

(5)

For each indicator, a value of Pk,d between more or less 5% of the corresponding objective Ok,d will be considered as good and thus, a green traffic light will warn the decision maker that this indicator has a correct value. A value upper or lower between 5% and 30% is considered as medium, an orange traffic light is shown. Finally, a value of Pk,d upper or lower than 30% means that the result is bad, a red traffic light is shown. Same kinds of warnings are used regarding the classes of indicator (3), (4) in order to determine the needs for improvement at first sight. Moreover, the warnings are just visual aids that can be adjusted. For example, in section 4, arrows are used to indicate the tendencies.

state: represents the performance of the response (of a crisis or a process) at one moment (1). This element allows the decision makers to evaluate the state of the process or the crisis. tendency: which permits to see changes over a period (in our case study, a period lasts 1 week). It is a very important element in order to detect a deviation in the response process and react before it causes problems more important or to see the effectiveness of the actions installation in real time (2). state by classes of indicators: represents an aggregation of the KPIs regarding their classes at each date (3). This element completes the state of a KPI informing stakeholders with an abstracted view. For example, a problem of responsiveness could be easily detected.

4.2 Studied Processes and Indicators Crisis processes modelling has been led thanks to the literature on relief management. According to the standard ISO 9000 – 2000, the highest decisional level: the strategic level (Fig. 3), is divided in three main domains: Supporting, Operational and Management (see section 2.1). Each domain is composed of processes which represent the tactical decisional level and finally each process is made up of activities, not detailed at in this paper, which are the operational level. This cartography is generic i.e. when a crisis occurs it could have few changes on the cartography (addition or precision regarding processes).

Our proposition relies on the hypothesis of commensurability of the indicators. So, the KPIs must be defined without any dimension. Practitioners have to choose the most meaningful indicators. 4. APPLICATION FOR A NON-GOVERNMENTAL ORGANISATION This section is dedicated to the illustration of the previous concepts through a case study done in partnership with the French Red Cross (FRC). 4.1 Description of the Case Study The first step of this case study consists in establishing the NGO’s requirement. This has been done through several interviews. The interviews led to the following requirements for the PMS system (in addition to section 2.2):

Fig. 3. Cartography of response process

10761


Methodology described into (Rongier et al, 2010) has been used to establish the most critical processes for the NGO: that are the operational processes (Fig. 3). Then, according to this methodology, KPIs are defined for each critical process. The KPI selected for the process “To supply” are shown in Table 1. Table 1. Summary of KPIs Equation

Aggregation

Data

Focus

Display of results

Instantaneous KPI Traffic performance value at 1 Process light st (state) (1 kind) date KPI Instantaneous Traffic value at 1 Crisis (1) performance light nd date (state) (2 kind) KPI Tendency (2) value on Process Arrow st (1 kind) 2 periods KPI Tendency value on Crisis (2) Arrow nd (2 kind) 2 periods Aggregation per KPI (3) class of value on Crisis Diagram indicator 1 period For each KPI, one objective value can be added by practitioners. Moreover, according to them, the initial values of 5% and 30% considering the thresholds described in section 3.3 can be used whatever the KPI. This assumption is deliberately strong but practitioners can correct it if necessary. (1)

4.3 Web-based Decision Support Tool This section aims at describing the main technical functionalities of the web-based tool. Its main aim is to support decision making during a crisis. So, the developed tool must permit: -

to easily visualise and navigate through processes, activities or indicators that are necessary to be studied, - to add measurements of indicators and to save them, - to register comments and to exchange reporting files in order to have the historic of the various actions which are carried out, - to analyse measurements in order to obtain a performance assessment, - to easily generate reports containing the various information on the answer to the crisis. When a crisis occurs, a specific cartography is created using the generic cartography (Fig. 1). Then the tool is implemented. The stakeholders could access the following web-pages: (i) Process; (ii) Measurements, (iii) Assessment, (iv) Collaboration, (v) scorecard and (vi) Cartography. i. ii.

Process page: this page gives access to the detail of the different tasks of a process. Measurements page: this page is used to enter measurements for each indicator for the selected

process and to join comments. The FRC has chosen a period of one week for the measurements i.e. stakeholders must enter their measurements in the tool once a week. It is possible to modify the saved values. The tool can provide assistance detecting missing values displaying a white traffic light. In a first approach, agreed by practitioners, the last known value of the indicator is used to complete the aggregation even if it is not completely satisfactory. The global result consists in a value that is not trustable as indicated by the white traffic light. Actions can be taken to find and insert the values afterwards. This last point is obviously important when an aggregation is done over a class of indicators and is. iii. Assessment page: The tool saves the data captured in the measurements page, makes an analysis and posts it in the assessment page, under the form of graphs, arrows and traffic lights (their use is detailed in the paragraph “traffic lights”) iv. Collaboration page: This page allows the exchange of documents and facilitates the collaboration between stakeholders. v. Scorecard page: Allows to display the results of all processes and thus of the total crisis response. vi. Cartography page: shows the processes of the crisis according to Fig. 3. As described in Table 1, each indicator or aggregation of the indicator values is associated to a specific visual aid embedded in the web-based system. The visualization of the state which represents the current performance of an element (indicator, process, and crisis) is done with traffic for the following reasons: -

these elements are easily understandable, the reading is simple which limits the risks of mistakes in interpretation, - the pictures of traffic lights do not take a lot of space when displayed. That permits to post several results on the same web page and to compare these results easily The three colors of the traffic lights are used in their common sense: -

green: the results are good orange: it is necessary to be vigilant, difficulties have to be considered - red: difficulties are encountered The visualisation of tendencies allows seeing the evolution of a process thanks to arrows. There are three kinds of arrows - downward: there is a deterioration of the results - medium: the results are stable - upward: there is an improvement These traffic lights and arrows appear on the pages cartography, for the global vision, and assessment, for the vision of one process. Histograms permit to visualize state by class of performance on the scorecard page.

10762


Table 2. Example of report obtained with the developed tool for a process

Examples of results are presented in table 2. These results show the values of the indicators and their objective. It is possible to download diagrams or tables from the tool to make reports. 5. CONCLUSIONS This article develops an approach for decision support system based on performance assessment. This disaster management system aims to guide stakeholders during the response phase and to improve their communication and collaboration. It relies on performance assessment through performance indicators assigned to specific classes. These classes (i.e. effectiveness, responsiveness and efficiency) highlight some dimensions of the performance. Moreover, our performance assessment model indicates both the position regarding its objective and the tendency of each indicator. These concepts have been applied to a case study driven for the French Red Cross. It allows us to complement the use of indicators (through their position and tendency) by the use of visual aids. This corresponds to practitioners’ needs. They require an easy web-based interface enabling to handle quickly the evolution of a crisis. However, it affords another important outcome which is the availability to store pertinent data. So, it can also be useful in a posterior evaluation used in order to ensure a good experience feedback. This paper deals with a problematic which tends to be developed. Actually, several NGOs are interested in such an approach. Thus, the main perspective is to test the PMS on a real case and validate the web-based tool with FRC and other NGOs. REFERENCES Ahmad, A. (2004). Disaster Management Through The New Millennium. Anmol Publications PVT. LTD, New Delhi.

Balcik, B. (2008). Relief Chain Planning and Management: Modeling and Analyzing Humanitarian Logistic Problems, Ph.D Thesis, University of Washington. Beamon, B. (2004) Humanitarian Relief Chains: Issues and Challenges, 34th International Conference on Computers and Industrial Engineering. Coppola, D. P. (2007) Introduction to international disaster management. Butterworth Heinemann, Burlington. Davidson, A.L. (2006) Key performance indicators in humanitarian logistics, Ph.D Thesis, MIT, Cambridge Gunasekaran, A. and Kobu, B. (2007) Performance measures and metrics in logistics and supply chain management: a review of recent literature (1995–2004) for research and applications, International Journal of Production Research, 45(12), 2819-2840 Hollnagel, E., Woods, D. and Leveson, N. (2006) Resilience engineering: Concepts and Precepts. Ashgate Publishing Company, Burlington. Kim, J. (2009) Activity-based framework for cost savings through the implementation of an ERP system. International Journal of Production research, 47(7), 1913-1929 Kovoor Misra, S. (1995) A multidimensional approach to crisis preparation for technological organizations: some critical factors. Technological Forecasting and Social Change, 48(2), 143-160 Lauras, M., Marques, G. and Gourc. D. (2010), Towards a multi-dimensional project Performance Measurement System. Decision Support Systems, 48(2), 342-353 Rongier, C., Gourc, D., Lauras, M., and Galasso, F. (2010) Towards a performance measurement system to control disaster response, 11th IFIP Working Conference on VIRTUAL ENTERPRISES, Pro-Ve’10, Saint-Etienne, France Rongier, C., Galasso, F., Lauras, M. and Gourc, D. (2010) A method to define a performance indicator system for the control of a crisis, 8th International Conference of Modelling and Simulation MOSIM 2010, Hammamet, Tunisia

10763