Knowledge Acquisition for an Organisational Memory ... - CiteSeerX

Proceedings of the 29th Annual Hawaii International Conference on SystemSciences- 1996

Knowledge Acquisition

for an Organisational

Memory System

A. Sowunmi, F.V. Burstein and H.G. Smith information Systems Department Monash University Melbourne, Victoria, AUSTRALIA

Abstract Decision support systems with a knowledge base containing specijic past decision situations regarding an organisation may be referred to as organisational memory systems. A record of these past decisions is assumed to be beneficial to decision-makers and their organisation. Building such systems involves collecting past decision situations porn a decision-maker. This is analogous to knowledge acquisition where an expert’s knowledge is elicited and modelled. In this paper, knowledge acquisition describes the task of eliciting past decision situations (cases) and their characteristics from an expert decision-maker. The resulting computer system provides case-based organisational memory to support decision-making. The paper proposes a knowledge acquisition method for the development of case-based organisational memory. The method combines a number of techniques and is illustrated by a real-world application.

1 Introduction It has been suggested that components of organisational memory span the entire range of information (internal and external) regarding a particular organ&&ion [ 11. This implies the existence of a variety of information sources and retention facilities. In this paper the concept of organisational memory is discussed from a decision support perspective.

DecisionSupportSystems@SS)areusedto improve

the processes and outcomes of decision-making. The emphasis is on supporting rather than replacing the human decisionmaker[2]. A DSS with a knowledge base of past decision situations specific to an organisation may be referred to as an Organisational Memory System (OMS). Knowledge of such past decisions (stored as cases) is beneficial to decision-makers and their organisation by providing information regarding historical events. Further, such past cases extend decision-makers’ experience thus providing a form of decision support,

1060-3425/96$5.00O1996IEEE

The resulting case-basedOMS fosters consistency in decision-making by ensuring that relevant past caseshave been considered. The decision-maker is not obliged to retain a mental record of all past decisions in the organisation This is advantageous where such a mental record is easily lost either through oversight or frequent change in staffing. Also, the cognitive load on the decision-maker is reduced. Such a system is presented as an OMS because it records and retains relevant decisionmaking information regarding an organisation. Building such an OMS involves collecting past cases from an expert decision-maker who has relevant problemsolving knowledge. This process is analogous to knowledge acquisition where an expert’s cognitive structures are elicited and modelled to build expert systems. Existing research acknowledges that the knowledge acquisition task requires considerable effort. The nature of this task does not favour the introduction of prescriptive methodologies which can be applied for all conditions. Bather it is advisable to provide a knowledge engineer with guidelines which can be adapted to suit the relevant problem domain. The type of OMS discussed in this paper differs from an expert system in a significant way. Expert systems were designed to replicate and possibly replace human experts in solving cognitive problems. With case-based OMS the intention is to support rather than replace a human decision-maker. This is done by providing the relevant past cases from which the decision-maker reasons analogically. This paper proposes a method for eliciting significant, past decision situations in the form of cases and their characteristics from a decision-maker for an OMS. The following sections classify organisational memory and discuss knowledge representation issues. The An organisational memory life-cycle is also discussed. overview of knowledge elicitation and modelling from a perspective of organisational memory is provided. The proposed method is illustrated with a project currently being undertaken 2.1 Classifications of Organisational Memory Organisational memory has been classified from several perspectives. Huber [3] noted that the components of organisational memory are typically located either

168

Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE

Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 within a human mind or a computer system. This paper addresses the issue of eliciting relevant components of organisational memory from an expert decision-maker and suggests how such information can be represented in a case-basedOMS. Stein [4] classifies organisational memory by level of abstraction (either concrete or abstract) and normative orientation (either prescriptive or descriptive), A casebased OMS can be described as either ConcreteDescriptive or Concrete-Prescriptive. The level of abstraction is concrete because actual decision situations are used. The normative orientation is descriptive where the contents of the decision situations provide illustrations for a certain context, a& prescriptive where explicit rules for decision-making exist. An additional dimension for classifying organisational memory is from the perspective of the person whose experience is being represented. In light of this, organisational memory may be either communal or idiosyncratic. Communal organisational memory describes the information elicited from groups of people who make decisions impacting on their organisation. The objective is to reach a consensus of views. Idiosyncratic organisational memory contains information which is elicited from each decision-maker separately. In this instance, differing (and perhaps conflicting) views are preserved and represented as alternative approaches for addressing future, related decision situations. Typically the latter classification would comprise decision situations from personal experience. In organisations where a supervisor’s views override those of the subordinates an idiosyncratic OMS will reflect more of the supervisor’s perspectives.

involves addressing a problem situation by remembering and possibly adapting previous similar situations as humans appear to do, i.e., reasoning by analogy [7, 81. CBR systems perform adaptation on similar cases to “solve” a current problem and often assume the role of decision-maker. We maintain that DSS should assist in exploring a current decision situation rather than replace the human decision-maker [9]. For this reason our use of CBR technology is concerned with functionality derived from case retrieval without adaptation. Consequently, the responsibility of making the “best” decision remains with the human decision-maker.

3 Life-cycle of Organisational Memory Organisational memory can be illustrated through Stein’s four processes of acquisition, retention, maintenance and retrieval [4]. A similar framework is suggested for the life-cycle of case-based OMS. The framework is based on three main stages: reception, retention and recall of information. These life-cycle stages can be compared with Knowledge-Based Systems (KBS) development phases.The reception stage is comparable to knowledge elicitation and modelling; the retention stage to knowledge representation and maintenance; and the

2.2 Knowledge Representation in Organisational Memory Systems Knowledge representation in OMS has developed along with advances in computer technology. Weaver and Bishop [5] recommended ways to build and manage an organisation’s “corporate memory” and discussed issues including data storage requirements and the utility of mainflame computers. More recently, Rao and Goldman-Segall[6] discussed the use of “multimedia” technology as it relates to storage, retrieval and presentation of contextual information (in the form of stories) within an organisation. They argued that the ability to retain the context of incidents occurring within an organ&ion is a major advantage of recording knowledge as stories. This approach is similar to the current use of cases to record organisational memory. Such approaches can be contrasted with systems where specific incidents have been generalised in order to formulate rules thereby filtering out contextual information. With case-based OMS, past decision situations are represented as cases. This approach employs concepts from Case-Based Reasoning (CBR) technology. CBR

Figure I. The Life-Cycle of an Organisational Memory System

3.1 The Reception Stage In the reception stage an expert decision-maker’s cognitive structures are articulated with the assistanceof a knowledge engineer. The role of the knowledge engineer is that of facilitator. With case-based organisational memory, expert decision-makers identify significant cases and their characteristics based on past experience. Further, Bcision-making heuristics employed by the expert which reflect domain knowledge are articulated in this stage. Key problems with general knowledge acquisition are: the difficulty in eliciting knowledge from experts, ambiguity between knowledge and its representation and

169


Proceedings of the 29th Annual Hawaii International Conference on SystemSciences- I996 cases, can be improved if the expert decision-maker deems it necessary.Typically this occurs where the expert refines some information previously articulated or is not satisfied with the knowledge representation in the OMS. Testing the performance of the system is a validation process.

the lack of a framework for validation by the expert [lo, 11, 121.These problems are addressedthrough the use of mediating and intermediate knowledge representations to bridge the gap between an expert’s conceptual model and the final system implementation formalism [ 131. 3.2 The Retention Stage

3.4 Evaluating the Performance of an Organisational Memory System

Maintenance and loss of information or knowledge are integral parts of the retention stage. Maintenance of the OMS requires the facility to add new cases where new aspects of the decision-making domain are present. This facility ensures that the knowledge represented in the OMS is dynamic to reflect what happens in reality. It is also important to record the context and outcome of each case in order to support later decision-making. Loss of information is addressed from two perspectives. First is the situation where loss of information is beneficial to an organisation, e.g., where an incident in an organisation’s past inhibits a decisionmaker from making objective judgments. By “forgetting’ such incidents (and removing them from the OMS) the organisation benefits through an improvement in the quality of decision-making. Second is the situation where loss of information is detrimental to an organisation’s success, e.g., where an error made in the past is not “remembered” by a decision-maker when a similar situation is encountered. Failure to recall such a case can lead to the same error being committed. This example emphasisesthe need to populate the case base (i.e.,, set of cases) with those decision situations where errors would be costly for an organisation. Through an appropriate retrieval mechanism past cases are presented by the OMS to alert the decision-maker, unaware of past mistakes, of the potential risks associated with such cases. Consequently, the possibility of repeating similar error situations is reduced. Three operations are associated with the retention stage. First, the relations between concepts in the domain are modified to reflect the expert’s current views. Second, the cases elicited from an expert decision-maker are represented using the elected implementation formalism. These two operations are comparable to knowledge representation where an expert’s views and judgments are represented in a KBS. Third, the expert indicates when particular cases are no longer relevant to the organ&&ion and should be removed from the OMS (i.e., loss of information).

The validation process is present within each stage of the system development life-cycle so as to minimise the possibility of distorting the expert decision-maker’s cognitive structures. Three of the five factors suggested by Gaschnig et al. [14] for the evaluation of expert systems are applied in the current discussion: . the quality of the retrieved cases, i.e., relevance to the context of a current decision situation, . the correctness of the relations existing between concepts and/or characteristics of cases and their degree of importance in a decision situation, as determined by the expert decisionmaker, . the effectiveness of the system. A major indicator to consider during validation is the relevance of retrieved cases to the context of a current decision situation It is through such a process that a decision-maker using the OMS can best determine whether or not the retrieved cases assist in decisionmaking. The correctness of the relations between concepts and/or characteristics of cases is also important during validation. Expert decision-makers can perform the validation process using graphical representations of their cognitive structures. The effectiveness of the system is discussed from the perspective of the trade-off between the cognitive effort expended by a decision-maker and the improvement in decision quality. The effectiveness of DSS is often evahmted based on two assumptions: there is a reduction in cognitive demands placed on decision-makers and hence an increase in their available cognitive processing capacity; and decision-makers use the “additional’ cognitive processing capacity to address decision situations more deeply, thereby improving decision ww. The first assumption can be argued to be valid where a decision-maker is not obliged to retain a mental record of past decision situations. The second assumption is more difficult to prove mainly because it is unrealistic to make generalisations from such a subjective assertion. Todd and Benbasat [15] argued, from empirical studies they conducted, that there is no evidence to support the veracity of this second assumption. They concluded that the use of computer-based decision aids does not imply that decision situations will be analysed more deeply. In light of their conclusion, it is more appropriate to investigate the issue based on results obtained from an individual rather than make general&&ions. An important

3.3 The Recall Stage The performance of the case-based OMS is tested in the recall stage. The expert decision-maker tests for adequacy and completeness of the system. This can be done either by using an actual decision situation or a hypothetical case and noting how relevant the retrieved cases are to the context of the current situation The performance of the OMS, i.e., the quality of retrieved

170


Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 result from their experiment is that decision-makers place a high premium on the opportunity to minimise the cognitive effort required to address decision situations. This suggests that the effectiveness of an OMS is measurable by the extent to which it minimises the cognitive demandsplaced on a decision-maker.

4 Advantages and Limitations of OMS for Decision Support A significant advantage in using an OMS for decision support is that the system augments a decision-maker’s memory. Kolodner [7] noted this as an advantage of casebased decision support. Orr 1161 and Kolodner also observed how people often find it easier to solve problems or perform a cognitive task if they had previously done something similar. Another advantage is the ability to retain experiences of expert decision-makers even after they leave an organisation. Knowledge gained from these experiences can be reused by other decision-makers. This is beneficial assuming that the cognitive effort required in solving problems is reduced as one gains knowledge from past, related situations. The decision support offered by an OMS is limited by the variety, content and usefulness of the cases stored in it. A system which lacks a sufficient variety of caseswill not provide support for a wide range of decision situations. Also, if the outcome of a decision is not explicitly represented in the OMS then the usefulness of such a case is questionable. A possible way to overcome the variety limitation is to provide a facility where the expert decision-makers are encouragedto identify a set of cases significant in the domain. The knowledge acquisition method proposed in this paper addressesthis limitation by attempting to elicit a representative set of cases. It is important to provide a facility to incorporate new casesinto the system as they are encountered In this way the system learns from experience. Our approach to deal with the content limitation is to ensure that each case represented in the OMS contains a record of the outcome of the corresponding decision situation. By addressingthe vuriefv and content limitations the problem of usefulness of stored caseshas been overcome. A more technical limitation can be observed from the quality of the retrieval algorithms employed in the system. This limitation is addressedby ensuring that cases are appropriately indexed so that retrieved cases are relevant to a decision-maker’s current context.

compare and contrast the set of elicited cases. Using the technique the expert decision-maker identifies the salient characteristics which distinguish the cases. These characteristics serve as indexes to uniquely identify cases and facilitate the retrieval of appropriate cases from the OMS. Also, a technique for creative and exploratory thought is used to direct attention towards aspects of personal experience which might otherwise be overlooked. The PMI technique [ 181is applied for this the purpose in the attempt to ensure adequate representation of the domain. With personal construct theory, a construct is a dimension of appraisal which people employ to evaluate their experiences and judgments. The entities which are relevant to a particular context are referred to as elements, (in our context, past decision situations). A construct is a model of a mental structure comprising dichotomous poles used in contrasting these elements. The rationale for considering personal construct theory stems from parallels with the theory of case-based reasoning. First, there is a similarity between Kelly’s [17] model of human activity and the human cognitive model on which CBR is built. Both models involve anticipation of future events based on an understanding of past events and experience. Second, the repertory grid technique focuses on idiosyncratic views of one’s experiences. Such views are embodied in CBR systems by representing an expert’s experience and past judgments. Knowledge acquisition techniques based on personal construct theory have been developed [19, 201. The method described in this paper utilises the constructs resulting from the application of the repertory grid technique to index and populate a case-basedOMS. The technique is applied noting some criticism on the lack of rigorous validation of grids 1211. Results from the application of the method indicate the usefulness of the repertory grid technique to compare and contrast decision situations. The objectives of the proposed method are to: . elicit a set of past decision situations representative of a specific decision-making domain in an organisation, l provide indexes for storing and retrieving cases from the OMS, 0 elicit decision-making heuristics. The method mainly addresses the reception stage of the OMS life-cycle. The method is divided into three main sections as shown in Figure 2 where the three sections have been numbered indicating the sequence of effecting the method.

5 The Proposed Method The proposed knowledge acquisition method is applied to elicit cases and their characteristics from an expert decision-maker. The method incorporates a combination of techniques adapted to suit the current context. The repertory grid technique [17] is used to

171


Proceedings of the 29th Annual Hawaii International

1996

are more abstract thereby making it difficult to identify an important way in which two elements are alike and different from a third. The verbal label used in distinguishing the elements provides the emergent pole of a bipolar construct. The reverse of this label is the implicit pole of the same construct. An example of a construct is the bipolar representation cold - warm showing that opposing poles are not necessarily simple negations of each other. These verbal labels (poles of a construct) are interpreted as candidate indexes for uniquely identifying casesin the OMS. In the second step the remaining set of elements (not used in the triad or dyad) are rated and positioned along the axis between the two poles. More constructs are generated by repeating the procedure for other triads or dyads. The constructs provide significant characteristics used for making judgments on cases. The third step involves the construction of the repertory grid. This grid is a two-dimensional matrix of n constructs by m cases where the elements of the matrix are the ratings determined in the preceding step. The grid cau be analysed and represented in a graphical form.

Design set of questions and didt

k characteristics from 2 to elicit further casea

Conference on System Sciences -

use case8 as repertory glid elements to didt dlarscteristics of -

Figure 2. Schematic representation ofproposed method Each section of the diagram comprises a range of activities involving both the knowledge engineer and the expert decision-maker. 5.1 Applying Segment 1 In segment 1, the knowledge engineer designs a set of questions to elicit the initial set of cases (see section 6 for :tae set used in the application of the method). These questions are formulated after a series of semi-structured interview sessionswith the expert decision-maker. During these interview sessionsthe knowledge engineer acts as a catalyst prompting for key domain concepts which are then used to formulate the questions. Each question prompts the expert decision-maker to recall a decision situation (case) where specific domain concepts have been activated. This presents the knowledge engineer with an initial set of casesto be represented in the OMS. In designing these questions the purpose is to elicit those past decision situations which affected the expert decision-maker and/or the organisation in some significant way. Also, the response to these questions provide instantiations of specific domain concepts. Each case is significant in a certain decision-making context.

5.3 Applying Segment 3

5.2 Applying Segment 2 The main purpose of this segment is to elicit candidate indexes for uniquely identifying each of the cases provided in segment 1. This is done by using the repertory grid technique to repeatedly compare and contrast cases (i.e., elements). The result of this process is a set of constructs which embody the expert decision-maker’s view of the domain, This process is described in more detail below. First, the expert decision-maker is presented with either two or three cases, randomly selected from the set provided in segment 1. Where three elements (triads) are used, the expert decision-maker is prompted to identify an important way in which two of the cases are alike and thereby different from the third. Where two elements (dyads) are used, the prompt is for an important difference between the two cases. It is more common to use triads for this first step; dyads are typically used where elements

172


The third segment further explores the domain identifying new cases which reflect constructs not previously mentioned by the expert decision-maker. The objective is to elicit a set of caseswhich are evenly spread across the domain. This is in keeping with the objective of the method, i.e., to elicit a set of cases which are representative of the domain resulting in a “good” case base. Continuation of the repertory grid technique in the third segment would require using characteristics of cases as elements to generate further cases. Such an approach has not been pursued for two main reasons. First is the cognitive demand placed on the expert decision-maker by having characteristics of cases as repertory grid elements rather than the casesthemselves. These characteristics can be difficult (or impossible) to use as elements where they are not entities which can be compared. The second reason is because repetition of the technique can result in the expert decision-maker reproducing previously elicited cases. We suggest using alternative approaches leading to the discovery of caseswhich might otherwise be ignored. The application of techniques for creative and exploratory thought allows the expert decision-maker to perceive characteristics of cases elicited in segment 2 in a novel way. An example of such a technique is the PMI [ 181. The PMl has been considered as a way to identify those cases which are infrequently encountered by the expert decision-maker. The subtle and often salient distinctions between superficially similar cases can also be identified. Application of the PM1 technique in the current method requires the expert decision-maker to think along the Plus direction to identify the cases where a particular

Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 characteristic was regarded as a positive factor for decision-making. Then, along the Minus direction where a characteristic contributed to a negative decision. Finally, along the Interesting direction where a characteristic played neither a positive nor a negative role but was worth noting for the decision-making. The PMl technique is used here ‘in order to direct attention to those aspects of existing experience which might otherwise be ignored’[ 181.Given that the objective of segment 3 is to elicit further cases from the expert decision-maker, the knowledge engineer asks questions such as “can you give an example of a decision situation where this characteristic constituted a positive/ negative/ interesting factor?” Each of the three dimensions of the PMI are explored sequentially. By providing explanations for past judgments the expert decision-maker elaborates on the decision-making strategies and heuristics which were employed. The PM1 technique is applied here by asking questions such as the following: Why would you consider this characteristic to be a positive/ negative/ interesting factor (in a specific decision situation)? How does this characteristic represent a positive/ negative/ interesting factor (in a specific decision situation)? By answering the why questions the expert decisionmaker provides justifications for judgments made. It is anticipated that these justifications will provide further insights to the strategies and heuristics employed. Answers to the how questions provide further illustrations regarding the heuristics which have been applied. Representing these strategies and heuristics as meta-rules in the case base component of the OMS results in a system which reflects relevant decision-making knowledge. l

l

fundamental precept of DSS where such systems assist the decision-maker to better structure a problem. The capacity to assist the decision-maker will be limited by the extent to which cases in the case base are representative of the relevant domain. Consequently, an attempt should be made to ensure that the value of the system does not decline due to such limitations. The feedback loop is introduced to address this issue by ensuring thorough evaluation of the domain within the limits of practical efficacy. The intention is to minimise the possible occurrence of instances where the case base lacks cases relevant to the context of a current decision situation. However, the case base of an OMS maintains the ability to “learn” by adding caseswhich illustrate new aspects of the domain as discussed in section 3.2. Such functionality is beneficial in dynamic enviromnents where new concepts/criteria are considered in the decisionmaking process. Through the feedback loop the expert decision-maker explores the domain in more depth. The feedback loop initiates a mechanism for eliciting a set of caseswhich are representative of the domain. This is expected after iterating through the feedback loop. The number of iterations depends on the particular instance where the method is used. By addressing the following issues the knowledge engineer will appreciate the extent to which the elicited casesrepresentthe domain. How do the cases in segment 1 compare with those in segment 31 What differences (if any) exist between both sets of cases? Are the cases in segment 3 better than those in segment 11 Do the cases in segment 3 shed new light on the problem domain? The knowledge engineer can ascertain tbis by observing whether the casesprovide further insights. How do the characteristics elicited from successive iterations compare with one another? Do they reflect deeper, rather than superficial, aspectsof the problem domain? If the characteristics elicited from successive iterations are different, do they prompt for further cases?On the other hand if the characteristics are not different then the implementation of the feedback loop may have to be tijusted. This is done in order to limit the extent to which such a situation results from the knowledge engineer’s shortcomings. Does the feedback loop foster a convergence of the complete set of elicited cases. Do cases elicited in segment 3 continue to exhibit novel charactelistics? When new cases no longer exhibit novel characteristics, compared with existing cases, it is assumedthat the domain has been adequately represented. This conclusion is tested by having a different expert decision-maker validate the adequacy and completeness of the elicited casesand characteristics. A different expert decision-maker (with knowledge of the same domain) l

l

l

5.4 Implementing the Feedback Loop The new casesprovided in segment 3 are used as input for another pass of segment 2. These new cases are used as repertory grid elements resulting in a feedback loop between segments 2 and 3. The feedback loop allows reexamination of the expert decision-maker’s knowledge, facilitates the recollection of relevant cases and validates the adequacy and completenessof the elicited cases. The justification for introducing the feedback loop stems from a significant distinction between case-based OMS used for decision support and CBR. With CBR systemswhen the case base does not contain casessimilar to a new situation, the new situation is added to the case base. The case base then grows with use and the system “learns” from such situations. Such an evolving casebase is acceptedas a strength of CBR systems. In contrast, from a decision support perspective such an evolving case base is more of a limitation. Instances where the case base does not contain “similar” cases reduces the value of the system. This is in keeping with a

173


l

Proceedings of the 29th Annual Hawaii International

1996

was more recent, and therefore more valuable, than any similar cases for either question. One question was not answered, the nzason given being that no such cases had been encountered. As a result, eleven unique cases were generated in segment 1.

brings a new perspective to the situation and can identify critical issues which are not reflected in the cases elicited from the first expert decision-maker.

6 Application

Conference on System Sciences -

of the Method

Table 1. Questions to elicit decision situations (cases)

In choosing an example to test the proposed method we considered a situation involving repetitive decisionmaking in a poorly structured domain. The Faculty of Computing and Information Technology at Monash University in Melbourne admits a number of students with prior tertiary qualifications from Australia and overseas. Typically such students would have begun a course of study and/or have relevant industry experience which can be converted into credit points towards their degree at Monash University. Decision-makers at the faculty level assess applications from such students and determine equivalence of prior studies and/or experience to the components of the chosen study programme. Given recent changes to the Faculty’s courses and the broad range of backgrounds of the applicants, the task of determining credit equivalence is a daunting one. Student applications for credit are referred to as cases. These cases can be classified broadly as either ‘straightforward’ or ‘complex’. With the straightforward cases there are existing agreements (rules) between the University and the relevant institutions regarding how much credit is granted to applicants, based on their prior qualifications. With the complex cases, on the other hand, such rules do not exist and consequently judgments are often made based on the decision-maker’s intuition. By applying a form of analogical reasoning for such complex cases (using past “similar” cases) it is anticipated that a higher level of consistency in subsequent decision outcomes will be attained. The case-basedOMS will perform a search operation in order to retrieve cases which are similar to the current credit application. The “most similar” cases are retrieved from the case base for the decision-maker to consider. We anticipate that these similar cases will be of value in deciding what credit to offer in new situations. We applied the proposed method to the knowledge acquisition stage for building a system to support the Faculty’s credit transfer decision-making. The system will provide decision support for those cases where neither rules nor precedents exist and will be responsive to changes in offerings by both the Faculty and other relevant institutions. The system is populated with cases elicited by our method, while identified characteristics provide candidate indexes. Decision-making heuristics identified in the application of the method are represented as me&rules in the case base component of the OMS. In applying the method a number of Observationswere made. Thirteen domain-specific questions were used for segment 1 to elicit an initial set of cases with the aim of providing a unique case for each question (see Table 1). Two questions generated the same case in response, the expert decision-maker arguing that the case in question

1. A case where information in the initial application was insufficient to make a decision. 2. A case where information in the application was excessive. In other words, a decision was made before going through the entire application. 3. A case from an institution which produces a large number of applications. 4. A case where credit was rejected. 5. A case where credit was granted, but different from what was requested for. 6. A case which involved protracted correspondence with the applicant while gathering further details. 7. A case where credit was provisionally granted. 8. A case where credit was repeatedly requested by the same applicant. 9. A typical case. 10. An interesting case. 11. A case supported by translations into English. 12. A case where the final decision was contrary to your recommendation. 13. A case where the applicant qualified more than two years prior to applying. With segment 2, a commercial knowledge acquisition tool NEXTRATM which automates the process of repertory grid construction was used. It offers support for processing grid data as follows: . random selection of elements (triads or dyads), . visualisation of a construct as a rating bar along which elements are positioned reflecting the expert decision-maker’s intuition, . rapid construction of repertory grids and other graphical output for data analysis. All eleven cases provided in segment 1 were used as repertory grid elements. The expert decision-maker had some difficulty distinguishing the cases in the entire set. This process also proved time-consuming. These two observations prompted a revision of the second pass of segment 2 (in the feedback loop) where fewer (eight) elements were used. Dyads (sets of 2 elements) were used for construct elicitation in the feedback loop in the attempt to reduce the cognitive demand on the expert decision-maker. Of the three graphical representations of repertory grid data offered by NEXTRA, the expert found the spatial cluster display easiest to understand. Appendix A shows the spatial cluster display generated by NEXTRA using an algorithm based on principal components analysis. The spatial distribution of

174


Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 cases along the multiple dimensions (representing the elicited constructs) indicates the degree of similarity between the cases. The more similar cases are closely located in the display. Four bipolar constructs (characteristics of cases) elicited from the expert decision-maker in segment 2 are shown in Appendix A. The construct applied for articulation/ advanced standing into second year applied for spec$c subject credit was repeated by the expert decision-maker in segment 2. As a result six unique characteristics were generated in segment 2, as mentioned earlier. The display in Appendix A was presented to the expert decision-maker as a graphical way to verify the similarities and differences between cases. The knowledge engineer prompted the expert decision-maker for distinctions between “similar” cases (i.e., closely located in the display) to further distinguish such cases. Where no distinctions were identified, the expert decision-maker was asked to rationalise the similarity between such cases. Applying segment 3 of our method resulted in the elicitation of decision-making heuristics. By providing explanations for judgments made regarding the elicited cases the expert decision-maker expressed relevant heuristics employed. These explanations were focused by applying the PMI technique, as described in section 5.3. The expert decision-maker and the knowledge engineer cycled the three-stage method and implemented the feedback loop resulting in the identification of 14 cases and 20 characteristics. The set of eleven cases provided in segment 1 were used as repertory grid elements in segment 2 resulting in the identification of six characteristics (as poles of constructs). Three new cases were provided in segment 3. Implementing the feedback loop resulted in the identification of eight new characteristics used in distinguishing the cases. In keeping with the proposed method, a different expert decision-maker (with knowledge of the domain) validated the adequacy and completenessof the casesand characteristics provided by the first expert decisionmaker. The purpose of this validation was to determine the extent of coverage of the domain by the first expert decision-maker. It was important to select a second expert decision-maker with specific knowledge of the cases elicited from the first expert and comparable involvement in the Faculty’s credit transfer decision-making. The second expert decision-maker ascertained the extent to which the cases and characteristics identified by the first expert decision-maker were representative of the domain. Further insights were provided by the second expert decision-maker. Six new characteristics were identified during a session using the repertory grid technique, similar to the way the technique was used for the first expert decision-maker (see Appendix B). These six characteristics

were presented to the first expert

decision-maker who acknowledged their importance in

further distinguishing the cases. The new characteristics elicited from the second expert decision-maker provide further candidate indexes for uniquely identifying casesin the OMS. This outcome underlines the importance of validating the adequacy and completeness of the cases and characteristics elicited from the first expert decisionmaker. Appendix B shows three bipolar constructs (characteristics of cases) indicating the further insights provided by the second expert decision-maker. Note that identical reference cases were used by both expert decision-makers. The performance of the method was determined by the extent to which the elicited cases characterised the complex (and infrequently encountered) decision situations in the credit transfer domain. Of the 14 elicited cases, eleven characterise such decision situations. The other three involve the more commonly encounteredcases where rules and/or precedentsexist.

7 Conclusions In this paper we have discussed organisational memory from the perspective of decision support. A casebased DSS is argued to be an OMS because it records and retains relevant decision-making information regarding an organisation. A case-based OMS stores organisational experience as cases. We identified the advantages, limitations and possible solutions relevant to case-based OMS. Using such systems, decision-makers reason by analogy from a collection of cases representing past experience. One of the most important aspectsof developing casebased systems, which has not been well researched, is collecting and representing knowledge in the form of cases. A knowledge acquisition method for developing case-based OMS was proposed and discussed. Application of the method resulted in the elicitation of a number of the ‘complex’ (and less obvious) cases and their characteristics from an expert decision-maker. Elicited cases are then represented in a case-basedOMS with their distinguishing characteristics providing a basis for indexing. Application of the method also resulted in the articulation of decision-making heuristics which are represented as meta-rules in the case base to reflect relevant decision-making knowledge. The use of a realworld decision-making environment illustrates the utility of the method. Further work with this research includes testing the quality of the case base. A prototype system is currently being developed using a commercial CBR shell, ESTEEM?. The prototype will serve as an OMS for the Faculty staff involved in credit transfer decision-making. This will allow us to assess the utility of the proposed method and the possibility of generalising it across decision-making

175


domains.

Proceedings of the 29th Annual Hawaii International Conference on System Sciences- 1996 Appendix A. Spatial cluster displqfiom

the expert decision-maker

applied for specific subject credit into non-computing area ca&f I

arliculatiorl

applied for

applied for . .

ez$nding

appliedfow specific subject credit

ease 13 = Ca# / articulation international c(Im$

applied for articulabonl advanced standing into second year

case;

~cass7 *applied for non-specified units/minor I

S9CjlBnCe

Appendix B.. Spatial cluster display@om the second expert decision-maker

CaoelO more thap 10 year dd I ,paMicatwn

essthan loyeardd cp&ific&ion

[2] Keen, P.G.W. and Scott Morton, M.S. (1978). Decision Support Systems:An Organisational Perspective.Mass.:

Acknowledgements

Addision-Wesley. [3] Huber, G.P. (1990). ‘A Theory of the Effects of Advanced Information Technologies on Organisational Design, Intelligence, and Decision Making’, Academy of Management Review. 15: 1. [4] Stein E.W. (1995). ‘Organisational Memory: A Review of Concepts and Recommendations for Management’, International Journal of Information Management 15: 1.

We are grateful to our colleagues and the reviewers who made insightful

suggestions

regarding

the issues

discussed in this paper. We also thank the two experts who contributed their time and knowledge to support the Credit Transfer Project.

References

[5] Weaver, B.N. and Bishop, W.L. (1974) The Corporate Memory. New York: John Wiley.

[l] Walsh, J.P. and Ungson, G.R. (1991). ‘Grganisational Memory’, Academy of Management Review. 16: 1,57-91.

176


Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 [6] Rae, V.S. and Goldman-Segall, R. (1995). ‘Capturing Stories in Organisational Memory Systems: The Role of Multimedia’, Proceedings of the 28th Annual Hawaii International Conference on Systems Sciences. [7] Kolodner, J.L. (1991). ‘Improving Human Decision Making through Case-BasedDecision Aiding’, AI Magazine, 12:2. [8] Kolodner, J.L. (1993). Case Based Reasoning. California: Morgan Kaufmann. [9] Burstein, F., Sowunmi, A., Smith, H., McMillan, A. and Cole, C. (1995). ‘Building an Intelligent Decision Support System’, in Proceedings of PanPacific Conference on Information Systems. [lo] Feigenbaum, E.A. (1977). ‘The Art of Artificial Intelligence : Themes and Case Studies of Knowledge Engineering’, Proceedings of Joint International Conference on Artificial Intelligence. [l l] Ford, K.M., Bradshaw, J.M., Adams-Webber, J.R. and Agnew, N.M. (1993). ‘Knowledge Acquisition as a Constructive Modelling Activity’, International Journal of Intelligent Systems. (eds.) K. Ford and J. Bradshaw. New York: John Wiley. 8: 1,9-32. [12] Clancey, W.J. (1993). ‘The Knowledge Level Reinterpreted: Modelling Socio-Technical Systems’, Intemational Journal of Intelligent Systems. (eds.) K. Ford and J. Bradshaw. New York: John Wiley. 8: 1,33-49. [13] Nwana, H., Paton, R., Shave, M. and Bench-Capon, T. (1992). ‘Domain-Driven Knowledge Modelling: Mediating and Intermediate Representations for Knowledge Acquisition’, EKAW ‘92. (eds.) Th. Wetter, K.-D. Althoff, J. Boose, B. Gaines, M. Linster and F. Schmalhofer. Berlin springer-vedag.

[ 141 Gaschnig, J., Klahr, P., Pople, H., Shortliffe, E. and Terry, A. (1983). ‘Evaluation of Expert Systems: Issues and Case Studies’, Building Expert Systems. (eds.) F. Hayes-Roth, DA. Waterman and D.B. Lenat. Mass.: Addison-Wesley. [ 151 Todd, P. and Benbasat, I. (1992). ‘The Use of Information in Decision Making: An Experimental Investigation of the Impact of Computer-Based Decision Aids’, MIS Quarterly. September. [16] Grr, J.E. (1990). ‘Sharing Knowledge, Celebrating Identity: Community Memory in a Service Culture’, Collective Remembering. (eds.) D. Middleton and D. Edwards. London: Sage Publications. [17] Kelly, G.A. (1955). The Psychology of Personal Constructs. New York: Norton. [ 181 DC Bono, E. (1988). Letters to Thinkers: Further Thoughts on Lateral Think&g. London: Penguin. [ 191 Boose, J. and Bradshaw, J. (1987). ‘Expertise Transfer and Complex Problems, Using AQUINAS as a Knowledge Acquisition Workbench for Expert Systems’, Proceedings of the Knowledge Acquisition for KBS Workshop, Banff. (eds.) T. Addis, J. Boose and B. Gaines. [20] Shaw, M.L.G. and Gaines, B.R. (1987). ‘An Interactive Knowledge Elicitation Technique using Personal Construct Psychology’, Knowledge Acquisition for Expert Systems: A Practical Handbook. (ed.) A.L. Kidd. New York Plenum. [21] Yorke, D.M. (1985). ‘Administration, Analysis and Assumption: Some Aspects of Validity’, Repertory Grid Technique and Personal Constructs: Applications in Clinical and Educational Settings. (ed.) N. Beail. Mass.: Brookline Books.

177
