problem domains in business has grown steadily since their introduction. Regardless of the chosen method of development, the most commonly cited problems ...
Article Selection of knowledge acquisition techniques based upon the problem domain characteristics of production and operations management expert systems
we present a mapping between these empirical studies and a generic taxonomy of expert system problem domains. In so doing, certain knowledge acquisition techniques can be prescribed based on the problem domain characteristics. With the production and operations management (P/OM) field as the pilot area for the current study, we first examine the range of problem domains and suggest a mapping of P/OM tasks to a generic taxonomy of problem domains. We then describe the most prominent knowledge acquisition techniques. Based on the examination of the existing empirical knowledge acquisition research, we present how the empirical work can be used to provide guidance to developers of expert systems in the field of P/OM. Keywords: expert systems development, knowledge acquisition, problem domain, empirical research, production and operations management
William P. Wagner, Mohammad K. Najdawi and Q.B. Chung Department of Decision and Information Technologies, Villanova University, Villanova, PA 19085, USA E-mail: william.wagner얀villanova.edu
Abstract: The application of expert systems to various problem domains in business has grown steadily since their introduction. Regardless of the chosen method of development, the most commonly cited problems in developing these systems are the unavailability of both the experts and knowledge engineers and difficulties with the process of acquiring knowledge from domain experts. Within the field of artificial intelligence, this has been called the ‘knowledge acquisition’ problem and has been identified as the greatest bottleneck in the expert system development process. Simply stated, the problem is how to acquire the specific knowledge for a well-defined problem domain efficiently from one or more experts and represent it in the appropriate computer format. Given the ‘paradox of expertise’, the experts have often proceduralized their knowledge to the point that they have difficulty in explaining exactly what they know and how they know it. However, empirical research in the field of expert systems reveals that certain knowledge acquisition techniques are significantly more efficient than others in helping to extract certain types of knowledge within specific problem domains. In this paper 76
1. Introduction The use and application of expert systems to various problems within the functional areas of business has grown steadily since their introduction. These systems have been developed completely in house, or purchased as proprietary software, or developed using an expert system shell. The most commonly cited problems in developing these systems are the unavailability of both the experts and knowledge engineers and difficulties with the rule extraction process (Hoffman, 1987). Within the field of artificial intelligence this has been called the ‘knowledge acquisition’ problem and has been identified as the greatest bottleneck in the expert system development process (Boose, 1989). Simply stated, the problem is how to acquire the specific knowledge for a well-defined problem domain efficiently from one or more experts and represent it in the appropriate computer format. Given the ‘paradox of expertise’, the experts have often proceduralized their knowledge to the point that they have difficulty in explaining exactly what they know and how they know it. However, recent empirical research in the field of expert systems reveals that certain knowledge acquisition techniques are significantly more efficient than others in different knowledge acquisition domains and scenarios. Adelman (1989) was one of the first to design experiments to compare objectively the effectiveness of different knowledge acquisition techniques. As a result, five determinants of the quality of the resulting knowledge base have been identified, namely domain experts, knowledge Expert Systems, May 2001, Vol. 18, No. 2
engineers, knowledge representation schemes, knowledge elicitation methods, and problem domains. This paper presents a mapping between the body of knowledge acquisition empirical studies and the different problem domains within the field of production and operations management (P/OM) in an effort to guide developers of P/OM expert systems in their choice of knowledge acquisition techniques. 2.
A generic problem domain taxonomy
Existing research in the field of knowledge acquisition has focused on several dimensions of the problem as determining factors. One primary determinant of the knowledge acquisition technique used to develop an expert system is the problem domain. To enhance research in the knowledge acquisition field, generic problem domain taxonomies have been developed that cut across functional areas. One commonly used taxonomy breaks problems into general categories of analysis, synthesis, and problems that combine analysis and synthesis (Clancy, 1986; Boose, 1989). This is reproduced as Table 1. Table 1: Generic problem domain taxonomy (Clancy, 1986) Analysis problems 쐌 Classification – categorizing based on observables 쐌 Debugging – prescribing remedies for malfunctions 쐌 Diagnosis – inferring system malfunctions from observables 쐌 Interpretation – inferring situation descriptions from sensor data Synthesis problems 쐌 Configuration – configuring collections of objects under constraints in relatively small search spaces 쐌 Design – configuring collections of objects under constraints in relatively large search spaces 쐌 Planning – designing actions 쐌 Scheduling – planning with strong time and/or space constraints Problems combining analysis and synthesis 쐌 Command and control – ordering and governing overall system control 쐌 Instruction – diagnosing, debugging and repairing student behavior 쐌 Monitoring – comparing observations to expected outcomes 쐌 Prediction – inferring likely consequences of given situations 쐌 Repair – executing plans to administer prescribed remedies
Expert Systems, May 2001, Vol. 18, No. 2
2. 1. Proposed mapping of P/OM expert system tasks to the generic problem domain Table 2 shows how generic task domains can be mapped into P/OM task domains, with selected examples of P/OM expert systems. The product life cycle approach was followed in identifying P/OM functions. These functions include forecasting of demand, capacity planning, aggregate planning, process analysis, selection and design, production scheduling, project management, facility location, inventory management, quality management, and training and work design. These tasks were then placed in the generic taxonomy based upon the generic task descriptions. Because of the complex nature of the operations business functions, some of them fall under multiple task domains. For example, project management is a planning function, a scheduling function and a monitoring, and command and control function. In addition, the process of mapping specific functions to the more abstract categories of analysis, synthesis, and the combination reveals some interesting characteristics of P/OM problems. Looking at the P/OM tasks that fall within the analytic category shows that all of these tasks involve taking a set of data inputs and identifying patterns in them. In contrast, the synthetic problems require that solutions be generated based upon the more general goals of the system and involve the search of a much larger set of potential solutions. Combinations of the two are typically the most ambitious types of expert systems in that they must perform in-depth analysis of large amounts of diverse input data, identify the problems and causes and design a possible solution. The difficulty in this may be the fundamental reason that so few of these types of expert systems have been attempted in the P/OM field (Eom, 1996). 3.
Knowledge acquisition techniques
In the strict sense knowledge elicitation should be viewed as one phase of the knowledge acquisition process but in much of the research the two are used interchangeably. The role that the human knowledge engineer will play in the knowledge acquisition process will vary considerably depending on the particular elicitation technique or method used. In some cases, it may be appropriate for the knowledge engineer to become an apprentice to the expert or participate somehow in the actual problem-solving process. Other times it may be better for the knowledge engineer to conduct an unstructured interview or simply observe the expert perform a given task. Many different techniques have been developed especially for knowledge engineers in these different situations or have been drawn from existing research in fields such as psychology, and several researchers have attempted to summarize these (Hoffman, 1987; Kim & Courtney, 1988; Boose, 1989; McGraw & Harbison-Briggs, 1989; 77
Table 2: P/OM task domains and expert systems Generic task domains
Analysis Classification Debugging Diagnosis Interpretation
Synthesis Configuration
P/OM task domains
Some examples of P/OM expert systems
ABC analysis, process analysis and selection Output rate and staffing level analysis, inventory level analysis, helpdesk Quality control, demand analysis Aggregate planning, statistical analysis, acceptance sampling
GARI (Descotte & Latombe, 1981) HELPDESK (Logan & Keyon, 1992)
Facility layout, distribution and logistics
WORKPLACE (Kangskool et al., 1987) XCON (McDermott, 1982) PEP (Stroebel et al., 1986) FADES (Fisher & Nof, 1984) PATRIARCH (Mortons et al., 1986) ISIS (Fox & Smith, 1984)
Design Planning
Product design, process design Capacity planning, facility location, aggregate planning, project management
Scheduling
Product scheduling, project management
Combination Command and control Instruction Monitoring Prediction
Quality management, inventory management, facility location, project management, procurement Training, certifications, skill development Project management Forecasting
Repair
Maintenance
Tuthill, 1990). Of these techniques, it should not be surprising that a survey (Cullen & Bryman, 1988) found that the most commonly used knowledge elicitation technique was the ‘unstructured interview’, where the knowledge engineer asks general questions and just hopes for the best. However, each technique requires different abilities from the knowledge engineer, the knowledge source, and allows a different set of knowledge representations to be used. Although the human-based knowledge acquisition techniques described in the following sections are certainly the most common used today, they are certainly not without their problems. Not only do they require an enormous amount of time and labor on the part of both the knowledge engineer and the domain expert, but they require the knowledge engineer to have an unusually wide variety of interviewing and knowledge representation skills in order for them to be successful. Research on the most common elicitation and representation techniques used by human agents is presented in the following sections. 78
DEFT (Pollitzer & Jenkins, 1985) SPC (Miller, 1986)
ASAP (Blanchard, 1994) LOGIX (Allen & Helferich, 1990) CASE (McClay & Thompson, 1989) COMPASS (Prerau et al., 1985) NOSTRADAMUS (Allen & Helferich, 1990) DELTA (Bonnisone & Johnson, 1983)
3. 1. Unstructured interviewing techniques Undoubtedly the most common technique currently used by knowledge engineers (Hoffman, 1987; Cullen & Bryman, 1988), it is difficult to describe unstructured interviewing as a true technique, since as its name implies it is just a wandering conversation between the expert and the knowledge engineer. A transcript of a typical unstructured interview is given as an example in Table 3. Even though this may be the case the unstructured interview still has a valuable place in the knowledge engineer’s bag of tools since it allows the greatest possible freedom for knowledge engineer and expert alike to explore the topic. Many researchers have documented and described its usage although its value as a real tool is usually downplayed by researchers in the field. In fact, an anthropological study of knowledge engineering cited the use of the unstructured interview as one of the biggest failings of knowledge engineers who were attempting to develop Expert Systems, May 2001, Vol. 18, No. 2
Table 3:
Excerpt from an unstructured interview (adapted from Hoffman, 1987)
I: What information are you given about airports? E: Now, just some rudimentary information comes with it. Common name, latitude and longitude, ah . . . no information comes with it about the ah . . . maximum number of airplanes on the ground or the port capability that is at the field. None of that comes along with it. I: What’s the difference between MOG and airport capability? E: Ah . . . MOG maximum on the ground is parknowledgeing spots . . . on the ramp. Airport capability is how many passengers and tons of cargo per day it can handle at the facilities. I: Throughout . . . ah . . . throughout as a function of . . . E: It all sorta goes together as throughput. If you’ve only got . . . if you can only have ah . . . if you’ve only got one parknowledgeing ramp with the ability to handle 10,000 tons a day, then your . . . your throughput is gonna be limited by your parknowledgeing ramp. Or, the problem could be vice versa. I: Yeah . . . E: So it’s a (unintelligible phrase).
expert systems (Forsythe & Buchanon, 1989). Developers, however, generally seem to assume that an unstructured interview is the only way to elicit an expert’s knowledge (Weiss & Kulikowski, 1984). While it is certain that no interview is completely unstructured, different types of unstructured interviews have been suggested by authors of general surveys of knowledge acquisition techniques. In the most extreme case, the knowledge engineer does not have a prepared set of detailed questions and the expert does not have ready replies or information at his/her fingertips. The result is that the interview takes the form of a wandering dialog in which open-ended questions are asked by the knowledge engineer. However, this can be helpful both in acclimatizing the expert to the ideas of artificial intelligence and in helping the knowledge engineer learn general ideas about the problem domain. It serves the additional purpose of building an essential rapport between the system developer and the human source. Unfortunately, this has been shown to be a time-consuming and inefficient process (Burton et al., 1987; Cooke & McDonald, 1987; Hoffman, 1987) and can offend the expert as being a ‘waste of time’ (Forsythe & Buchanon, 1989). The difficulties of the unstructured interview become apparent when one views a sample from an actual interview and sees how inefficient it can be (see Table 3). Expert Systems, May 2001, Vol. 18, No. 2
3. 2. Structured interviewing techniques The notion of the structured or ‘focused’ interview has its roots in psychotherapy research (Merton et al., 1956). From his studies of the effectiveness of wartime radio programs, Merton and his colleagues identified the characteristics of a focused interview which they went on to use effectively in the field of psychotherapy. Basically, they stated that interviews are structured by the careful preplanning of a series of questions and their order. In addition, the allowed activities of the interviewer are also carefully specified. The implied hypothesis is that by providing structure to the interview it can be made more efficient. From this work, psychologists developed other interviewing techniques and tools which were designed to structure the interview process and have been, in turn, generally applied to the knowledge elicitation problem. These techniques can often be applied to situations where the expert is being interviewed while actually performing a task, or where the task is simulated or reconstructed by case studies or scenarios or simply from the expert’s own past experience. Elicitation techniques most commonly discussed in the literature include protocol analysis (Newell & Simon, 1972; Hart, 1985; Cullen & Bryman, 1988), repertory grids (Boose, 1989), prototyping (Waterman, 1986; Grabowski, 1988), multidimensional scaling (Elliot, 1986; Boose, 1989), cluster analysis (Cooke & McDonald, 1987), event recall (Hoffman, 1987), discourse analysis (Belkin et al., 1987) and card sorting (Burton et al., 1987). Some rudimentary structuring can be given to the interview process by having the expert perform a particular task while the knowledge engineer asks questions freely. The task may be typical of the problem-solving situation which the knowledge engineer wishes to explore or it may be a special case, identified in earlier sessions, which the knowledge engineer wishes to use to have the expert refine previously elicited knowledge. The simplest task the knowledge engineer could give the expert could be to prepare a brief lecture designed to lay out the main themes and ideas associated with the particular problem domain. Obviously, this type of task would be more appropriate for early knowledge acquisition sessions whereas the special task would be better for when the knowledge engineer was more familiar with the particular domain (Hoffman, 1987). It should be noted that the tasks which are used as the basis for structuring the interview can be either actual tasks or simulated tasks. This method of structuring the interview process by using specific tasks has been termed ‘constrained processing’ (Hoffman, 1987) and the different tasks were grouped as in Table 4. Interviews can be structured even further by having the knowledge engineer use prepared lists of questions. Different ways of structuring interviews in this way have been 79
Table 4: Task types (Hoffman, 1987) Familiar task activities 쐌 Unobtrusive observation 쐌 Simulated familiar tasks Interview tasks 쐌 Unstructured 쐌 Structured Constrained processing tasks 쐌 Case-based reasoning 쐌 Event recall 쐌 Scaling and sorting tasks 쐌 Creative problem solving 쐌 Decision analysis
studied for some time in the social sciences and the cognitive psychology literature (Osborn, 1953). 3. 3. Protocol analysis Protocol analysis is one of the most frequently mentioned elicitation techniques in the knowledge acquisition literature. Cullen and Bryman (1988) found it to be second only to unstructured interviews in actual usage. Suggested by Newell and Simon (1972), subjects are asked to ‘think aloud’ while solving a problem or making a decision. These verbalizations are usually taped and then transcribed and the transcription is analyzed using a particular coding scheme. The transcript itself is termed a ‘protocol’ and may be used to refer to a word-for-word record or a summary of the major points. Whatever the form of the protocol, it should enable the knowledge engineer to easily access, index and code specific pieces of information. Depending on the problem domain it may be desirable also to generate ‘motor’ protocols or even ‘eye-movement’ protocols to more clearly understand an expert’s performance of a task (Tuthill, 1990). Motor protocols require that the expert’s physical movements be closely observed and noted by the knowledge engineer, which may be appropriate for acquiring certain types of expertise. At an even more subtle level, noting the movement and visual focus of the eyes of the expert as a task is being performed may reveal something of the sensory experience of the expert as he/she performs the task (Tuthill, 1990). However, all protocols can be classified as being either ‘concurrent’ or ‘retrospective’ (Grabowski, 1988). Concurrent protocols are records of the expert’s thought processes at the same time that he/she is solving a problem, while retrospective protocols are records of the expert’s review of his/her verbalizations after the task is completed. These are often used when it is felt that the task of verbalization has interfered with the expert’s performance of the actual 80
task (Grabowski, 1988). An example of a ‘concurrent’ protocol of an expert’s attempt to classify ancient Greek pottery is given in Table 5. The actual transcript of the expert’s thoughts while faced with a given task, e.g. the transcript given in Table 5, is what is called the ‘protocol’. This transcript may in turn be translated by the knowledge engineer into a more formal protocol which attempts to summarize the major points in a format designed for easy access (e.g. using indexing, notation or special coding systems). Once the protocol has been worked into the desired format, the actual analysis of the protocol by a knowledge engineer can begin. The usual method that knowledge engineers use is called ‘process tracing’ and is drawn from cognitive psychology (McGraw & Harbison-Briggs, 1989). Analysis involves breaking down the decision rules used by the expert into typical, naturally recurring decision rules. These can then be refined further by either the domain expert or another external expert before and after they are implemented in the final system. Protocol analysis has become popular as an elicitation tool because it forces the expert to focus on a specific task Table 5: A hypothetical concurrent protocol (Expert examining collection of ancient pottery from Greece) ‘The first thing to do is to sort the pottery into different groups, say for example, rim shards in one group, handles in another, neck shards and body pieces in yet another. Odd pieces of ceramic can be grouped by looking at whether they are painted on the inside and outside or just on the outside, the thickness, style and type of firing used. The most important pieces for the purposes of identification are the rim shards and handles. For example, often the rim shards of amphorae, which are the most commonly found ceramic ware, are stamped with the origin of the contents. So if the amphora contained wine from Rhodes, it would have a stamp that would tell us it was from Rhodes and would also allow us to speculate about the date of the amphora, and hence, the date of the deposit where it was found. Rim shards are important because they give us a good idea of what the shape of the original piece was and thus help us to date the piece. For example, a rim like this one must come from a fourth-century lekythos, while this one is obviously a second-century sigulata ware; probably from Sicily. The best situation is to have a rim shard large enough that some of the painted design, if it was painted, still can be made out. This beautiful rim here has enough left on it that I can tell it’s from a Classical ripe red figure vase; probably a large krater . . . .’
Expert Systems, May 2001, Vol. 18, No. 2
or problem without interruptions from the knowledge engineer. It forces the expert to consciously consider the problem-solving process and so may be a source of new self-understanding. It is also very flexible in that many different types of tasks (simulations, special cases etc.) may serve as a basis for the protocol. Having a record encourages the knowledge engineer to target topics and possibly develop further structured interviews around missing steps in the process. It also allows the knowledge engineer to exercise a great degree of flexibility in the choice of analysis used to structure the protocols. It has been successfully applied to developing expert systems (Hoffman, 1987) and early results of comparative experiments show that it is more efficient than unstructured interviewing (Burton et al., 1987), although the same set of experiments shows clearly that it is less efficient than other non-traditional knowledge acquisition methods such as card-sorting and goal decomposition. Also, on a practical level, protocol analysis requires little equipment or special training for the knowledge engineer. The main disadvantage of protocol analysis is the very necessity of forcing the expert to verbalize his/her actions. It is often the case that expertise has become so proceduralized that the expert is either unable to express it or is completely unaware of it. This phenomenon is more commonly referred to as the ‘paradox of expertise’ (Hoffman, 1987) and is one of the major motivations for research in the field of knowledge acquisition. Not only may experts be unaware of their problem-solving methods, but also they may actually verbalize them incorrectly and thus introduce error or bias into the resulting system. Especially when special or difficult test cases are used as cues the expert may experience considerable discomfort in trying to verbalize the problemsolving process. Thus the appropriateness of protocol analysis may depend heavily on the type of task being studied and the personality and ability of the expert to be introspective and verbalize thought processes. Protocols can also be very time-consuming to generate and may result in more data than the knowledge engineer can efficiently handle – especially true of larger knowledge acquisition tasks. While protocol analysis involves little interaction between the expert and the knowledge engineer, several elicitation techniques have been suggested which require the knowledge engineer to actively participate in the problemsolving process. These techniques capitalize on the idea that the knowledge engineer must become somewhat of an expert in order to successfully translate the expert’s knowledge into a machine representation. Thus the interview may be treated as a tutorial where the expert delivers a lecture which the knowledge engineer may paraphrase or use to solve similar problems (Johnson & Johnson, 1987; Welbank, 1987). The knowledge engineer may become even more actively involved by playing the role of an Expert Systems, May 2001, Vol. 18, No. 2
apprentice or otherwise participating in the expert’s problem-solving process (Welbank, 1987). Making the knowledge engineer become like the expert is certainly the most time-consuming approach to knowledge elicitation but ensures the highest quality resulting system. The inherent difficulties of requiring the knowledge engineer to learn all the expertise in order to translate it into a suitable machine representation are what have motivated much of the work in designing expert system shells with more naturalistic interfaces which enable the expert to enter his/her expertise directly into the system. The logical extensions of this approach are the techniques of automated or ‘guided’ self-elicitation which are discussed in a later section. 3. 4. Psychological scaling Other interviewing techniques which have been proposed in the literature have been drawn directly from psychology. These include multidimensional/psychological scaling, network scaling, discourse analysis, cluster analysis and card sorting. Many of these techniques combine elicitation and structuring aspects and thus are difficult to consider as simply ‘elicitation’ or ‘structuring’ techniques. It was hoped that techniques such as these would be more objective than more traditional interviewing methods and would be especially useful in the conceptual and refinement stages of the knowledge acquisition process (Cooke & McDonald, 1987). One empirical comparison of elicitation techniques supports this contention somewhat in that it found that non-traditional techniques such as card sorting and goal decomposition performed better than protocol analysis and interviewing (Burton et al., 1987). A number of different techniques fall under the heading of what may be called ‘psychological scaling’ techniques. These include multidimensional scaling, network scaling and hierarchical cluster analysis. Generally speaking, experts are asked to rate the similarity of different objects (usually chosen beforehand) and this rating is portrayed as a distance on a seven-point scale ranging from no similarity to completely similar. The purpose of this is to discover the expert’s rank ordering of objects within a problem domain. The most complex method within this group is probably multidimensional scaling. Described early in the literature by Kruskal (1977), it is based on the use of the least squares method to fit the elicited data. A grid of data is obtained by comparing the similarity of a set of objects (usually domain concepts) on a scaled number of different dimensions. This is supposed to give a global picture of the space in which the objects lie. The location of the objects in the different dimensions is then inferred using the least squares method. This requires that both the objects and dimensions be identified beforehand and that they should be representative of the larger problem domain without contradictions. Thus this technique should probably be considered more of a structuring tool than an elicitation tool, although it is designed 81
to elicit the expert’s knowledge regarding the relatedness among a set of objects on a set of dimensions. In this sense it can be considered to be an elicitation tool much as the analytic hierarchy process (Saaty, 1981) elicits data about objects on a hierarchy of dimensions. Even though Cullen and Bryman’s survey (1988) found this to be the least used knowledge elicitation technique, multidimensional scaling has been used extensively by researchers (Butler & Carter, 1986; Cooke & McDonald, 1987) and is well suited to use in automated knowledge acquisition programs such as AQUINAS, KITTEN, KSSO, PATHFINDER and PLANET (Boose, 1989). Network scaling is similar to multidimensional scaling in that the expert or knowledge engineer must estimate the degree of relatedness or distance between pairs of concepts. With network scaling, however, concepts are represented as nodes identified previously by the expert and the relationship between selected pairs of nodes is represented with a link. Given the expert’s estimate of relatedness between the paired nodes, the knowledge engineer uses an algorithm to determine whether a link should exist between the paired nodes and also what the strength of the relationship is (Schvaneveldt et al., 1985). Network scaling again combines elicitation and structuring processes, and would appear to have many of the advantages and disadvantages associated with semantic networks which have been used extensively in database design (Tuthill, 1990). 3. 5. Card sorting Card sorting or concept sorting techniques are also used to help structure an expert’s knowledge. As its name implies, it involves having the knowledge engineer write the names of previously identified objects, experiences and rules on cards which the expert is asked to sort into groups. The expert describes for the knowledge engineer what each group has in common and the groups can then be organized to form a hierarchy. Like multidimensional scaling, some empirical research (Burton et al., 1987) suggests that card sorting may be a more efficient elicitation technique than some of the more traditional techniques such as protocol analysis or interviewing. It has been used with some success to develop applications described in the literature (Chi et al., 1981; Gammack & Young, 1985). It has also been suggested that it is a tool which could be easily implemented on a computer as an automated knowledge acquisition tool (McGraw & Harbison-Briggs, 1989).
1989; Dhaliwal & Benbasat, 1990), the primary emphasis to date has been on developing new knowledge acquisition tools and methods. This paper focuses on examining the impact of recent empirical work. A review of the knowledge acquisition literature shows that both conceptual and empirical research has lagged behind technique-oriented research. Experiments and case studies have focused on comparing and evaluating knowledge acquisition techniques. However, the empirical work has suffered from a general lack of control and precision (Dhaliwal & Benbasat, 1990). One possible reason for this deficiency in knowledge acquisition empirical studies may be due to the fact that the needed theoretical work which should be used to support the systematic generation of hypotheses regarding the knowledge acquisition process has also lagged. Central to the development of knowledge acquisition theory is the development of a general model of possible knowledge acquisition phenomena (Holsapple & Wagner, 1996). Creation of such a model necessarily entails the development of a formal language to describe it. One category of ‘human’ or ‘manual’ knowledge acquisition techniques draws on research in psychology to design methods that are intended to help knowledge engineers conduct elicitation and structuring processes more efficiently or accurately. Examples of such methods include card sorting, protocol analysis, and the various structured interviewing techniques (Hoffman, 1987). There have been a few recent efforts to empirically test the usability of different knowledge acquisition tools and techniques. These are summarized in Table 6. Burton et al. (1987) tested the ability of various knowledge elicitation methods to elicit knowledge about classifying different rocks. And Kitto (1988) compared the relative efficiency of several automated knowledge acquisition tools. Such research is important because it serves to break new ground, but it needs to be conducted in a more systematic and rigorous manner (Holsapple et al., 1993). Previous researchers have recognized the need for sound empirical research to compare the effectiveness and efficiency of knowledge acquisition tools and methods. Fellers (1987) concluded that more research was needed to answer the following. (1) (2) (3)
Is there one best elicitation technique for knowledge acquisition? If not, what is the best combination of techniques? Which techniques are most suitable under which circumstances? What skills are required in order to utilize each of these techniques?
4. Empirical research on knowledge acquisition techniques
(4)
Work on the knowledge acquisition problem currently follows along three major interlocking lines. We describe these as technique oriented, empirical studies, and conceptual research. As has been noted in the literature (Boose,
One knowledge acquisition researcher designed an experiment to test the ability of three different knowledge acquisition methodologies to elicit different types of heuristics (Grabowski, 1988). The three methods tested were
82
Expert Systems, May 2001, Vol. 18, No. 2
Table 6: Summary of empirical knowledge acquisition research (adapted from Dhaliwal & Benbasat, 1990) Studies
Knowledge acquisition techniques
Michalski & Chilausky (1980) Hart (1985)
Moderating variables
Problem domain
Dependent variables
Interviewing Not considered Inductive learning
Diagnosis
ID3 induction and Not considered interviews
Diagnosis
Percentage of correct diagnosis generated Comparison to known cases
Messier & Hansen Interviews (1987) Protocol analysis Expert walkthroughs
Human vs Interpretation reconstructed knowledge sources
Burton et al. (1987)
Interviews Protocol analysis Goal decomposition Card sorting Multidimensional scaling
Introvert vs extrovert; cognitive styles
Classification
Holsapple & Raj (1994)
Interviewing Protocol analysis
Domain complexity
Planning
Burton et al. (1990)
Structured interviews Protocol analysis Card sorting Laddered grids
Expert vs non-expert; two classification domains
Classification
Not considered
Command and control
Knowledge engineer, and domain expert
Command and control
Grabowski (1988) Scenarios, simulated different tasks, and actual familiar tasks Adelman (1989) Top-down vs bottom-up interviewing
Expert Systems, May 2001, Vol. 18, No. 2
Results
Inductive learning performed better than interviewing Induction performed much better than interviewing Knowledge Protocol analysis engineer’s opinion had limited of the quality of usefulness for knowledge certain types of acquired knowledge Time taken to Protocol analysis capture takes longer and knowledge; yields less time to code into knowledge; rules; introverts need number of rule longer interviews clauses; but generate more completeness of knowledge than rule set extroverts Efficiency and Interviewing is quality of more efficient and knowledge as accurate for simple measured by cases but protocol number of nodes is more efficient for and arcs and their complex cases accuracy Efficiency of Protocol analysis process performed poorly in classification domain; card sorting and grids performed better than interviewing; external validation of experts important Overlap of Found 30% overlap heuristics between heuristics elicited by the three different methods Accuracy of Found no elicited rules significant variation compared to a except for that due golden mean set to domain expert
83
scenarios, simulated different tasks, and actual familiar tasks. Heuristics were divided into two categories: (a) those that all subjects identified regardless of knowledge acquisition method and (b) those that only individual subjects identified. These were further broken down as conceptual, operational, and logistical heuristics. Overall, she found a 30% overlap in the heuristics generated by each of the knowledge acquisition methods she tested. Of the heuristics that did not overlap, she identified conceptual, logistical, and operational heuristics that were distinct to each method. But given that the task studied (piloting a boat in a harbor) was operational in nature, her results were not surprising. In an actual experiment to discover the source of the greatest variation in the knowledge acquisition process, Adelman (1989) identified five determinants of knowledge base quality. As noted earlier, these are (1) (2) (3) (4) (5)
domain experts knowledge engineers knowledge representation schemes knowledge elicitation methods Problem domains.
Adelman varied the domain experts, the elicitation methods and the knowledge engineers in an attempt to see which if any had the greatest effect on the quality of the knowledge base. Five of the six knowledge engineers had PhD degrees and one was ABD (all-but-defence), but all had extensive training in both top-down and bottom-up elicitation techniques. The relative accuracy of each was compared to a ‘golden mean’ rule set derived prior to the elicitation sessions. Although a long line of psychological research has been devoted to describing interviewer effects, which are analogous to the potential effects of a knowledge engineer, no significant effects were observed in this set of experiments. Interestingly, the only significant source of variation came from the domain experts themselves. The best-known experimental research on knowledge acquisition methods is that of Burton et al. (1987). By varying the different knowledge acquisition techniques among different groups of experts, each of whom was tested for cognitive style, they discovered several specific things. Among their findings was that protocol analysis took the most time and elicited less knowledge than the other three techniques they tested (interviewing, card sorting, and goal decomposition). Not surprisingly, they also found that introverts needed longer interview sessions but generated more knowledge than extroverts. Interestingly, the rarely used techniques of goal decomposition and card sorting proved to be as efficient as the more common interviewing technique and more efficient than the commonly used protocol analysis. This experiment was criticized somewhat for its lack of rigor (Dhaliwal & Benbasat, 1990; Holsapple et al., 1993). One measure of technique efficiency was the time it took to code the transcripts into pseudo-rules, while the number 84
of rules or clauses was taken as a measure of acquired knowledge. Coding time does not fully account for temporal differences among knowledge acquisition methods and there are also serious drawbacks to using the number of coded rules as a measure (Holsapple & Whinston, 1986; Dhaliwal & Benbasat, 1990). The results may also have been confounded by unmeasured differences among the experts and the knowledge engineers. These various experimental studies are symptomatic of a recognized need for empirical investigation of knowledge acquisition phenomena. The small number of such studies is, at least in part, indicative of the difficulty in conducting them. The few pioneering studies are typified by confusing terminology, conflicting operationalizations and the proliferation of ad hoc taxonomies. In addition, results are conflicting and no clear pattern has emerged. There are problems controlling for effects of moderator variables and in operationalizing the measurement of dependent variables. In the light of these problems, Dhaliwal and Benbasat (1990) concluded that empirical knowledge acquisition work should concentrate on case studies rather than experiments, at least in the near term. A strategy for addressing some of these experimental obstacles has also been proposed (Holsapple et al., 1993). 5. Conclusions From this examination of the different knowledge acquisition techniques used in expert systems development and the results of recent empirical studies we can begin to make some more specific conclusions. First, though the problem domains studied are generally drawn from problems in the classification or command and control type, it would appear that protocol analysis does not perform as well as other more non-traditional techniques such as card sorting. The fact that four of the seven studies use tasks from analytic domains suggests that these are the most common type of expert system application domains. Being data-driven tasks, the use of inductive techniques seems more likely to perform well than interviewing techniques. Where induction cannot be used, knowledge acquisition techniques for organizing highly structured interviews, such as card sorting, seem to work better than interviewing. In either case, well-structured knowledge acquisition techniques seem to work best in analytic problem domains and protocol analysis performs poorly in all of the comparative studies. These results are less supported in the one experiment conducted in what we have described as a synthetic problem domain (Holsapple & Raj, 1994). Holsapple and Raj found that interviewing performed better than protocol analysis for simple problems whereas the reverse was true for complex problems. This suggests the possibility that, as we move into the more difficult to model synthetic domains such as design and planning, techniques such as protocol analysis may be more appropriate. It would seem that the Expert Systems, May 2001, Vol. 18, No. 2
difficulty in modeling these less structured domains might be one reason that there are relatively few comparative knowledge acquisition studies in the synthetic and combined synthetic/analytic domains. The two studies in the command and control domain do not offer much guidance as to which knowledge acquisition techniques work best. The fact that Adelman (1989) found no significant effect when he varied the knowledge acquisition technique may indicate that the choice of knowledge acquisition technique may not matter as much for problem domains that combine both analytic and synthetic aspects. 5. 1. Implications for developing P/OM expert systems The application of empirical knowledge acquisition research to the problem of choosing an appropriate knowledge acquisition technique for developing an expert system application in the P/OM field suggests several directions. First, if one is going to work in an analytic problem domain such as quality control or acceptance sampling, knowledge acquisition techniques that provide a high degree of structure to the interviewing process seem to work best. Protocol analysis, though fairly commonly used, is relatively inefficient for analytic problems, while the most popular technique of using unstructured interviewing is one of the least efficient and least satisfying from the standpoint of the expert. So it may be worth exploring some of the non-traditional knowledge acquisition techniques when working on these types of applications. For the more difficult synthetic and combination problem domains the evidence is not as clear. However, the Holsapple and Raj study (1994) seems to indicate that problem complexity may be one determinant of the appropriate knowledge acquisition technique to choose. So if we were to develop a highly robust expert system for project management in P/OM then we might suppose that protocol analysis might be more efficient than interviewing. The fact that interviewing is more efficient for simple domains may imply that it is best used for initial knowledge acquisition sessions, when the problem complexity is not yet developed clearly. For those studies that did consider the effect of moderator variables, it seems clear that, no matter what type of problem domain, developers of expert systems in the field of P/OM should consider their potential impact. The impact of the cognitive style of the expert, domain complexity, along with other attributes of the domain expert all seem to be important factors in the quality of an expert system regardless of the problem domain. It is hoped that further research will clarify some of these issues with respect to the effect of moderator variables and problem domains. References Adelman, L. (1989) Measurement issues in knowledge engineering, IEEE Transactions on Systems, Man and Cybernetics, 19, 483–488. Expert Systems, May 2001, Vol. 18, No. 2
Allen, M.K. and O.K. Helfreich (1990) Putting Expert Systems to Work in Logistics, Oak Brook, IL: Council of Logistics Management. Belkin, N.J., H.M. Brooks and P.J. Daniels (1987) Knowledge elicitation using discourse analysis, International Journal of Man–Machine Studies, 27, 127–144. Blanchard, D. (1994) Installing equipment with virtual reality, OR/MS Today, October, p. 8. Bonnisone, P.P. and H.E. Johnson (1983) Expert Systems for Diesel Electric Locomotive Repair: Knowledge-based System Report, New York, NY: General Electric Company. Boose, J. (1989) A survey of knowledge acquisition techniques and tools, Knowledge Acquisition, 1 (1), 3–38. Burton, A.M., N.R. Shadbolt, A.P. Hedgecock and G. Rugg (1987) A formal evaluation of knowledge elicitation techniques for expert systems: domain 1, in Research and Development in Expert Systems IV, D.S. Moralee (ed.), Cambridge: Cambridge University Press. Burton, A.M., R. Schweickert, N.K. Taylor, E.N. Corlet, N.R. Shadbolt and A.P. Hedgecock (1990) Comparing knowledge elicitation techniques: a case study, Artificial Intelligence, 1 (4), 245–254. Butler, K. and J. Carter (1986) The use of psychologic tools for knowledge acquisition: a case study, in Artificial Intelligence and Statistics, W.A. Gale (ed.), Menlo Park, CA: Addison-Wesley, 295–320. Chi, M., P. Feltovich and R. Glaser (1981) Categorization and representation of physics problems by experts and novices, Cognitive Science, 5, 121–152. Clancy, W.J. (1986) Heuristic classification, Artificial Intelligence, 27, 298–350. Cooke, N. and J. McDonald (1987) The applications of psychological scaling techniques to knowledge elicitation for knowledgebased systems, International Journal of Man–Machine Studies, 26, 81–92. Cullen, J. and A. Bryman (1988) The knowledge acquisition bottleneck: time for a reassessment?, Expert Systems, 5 (3), 216–224. Descotte, T. and J.C. Latombe (1981) GARI: a problem solver that plans how to machine mechanical parts, Proceedings of the 7th International Joint Convention on Artificial Intelligence (IJCAI), Vancouver, Canada, pp. 766–772. Dhaliwal, J.S. and I. Benbasat (1990) A framework for the comparative evaluation of knowledge acquisition tools and techniques, Knowledge Acquisition, 2 (2), 145–166. Elliot, L. (1986) Analogical problem solving and expert systems, IEEE Expert, 1 (2), 17–28. Eom, S.B. (1996) A survey of operational expert systems in business (1980–1993), Interfaces, 26 (5), 50–70. Fellers, J.W. (1987) Key factors in knowledge acquisition, Computer Personnel, 11 (1), 10–24. Fisher, E.L. and S.Y. Nof (1984) FADES: knowledge-based facility design, Annual International Industrial Engineering Conference Proceedings, Chicago, IL, pp. 74–82. Forsythe, D. and J. Buchanon (1989) Knowledge engineer as anthropologist, IEEE Transactions on Systems, Man and Cybernetics, 3, 472–482. Fox, S. and D. Smith (1984) ISIS: a knowledge-based system for factory scheduling, Expert Systems, 1 (1), 25–49. Gammack, J.G. and R. Young (1985) Psychological techniques for eliciting expert knowledge, in Research and Development in Expert Systems, M.A. Bramer (ed.), London: Cambridge University Press, 105–112. Grabowski, M. (1988) Knowledge acquisition methodologies: survey and empirical assessment, Proceedings of ICIS, Minneapolis, MN. 85
Hart, A. (1985) Experience in the use of an inductive system in knowledge engineering, in Research and Development in Expert Systems, M.A. Bramer (ed.), London: Cambridge University Press, 117–126. Hoffman, R. (1987) The problem of extracting the knowledge of experts from the perspective of experimental psychology, AI Magazine, 8 (2), 53–67. Holsapple, C. and V. Raj (1994) An exploratory study of two KA methods, Expert Systems, 11 (2), 77–87. Holsapple, C. and W.P. Wagner (1996) Process factors of knowledge acquisition, Expert Systems, 13 (1), 55–62. Holsapple, C. and A. Whinston (1986) Manager’s Guide to Expert Systems, Homewood, IL: Dow Jones-Irwin. Holsapple, C., V. Raj and W. Wagner (1993) Knowledge acquisition: recent theoretic and empirical developments, in Recent Developments in Decision Support Systems, C. Holsapple and A. Whinston (eds), Berlin: Springer. Johnson, L. and N.E. Johnson (1987) Knowledge elicitation involving teachback interviewing, in Knowledge Elicitation for Expert Systems: A Practical Handbook, A. Kidd (ed.), New York: Plenum. Kangskool, K., J. Goldman and M. Leonard (1987) Automating the design of work systems, Proceedings of the IIE Integrated Systems Conference, Atlanta, GA: IIE Press. Kim, J. and J. Courtney (1988) A survey of knowledge acquisition techniques and their relevance to managerial problem domains, Decision Support Systems, 4 (3), 269–284. Kitto, C. (1988) Process in automated knowledge acquisition tools: how close are we to replacing the knowledge engineer?, in Proceedings of the Third Knowledge Acquisition for KnowledgeBased Systems Workshop, Banff, Canada, pp. 14.1–13. Kruskal, J. (1977) Multidimensional scaling and other methods for discovering structure, in Statistical Methods for Digital Computers, K. Enslein, A. Ralston and H.S. Wilf (eds), New York: Wiley. Logan, D. and J. Keyon (1992) Help desk: using AI to improve customer service, in Innovative Applications of Artificial Intelligence 4, A.C. Scott and P. Klahr (eds), San Jose, CA: AAAI Press/The MIT Press. McClay, W.J. and J.A. Thompson (1989) Harnessing detailed assembly process knowledge with case, in Innovative Applications of Artificial Intelligence, H. Schorr and A. Rappaport (eds), AAAI Press/The MIT Press. McDermott, J. (1982) A rule based configurer of computer systems, Artificial Intelligence, 19, 39–88. McGraw, K.L. and K. Harbison-Briggs (1989) Knowledge Acquisition: Principles and Guidelines, Englewood Cliffs, NJ: Prentice-Hall. Merton, R.K., M. Fiske and P. Kendall (1956) The Focused Interview, Glencoe, IL: Free Press. Messier, W. and J. Hansen (1987). A case study and field evaluation of EDP-XPERT, Working Paper. Michalski, R.S. and R.L. Chilausky (1980) Knowledge acquisition by encoding expert rules versus computer induction from examples – a case study involving soybean pathology, International Journal of Man–Machine Studies, 12, 63–87. Miller, R. (1986) Artificial Intelligence Applications for Manufacturing, Madison, GA: SEAI Technical Publications.
86
Mortons, T.E., S.R. Lawrence, and G.L. Thompson (1986) MRP-Star, Patriarch’s planning module, Unpublished manuscript. Newell, A. and H. Simon (1972) Human Problem Solving, Englewood Cliffs, NJ: Prentice-Hall. Osborn, A. (1953) Applied Imagination: Principles and Procedures of Creative Thinking, New York: Scribner’s. Pollitzer, E. and J. Jenkins (1985) Expert knowledge, expert systems and commercial interests, Omega, 13 (5), 407–418. Prerau, D.S., A.S. Gunderson, R.E. Reinke and S.K. Goyal (1985) The COMPASS expert system: verification, technology transfer and expansion, Proceedings of the 2nd Conference on Artificial Intelligence Applications: The Engineering of Knowledge-based Systems, Washington, DC: IEEE Computer Society, 597–602. Saaty, T.L. (1981) The Analytical Hierarchy Process, New York: McGraw-Hill. Schvaneveldt, R., F. Durso, T. Breen, N. Cooke, R. Tucker and J. DeMaio (1985) Measuring the structure of expertise, International Journal of Man–Machine Studies, 23, 699–728. Stroebel, G., R. Baxter and M. Denney (1986) A capacity planning expert system for IBM System 38, IEEE Computer, 19 (7), 42–52. Tuthill, G.S. (1990). Knowledge Engineering: Concepts and Practices for Knowledge-Based Systems, Blue Ridge, PA: TAB Books. Waterman, D.A. (1986) A Guide to Expert Systems, Reading, MA: Addison-Wesley. Weiss, S. and C. Kulikowski (1984) A Practical Guide to Designing Expert Systems, Totowa, NJ: Rowman and Allanheld. Welbank, M. (1987) Knowledge acquisition: a survey and British Telecom experience, Proceedings of the First European Workshop on Knowledge Acquisition for Knowledge-based Systems, Reading University, pp. c6.1–9.
The authors William P. Wagner Professor Wagner received his PhD in information systems from the University of Kentucky. He has been teaching and conducting research in information systems at Villanova University since 1991. His teaching and research interests span the areas of expert systems, ERP systems and e-commerce and have appeared in internationally known journals such as Expert Systems, Journal of Computer Information Systems and Information Management Journal. He has presented numerous research papers at regional and international conferences such as Decision Science Institute, INFORMS, AIS and the NATO Advanced Studies Institute.
Expert Systems, May 2001, Vol. 18, No. 2
Mohammad K. Najdawi
Q.B. Chung
Mohammad K. Najdawi is the Associate Dean and Professor of Operations and Information Management in the College of Commerce and Finance at Villanova University. He received his BSc from Slovak University, MSc from the London School of Economics and PhD from the Wharton School of the University of Pennsylvania. He has taught classes in operations management, supply chain management, decision processes and MIS. His publications have appeared in, among others, Management Science, European Journal of OR, International Journal of Production Research, Journal of Business Logistics, International Journal of Production Economics, Communications of the ACM and End User Computing. He served as Associate Editor for Communications of ACM and International Journal of Operations and Quantitative Management. Dr Najdawi worked as a consultant to ARAMCO, UNDP and Georgia Pacific. He is an active member of INFORMS and DSI.
Q.B. Chung is an Assistant Professor of management information systems at Villanova University. He earned his PhD in management from Rensselaer Polytechnic Institute. His research focus is on enhancing the quality of managerial problem formulation through knowledgebased modeling environments. His papers have appeared in academic journals including Expert Systems with Applications, Intelligent Systems in Accounting, Finance and Management, Journal of Knowledge Management, European Journal of Information Systems, Information and Management and Omega.
Expert Systems, May 2001, Vol. 18, No. 2
87
