CHALLENGES AND METHODS IN THE QUANTIFICATION OF ...

Proceedings of IMECE ‘04 2004 ASME International Mechanical Engineering Congress November 13-19, 2004, Anaheim, California

IMECE2004-60501 CHALLENGES AND METHODS IN THE QUANTIFICATION OF DESIGN ERRORS AND SOLUTION ELEMENTS Lawrence P. Chao Department of Mechanical Engineering Design Division Stanford University Stanford, California, 94305-4022, USA [email protected]

Kosuke Ishii Department of Mechanical Engineering Design Division Stanford University Stanford, California, 94305-4022, USA [email protected]

ABSTRACT To error-proof the design process, tools such as Design Process Failure Modes and Effects Analysis and Project Quality Function Deployment mitigate risk through thorough understanding of the consequences of both the process-level errors that can occur and the solution elements that mitigate them. However, the quantification of design errors and prioritization of other elements are complicated by the temporal and spatial distance of the decisions from the end-result. This paper discusses measures for design elements in the context of process-based analysis, including the design errors, tasks, and project resources. The Risk Priority Number is the standard measure of criticality of failure modes and their effects. However, alternatives to the traditional RPN have emerged in forms such as expected and life-cycle cost as well as QFDbased techniques. The paper explores the benefits and challenges of these traditional and new measures and concludes with a discussion into converting between the measures.

Table 1. Product NASA Mars Climate Orbiter Ford Explorer / Firestone Tires

Intel Pentium Hyatt Regency Kansas City

LOSSES FROM HISTORICAL DESIGN FAILURES Date Costs 1999 $134.5 million for launch and 19912000 1994 1981

AA Flight 191 DC-10 1979 Kemper Arena

1979

operating, $125 million orbiter 6.5 million recalled tires; lawsuits including a $7.5 million injury settlement and $41.5 million settlement with states ~6 million chips sold in 1994 $140 million in judgments and settlements, $3 billion in lawsuits filed; more than 90% of claims were settled $100 million in lawsuits estimated, worst aviation disaster at the time $23.2 million for construction

Hartford Civic Center 1978 Stadium capacity of 10,000 seats unavailable for two years Grand Teton Dam 1976 $4,575,000 in design and construction, $1 billion in property damage, 100,000 acres of farmland and 16,000 herd of livestock lost Ford Pinto 1971- Lawsuits including a $128 million 1976 settlement; 1.5 million recalled cars in 1978 Chevrolet Corvair 1960- Lawsuits including a $2.4 million 1963 settlement; production ended in 1969; 235,000 sales in 1965 to only 6,000 in 1969 de Havilland Comet 1953- 21 planes built between 1952-1954, 1954 7 crashed; 2 with passengers

KEYWORDS: design, error, solutions, quantify, RPN, expected cost, FMEA, QFD, criticality, risk

1. INTRODUCTION The goal of design process error-proofing is to develop methods and strategies to predict and prevent errors in product development. Already tools like Design Process FMEA and Project QFD help identify both errors in and solution elements for design. Our research has engaged organizations in various industries, gathering data on design errors and methods. In addition, historical design error benchmarks have been an insightful source on the impact of design errors and failures. Table 1 lists monetary and human losses of several famous design errors.

Tacoma Narrows Bridge H.M.S. Titanic

Dead 0 >203

? 114

273 0 0 11

>500 ?

>56

1940 $6.6 million for construction

0

1912 $7.5 million for construction

1491

These benchmarked cases illustrate that simple mistakes can result in huge losses, in both dollars and lives lost. The actions necessary to prevent such catastrophic failures would only have cost a portion of the loss.

1

Copyright ©2004 by ASME

Challenges and Methods in the Quantification of Design Errors and Solution Elements Chao and Ishii Table 3.

Having measures of the criticality of design errors and benefits of solution elements assists the scoping of problems and prioritization when time and resources are limited. Several measures are already common in measuring risk including the traditional Risk Priority Number and expected cost for FMEA. Quantification and prioritization is an important step in making decisions to prevent future design errors.

Effect Hazardous without warning Hazardous with warning very high high moderate low very low minor very minor none

Table 4.

2.1.1. Occurrence

2.2. Expected cost Many risk management methods, including decision analysis and options valuation techniques, apply statistical decision theory and systems analysis towards a metric like expected cost. Rather than using ordinal scales to quantify criticality, expected cost quantifies risks of failure modes on an absolute and “meaningful” scale. By tying risk to cost, engineers can minimize life-cycle cost by comparing failure costs to solution costs for better reliability, improved serviceability, and better diagnosis and future detection. Expected cost uses probability instead of occurrence and cost instead of severity, with detection ignored, as the components of risk.

AIAG OCCURRENCE RANKINGS FOR FMEA

High: Repeated failures Moderate: Occasional failures

Low: Relatively few failures Remote: Failure is unlikely

AIAG DETECTION RANKINGS FOR FMEA

Detection Difficulty Ranking Can be detected as error occurs or immediately following error, no permanent damage 1 - 2 Can be detected during assembly: early or with high certainty 3-4 Can be detected during assembly: late or with low certainty 5-6 Can be detected during service or routine maintenance 7-8 Very unlikely to be detected during ownership 9 - 10

Occurrence is the likelihood that an action will lead to the given result. If statistical data is present the column could be based on that data. If the rates are accurate, this information can be used to predict overall failure rates. The Automotive Industry Action Group (1996) suggests the following Occurrence rankings, listed in Table 2. An alternative is to use a four level (0-1-3-9) ranking. By reducing the number of levels the time to fill out the FMEA is reduced. However, the resolution of the results is also reduced.

Possible Failure Rates 1 in 2 1in 3 1 in 8 1 in 20 1 in 80 1 in 400 1 in 2000 1 in 15,000 1 in 150,000 1 in 1,500,000

Ranking 10 9 8 7 6 5 4 3 2 1

The Detection Method / Current Controls column discuss the current methods for identifying the failure modes. The Detection column is used when trying to reduce total risk by rating how difficult it is to detect the error after it occurs. If a failure mode has high risk because of poor detectability, the Detection column allows for a quick review of current methods. Table 4 lists general guidelines for detection difficulty.

2.1. Risk Priority Number The Risk Priority Number (RPN) used in Failure Modes and Effects Analysis, or FMEA, (Stamatis 1995) is the product of the row of scaling factors. Each potential failure mode and result will have an RPN. The Occurrence, Severity, and Detection metrics give an indication as to what is causing a particular error mode to have a high risk. A Pareto analysis of the RPN’s from the FMEA can be used to identify areas where errors are likely to be most significant. This can yield information about what types of failure modes are most trouble prone. Once the highest risk drivers have been identified, steps can be taken to reduce the risk associated with the error mode.

Probability of Failure Very High: Failure is almost inevitable

Severity of Effect when a failure mode affects safe device operation without warning when a failure mode affects safe device operation with warning device inoperable: loss of primary function device operable: at a highly reduced level of performance device operable: at a reduced level of performance device operable: at a slightly reduced level of performance device operable: defect noticed by most customers device operable: defect noticed by average customer device operable: defect noticed by discriminating customers almost no effect

2.1.3. Detection

2. TRADITIONAL MEASURES OF ERROR

Table 2.

AIAG SEVERITY RANKINGS FOR FMEA

Ranking 10 9 8 7 6 5 4 3 2 1

2.2.1. Probability

In considering the probabilities associated with design errors, one can look at the likelihood that a failure mode occurs or is caught and roll it back or at the likelihood of an errorinducing action occurring during development. Probabilities can be calculated based on observed and empirical data or on the theoretical complexity of the type of task. Williams (1986) developed a data-based method of estimating human error probabilities based on the complexity of the task and a user’s familiarity with the given task which can be multiplied by certain error-producing conditions. Detailed in Tables 5 and 6, the Human Error Assessment and

2.1.2. Severity

The Severity column rates the gravity of the end effect in the previous column. Table 3 lists the AIAG guideline for severity. The severity is based on the assumption that the error has occurred.

2


Challenges and Methods in the Quantification of Design Errors and Solution Elements Chao and Ishii Reduction Technique (HEART) is an error management tool based on studies of safety in the nuclear industry.

Table 5.

Analysis, Work Safety Analysis, and Action Error Analysis (Leveson 1995). Most of the data are based on operational task analysis and task models rather than on cognitive models. Studies like WASH-1400 (1975) and NUREG/CR-1278 (1980), listed in Table 7, found failure rates of human operators.

NOMINAL ERROR PROBABILITIES IN HEART (WILLIAMS 1986)

Generic Tasks

Error Probabilities Nominal 5th %tile

Totally unfamiliar, performed at speed with no idea of likely consequence Shift or restore system to a new or original state on a single attempt without supervision or procedures Complex task requiring high level of comprehension and skill Fairly simple task performed rapidly or given scant attention Routine, highly practiced, rapid task involving relatively low level of skill Restore or shift system to original or new state following procedures, with some checking Completely familiar, well designed highly practiced routine task, oft-repeated and performed by well motivated, highly trained individuals with time to correct failures but without significant job aids Respond correctly to system even when there is an augmented or automated supervisory system providing accurate interpretation of system state Miscellaneous task for which no description can be found

0.55

Table 7.

95th %tile

0.35

0.97

Activity

Error Rate 3 × 10 –2 3 × 10 –1 10 0.2 – 0.3 0.1 – 0.9 (0.5 avg.) –4 –3 10 – 5×10 –3 (1×10 avg.) –3 –2 5×10 – 5×10 –2 (5×10 avg.) –3 –2 10 – 10 –3 (3×10 avg.)

0.26

0.14

0.42

0.16

0.12

0.28

Error of omission/item embedded in procedure Simple arithmetic error with self-checking Inspector oversight of operator error General rate / high stress/dangerous activity Checkoff provision improperly used

0.09

0.06

0.13

Error of omission / 10-Item check list

0.02

0.007

0.045

Carry out plant policy / no check on operator

0.003

0.0008

0.007

Select wrong control / group of identical, labeled, controls

0.0004 0.00008

0.009

0.00002

6E-06

0.00009

0.03

0.008

0.11

–3

These models seem to lend themselves well to estimating probabilities for design errors. The only question is how relevant these models are for design processes and tasks. They derive from operational errors and mistakes, rather than design or conceptual errors. 2.2.2. Cost

Monetary cost is a good measure of error for many reasons, the primary being that the bottom line for most companies is measured in dollars and cents. It is a very tangible and transparent measure which can be easily understood. Figure 1 shows the costs associated with product quality.

In addition to the nominal probabilities, error-producing conditions may cause the operator to violate the condition, resulting in a probability increase by a multiplicative factor.

Table 6.

ACTIVITY AND ERROR RATE (WASH-1400 AND NUREG/CR-1278)

HEART ERROR PRODUCING CONDITIONS (WILLIAMS 1986)

Error Producing Condition Unfamiliarity with a situation that is potential important, but which is either novel or occurs only infrequently Shortage of time for error detection and correction Operator inexperience (for example, a newly qualified technician) An impoverished quality of information conveyed by procedures and person-to-person interaction Little or no independent checking of testing of output A conflict between immediate and long-term objectives No diversity of information input for veracity checks A mismatch between the educational achievement level of an individual and the requirements of the task An incentive to use other, more dangerous procedures

Failure cost

Factor 17x

Total quality cost

COST

11x 3x Cost of appraisal, prevention, and control

3x 3x 2.5x 2.5x 2x

PRODUCT QUALITY

Figure 1.

PRODUCT QUALITY VERSUS COST (HOLLOCKER 1986)

2x

The most obvious quality costs are the failure costs. External failure costs include customer support, product introduction, reporting, and change initiation costs. Internal failure costs include problem reporting, test environment,

Other methods for human error probability have been suggested, like Procedure (Task) Analysis, Operator Task

3


Challenges and Methods in the Quantification of Design Errors and Solution Elements Chao and Ishii corrective action, debugging and rework costs. The difference between the two is in when the defects are discovered: either before or after shipment of the product to the customer. The other costs in product quality are associated not with failures directly but the maintenance of them. Appraisal costs include the testing efforts, simulations, reviews, inspections, and audits during product development for verification and validation. Prevention costs are expended to minimize appraisal and failure costs. They can include training, application, or support costs for analysis and methodologies, assurance programs, and support (Hollocker 1986).

necessarily twice as likely. Mock and Grove (1979) argue it is valid to rank failures along a single ordinal dimension (e.g., Severity) but multiplying ordinal scales is not an “admissible transformation.” 2.3.2. Expected Cost

The difficulty with most decision analysis and options methods is the significant effort in executing the analysis. It typically requires many man-months of effort and also requires trained facilitators. The tendency is for the assessment to become unwieldy as the number of uncertainties becomes large. Decision analysts often reduce the problem to seven assessed uncertainties. Design parameters, including organizational and market factors, can easily number in the hundreds. There is also difficulty recognizing the influence of product development decision and activities on key variables. Engineers and managers are often not always aware of all the factors that will determine whether the product will offer high value to the customer and the company. Using such analysis without familiarity with all the factors can lead to decisions on inappropriate models (Sarbacker 1998). Identifying the cost and probability can be difficult. One solution is to estimate probability and cost corresponding to the estimated 10, 50, and 90th percentiles (low, nominal, high). Accurate information is not always available, and the use of the numbers can create a false sense of certainty. The predicted risk could be much different if even one factor is overlooked.

2.3. Difficulties with these measures The problem with quantifying the criticality of a design process error is the difficulty it foreseeing comprehensively and accurately the ways one mistake can impact the rest of the system. Because of the sheer number of possibilities, it is difficult to judge severity, occurrence, and detection of a simple design mistake, such as forgetting to change units. Design errors are often latent errors (Reason 1990) whose effects can be hidden initially, even if they can drastically alter the system to deviate it dramatically from its intended function. The great spatial and temporal distance of the original error from its end consequences is common with these latent design errors. A simple error can have drastic consequences. The Mars Climate Orbiter showed that a simple unit error can result in the loss of a hundred-million-dollar NASA craft. However, depending on which sets of units were not converted properly, the actual resultant failure could have varied greatly. Some may be easier to catch than others.

2.3.2.1 Cost

There are several issues with using cost for errors. The major difficulty is obtaining this sensitive data. After extensive contact with industry, even generic cost data has proven to be difficult to obtain. There are many reasons for this. Confidentiality and proprietary issues are a major hurdle. Experience has shown that even sharing numbers within the same organization is not a given. Each group has its own concerns about sharing information due to the legal and ethical ramifications. While models on human error probabilities such as HEART exist, systematic models on costs and losses from design errors are lacking. Tools in industrial engineering can estimate rework cost as well as survey reputation cost, but an engineer still cannot easily comprehend the eventual end effects of latent errors early in the life-cycle, much less measure them. Because design process errors are at the beginning, or at least early, in the cause-effect chain, multiple failures can result from a single design error, and vice versa.

2.3.1. RPN

The Risk Priority Number is the output of most FMEA’s. Kmenta et al (1999) have argued that the RPN used to evaluate risk in FMEA is not sufficient for making cost-driven decisions and, at best, provides a qualitative assessment of risk. The components of the RPN have inconsistent definitions, resulting in questionable risk priorities. The ratings for Severity are not related to any standard measure. The values can only provide a relative ranking of “consequences” or “hazard” for a particular FMEA. Detection is not related directly to probability or any other standard measure. In addition, Detection has several disparate definitions. It is unclear if different detection points should be different failure modes with different detection values. Scenario-based FMEA considers different detection points as different scenarios, rather than quantifying the ease of detection. Given that design process errors are latent and may or may not be caught anywhere through the design process, a scenario-based approach might be desirable. Severity, Occurrence, and Detection are ordinal scales used to rank-order items. The magnitude of the RPN is used to prioritize failure risks. However, ordinal measures preserve order but their magnitudes are not “meaningful.” For example, an Occurrence rating of 8 is more likely than 4 but it is not

Table 8.

COST OF DETECTING AN ERROR ALONG THE LIFE-CYCLE (DEWAR 2003)

At the vendor At the customer’s receiving inspection On the customer’s assembly line At the customer’s outgoing inspection On the customer’s end product in service

4

$1 $10 $100 $1,000 $10,000


Challenges and Methods in the Quantification of Design Errors and Solution Elements Chao and Ishii • The cost of an error is also dependent on when the failure is measured or the error is discovered. For example in Table 8, the lost costs of detecting an error can vary by many folds depending on when it is considered. For these reasons, scenario-based analysis of errors has advantages in consistency. Sometimes, there are no clear objective measures in quantification of certain undesirable consequences. While certain failure modes can be quantified more easily, by using numbers such as “rework” or “repair” costs, not everything is reversible. In addition, rework has become accepted as a natural part of the design process and some times is not even considered, despite the additional resources that are necessary. The most dramatic example of the challenge in bringing in monetary units into error cost comes with errors which impact safety. The phrase “cost of human life” has been used ad infinitum. Nonetheless, it is very difficult to pinpoint a universal number which is agreeable. There are many ethical and public relations issues with assigning a number to life. One of the most infamous examples was Ford’s costbenefit analysis of the Pinto around 1970. The analysis that Ford conducted estimated that it was not cost-efficient to add a $11 per car cost in order to correct a flaw (totaling $137 million) in comparison to the estimated $49.5 million cost of deaths ($200,000 each), burn injuries ($67,000 each), and repairs ($700 each). Ford underestimated the number of injures and neglected to include the cost of fire department calls, $350 million a year in the 1960’s, in its cost-benefit analysis. This example illustrates that it is difficult to capture all the costs associated with such events. In addition, the public relations impact when this analysis was revealed was perhaps greater than the observed losses, as well as harder to quantify [1]. Often, lives are valued on criteria like expected earnings. This can implicitly translate into a cheaper value of life in poorer neighborhoods or less-developed nations. Although nearly 3000 people died in the 1984 chemical leak in Bhopal, India, the settlement was less than half that of the Exxon Valdez oil spill. In fact, the low valuation of life was made explicit in the legal arguments of Union Carbide regarding the extent of the company’s liability and settlement (Herkert 1994). The company paid a compensation of $470 million to a trust in 1989. Claimants who received payments have been paid $600 for injuries and less than $3,000 for deaths [6]. After the September 11th attacks on the United States, the U.S. Government tried to move quickly to compensate the families of the victims of the attacks. The Victim Compensation Fund [2] is a government program to compensate the surviving victims and families of victims of 9/11 based on rules including income and family circumstances. For example, these are estimates of how much the master of the fund, Kenneth Feinberg, was prepared to pay the families of victims who died: •

• •

A 55-year-old janitor survived by a wife and two children: $540,000. A 26-year-old unmarried, childless apprentice electrician: $670,000. A widowed 67-year-old coffee-shop waitress with two adult children: $500,000.

All get $250,000 plus $50,000 per dependent for pain and suffering as part of the overall package outlined [5]. As of March 2004, the compensation fund has so far paid out 1,800 awards averaging about $1.4 million each. Feinberg has estimated that $5 billion eventually would be paid from the fund [7]. The largest death award was $6 million, while checks for the injured range from $500 to $6.8 million, awarded to a man burned over 85 percent of his body in the Pentagon attack. Though the fund is not an attempt to place “a value on human life,” as soon as Feinberg took the position, grief-stricken families complained that his formula kept awards too low [4]. There are other issues which may also take place in which the effects are not even completely known. Having a faulty product, especially one that say costs human-life, can seriously damage a company’s reputation. It is difficult to quantify consistently this reputation cost. There are metrics like consumer confidence surveys and stock prices perhaps, but there are many other factors which impact those values as well. 2.3.2.2 Probability

It can be very difficult to accurately predict probabilities. For the Pinto example, a major reason the predicted cost was undervalued was that Ford underestimated the injury to death ratio to be 1:1. According to Clemens (2002), probabilities can be misleading, depending on the context. Societal expectations can vary from being a Hall of Fame baseball player who has a .333 batting average which really is a 67% error rate to a transit worker who can be fired after a single error. In addition, mismatches are present between analytical predictions and experience. For example, from WASH-1400, the human error probability for a routine repetitive task is about 0.003 to 0.01 per individual operation. For a concert pianist who plays Mozart’s 17th Piano Concerto, K.453, 1st movement, which has 3996 individual keystrokes, this predicts from 12 to 40 errors. Obviously, concert pianists do not make this many mistakes. In NUREG 1150, the Nuclear Regulatory Commission (NRC) included a semi-formal process of eliciting judgments about key events and parameters from contractors and in-house experts. The purpose of NUREG 1150 was to estimate uncertainties and consequences of severe core damage accidents in nuclear power plants. The insights from the expert elicitation process in the study showed biases could unknowingly enter judgments, and many experts were surprised by their own lack of appreciation for broad uncertainty ranges. There was agreement that averaging was an appropriate procedure for providing information for a base case. The main problem encountered was poor definition of the

A 30-year-old stockbroker survived by a wife and two children: $4.3 million.

5


Challenges and Methods in the Quantification of Design Errors and Solution Elements Chao and Ishii events and uncertain quantities (Keeney and von Winterfeldt 1991). An additional difficulty is that stress and other error producing conditions change the probability of failure, shown in Figure 2. Though HEART does try to capture this to a certain degree, stresses are difficult to model and predict.

INFLUENCE OF STRESS ON FAILURE PROBABILITY (CLEMENS 2002)

Though one can use decision and probabilistic risk analytic techniques to build trees creating cause-effect chains, when there are multiple causes it is difficult to separate and quantify the contribution of each. Just as the design process is dynamic and hard to predict, the operation of a product can be even worse. To estimate the probabilities of each tree would result in multiplying “best guess” after “best guess.” Though expected cost feels more meaningful than RPN, the product of inaccurate or imprecise numbers is, in reality, no better.

3.2. Design error quantification Design process errors occur early in the product life-cycle, and as such, early in the cause-effect chain of failure modes. Therefore, the end effects and severity resultant from these errors can be hard to gage. Even when the scope of severity is limited, it is hard to predict all the ways a design error will impact a system. There are challenges in quantification which must start with definition, detection, and scoping of the errors. What is worse - a mistake early in the design process or late in the design process? With errors made earlier in the process, there is more opportunity to catch the errors, the root cause is easier to find, but it can “infect” more of the process, result in more delays and cost, as well as be harder to predict the effects. Errors made later are closer to tangible results and therefore easier to visualize and detect but are more separated from the initial requirements and specifications.

3. QFD METHODS TO QUANTIFY DESIGN ERRORS 3.1. Background The main applications considered here in quantifying design errors and other design elements are towards errorproofing the product development process. Because of this context, some of the methods and motivations for quantifying and prioritizing these elements are different than for an analysis end resultant failure modes. Errors are traditionally prioritized by quantifying and ranking the criticality of what might occur, either by a traditional RPN or an expected cost approach, to name a few. The difficulty with either, however, comes from consistently quantifying the severity or cost of a particular design error. The goal here is to explore process-level methods which are both meaningful to use and practical to complete for design engineers. Quality Function Deployment (QFD) is one tool which can provide a framework for design and allow decisions to prioritize either tasks, errors, or other design process components. QFD is a structured approach for translating customer requirements into design specifications. It is a

Customer Requirements

Engineering Metrics

Figure 3.

6

I

Design Errors Engineering Metrics

Figure 2.

powerful tool that ensures proper communication between the client and design team (Clausing 1994). QFD plays a central role in the design process and can use information compiled from Value Graph and Customer Value Chain Analysis. Many other design tools can be based on the QFD results, like Failure Modes and Effects Analysis and Cost-Worth Analysis (CWA). Design Process FMEA (Chao and Ishii 2003b) uses a QFD-based measure in a modified Risk Priority Number called the “Error Priority Number” (EPN) to triage design errors and tasks, rather than the failure modes and end effects. Instead of multiplying the ordinal scales of severity, occurrence, and detection, the three columns are modified. Instead of severity, an “importance” score is used, normalizing the relative weights from a QFD to the same 0 (low) to 9 (high) rating scale used in a traditional RPN component. Occurrence and detection are redefined to consider relative likelihoods on the scale to 10. Rough, relative approximations can still be made of high-medium-low using 9-3-1. Occurrence is now the likelihood that this design error happens in a task, and detection is now the likelihood of not immediately discovering the error, not the resultant failure mode end effect, in the task. An alternative is to use importance scores calculated based on the task fulfillment of customer requirements. In this case, all errors committed in a particular task are given the same priority. The following sections discuss variations, including quantifying the error versus task, RPN versus expected cost, and local effects rather than entire scenarios.

II

QFD TO VALUE THE “SEVERITY” OF ERRORS


Challenges and Methods in the Quantification of Design Errors and Solution Elements Chao and Ishii Engineering Metrics

n

ECI w =

∑w ⋅ c =1

Figure 4.

smax n

II

QFD TO VALUE THE “IMPORTANCE” OF TASKS

By using QFD as a framework, the important parameters are identified ahead of the time, and there is a simple scale to correlate the relationships between the process tasks and the customer values. The QFD-derived importance scores can be normalized to a 0-9 scale and be aligned the scale used in traditional QFD/FMEA. Through this QFD analysis, errors can be considered failures of the task and associated with the value of the task. The advantage of quantifying design tasks is that there are usually less tasks than errors and that this prioritization can also be used to scope the FMEA analysis. However, it might be necessary to expand the analysis of engineering metrics to include subsystem metrics that apply to subcomponents rather than the product as a whole. Chao and Ishii (2003a) performed a sensitivity analysis on the effects of including part characteristics to relate design tasks to customer requirements and engineering metrics. The average discrepancy between the parts and engineering metrics scores was only about 5%. As discussed, the cause-effect chain for design tasks and errors can be extremely long and have many branches, especially if one looks at the chain beyond the development process and further into the life-cycle at manufacturing, production, operation, service, and so on. Design Structure Matrix (DSM), Gantt charts, and other project management tools can model the product development process to aid schedule optimization. Though Design Task QFD looks at the direct relationship of tasks with customer requirements, an advanced analysis to consider would be to explore the relationships of tasks with each other. One way to confine the analysis is to see how an error in one task propagates through to the rest of the development process. The effects of a task i on the other tasks can be found through the DSM in conjunction with the results of the House II. In addition, the DSM can also be modified (DSM+) such that importance scores of each task are stored along the diagonal of the matrix which is normally unused (e.g., importance of first task is stored in row 1, column 1). For a set of n tasks, an error in task i could have a severity defined in Equation (2), equal to the proportion of each related, subsequent design task as a fraction of their correlation, ri,k, with each correlated design task k multiplied by their customer weight, wk, found through Design Task QFD, of that respective task. The elements ri,j are the elements in the i-th column and the j-th row of the DSM.

smax − sc ,1 − sc , 2

c

I

Design Tasks Engineering Metrics


An approach is to consider the errors and the immediate effects (i.e., defects) rather than the end effects and consequences (i.e., failures). One way to quantify the criticality is by valuing errors in a DFSS approach through the customer using QFD. The application of Design Error QFD, illustrated in Figure 3, relates customer requirements to the local effects of the design defects as an alternate method of defining severity (Chao and Ishii 2003b). Failure to satisfy the customer’s feature, cost, or time requirements should be considered an error for the organization. Cost and revenues or sales volume are not always the best metric, particularly for organizations such as NASA whose goal is not to sell products or gain profit. Using this definition for error, it is possible to use the Voice of the Customer (VOC) from QFD to help quantify and link errors to the design process. Each mistake or error in a design task is decomposed by the desired customer requirements that are not achieved and the engineering metrics which reflect it. Rather than mapping engineering metrics to parts in House II, design errors are correlated to the engineering metrics. This values the errors according to the customer perspective, which is important in defining quality of the design and identifies critical errors. A rule of thumb for QFD is to use around a dozen requirements and metrics. However, for error analysis, perhaps more specific terms are needed like “design variables” based on engineering specifications instead of engineering metrics. The weighting of design errors can also aid indexing. The error commonality index (Chao and Ishii 2003a) is an indicator which can be used in to organize and identify similar error modes and compatible error-proofing solutions. The weight of each error category, wc, derived from voice of the customer in the QFD can also be used to customize the ECI based on the utility of each of the different error category scored, sc. The customer-weighted error commonality index is shown in Equation 1.

(1)

3.3. Design task prioritization Another aspect to consider is the importance of different task along the process. Mistakes in different parts of the process have different consequences. Quantification can be simplified by asking the engineers to relate the importance of the task to the original customer requirements through Design Task QFD (Chao and Ishii 2003a). This QFD connects the end-user customer requirements with design tasks. Shown in Figure 4, House I is similar to that in Design Error QFD, while House II correlates the tasks in the design process to metrics.

7


Challenges and Methods in the Quantification of Design Errors and Solution Elements Chao and Ishii

ri ,k

n

Si = ∑ k =1

n

∑r

wk

n

ri ,k

k =1

(n − 1) ⋅ rmax

Si = ∑

(2)

The issues with this are if one should consider the “zero” entries in the DSM and further dilute the severity. Another possibility is to calculate the entire thing recursively, as in Equation (5).

If the subsequent design tasks are of little importance, than the severity of an error in this design task should be minimal. For example, Figure 5 shows a dependency diagram for one task a, which impacts three other tasks, b through d.

n

ri ,k

k =1

(n − 1) ⋅ rmax

Si = ∑ b a

DEPENDENCY DIAGRAM OF FOUR TASKS

The related Design Structure Matrix would look like Figure 6, where the values of ra,b, ra,c, ra,d will be non-zero.

a

Figure 6.

b

c

d

rb,a

rc,a

rd,a

rc,b

rd,b

ra,b ra,c

rb,c

ra,d

rb,d

rd,c

a

rc,d

c

To find the severity from task a, we would simply refer to the first column of the DSM, which tells us how task a impacts the others. Considering the subsequent process effects, the severity for an error in task a would be calculated as in Equation (3):

ra ,b + ra ,c + ra ,d

wb +

ra ,c ra ,b + ra ,c + ra ,d

wc +

ra ,d ra ,b + ra ,c + ra ,d

wd

e

d

FOUR TASK DESIGN STRUCTURE MATRIX

ra ,b

(5)

b

Figure 7.

Sa =

Sk

The issue with this is that the “last” task must still be calculated somehow, like using the Design Task QFD, and there is the strong possibility of looping. Another way to use DSM would be simply with a binary (0/1) representation of whether the task is done correctly. If that were done, one could consider assigning a proportion of the “total” customer weight to each of the tasks, and simply using DSM to track which tasks would be affected, and summing them. For example, in the following situation of Figure 7 with six tasks related as such, an error in task, d, will only impact tasks e and f.

c d

a b c d

(4)

i, j

j =1

Figure 5.

wk

f

IMPACT OF AN ERROR IN TASK D

The severity of this error in d is therefore the sum of the weightings found through QFD for tasks d, e, and f. Of course, this method would more strongly rate earlier errors in a “serial” process versus latter errors. An error in task a above would be a “100%” or maximum rating as it would, in this depiction, affect every other task. Though in some situations, there might be parallel tasks like shown in Figure 8 where this will not occur. Presumably, the different parallel tasks would come together at some point for a unified system or product.

(3)

where the w’s are found from the Design Task QFD. This calculation can give a quick idea of which tasks impact the most important subsequent tasks. The tasks with higher severity scores might be areas where the team would want to introduce additional design reviews. An alternative is to proportionalize the weight of all the correlations by normalizing it against the sum of the maximum ratings – probably a “9” - rather than simply the sum of the correlations, as shown in Equation (4).

a

c d

b Figure 8.

8

e

INDEPENDENT, PARALLEL SETS OF TASKS


Challenges and Methods in the Quantification of Design Errors and Solution Elements Chao and Ishii Next, a Design Task QFD House II correlates the engineering metrics with the design tasks, thereby linking the Design Process FMEA to the QFD. Methods like HEART can be used. Then, determining which engineering metrics these mistakes will impact and rolling back to scenario-based FMEA will allow use of costs as the severity measure instead of RPN. Finally, multiplying the probability and cost produces the expected cost of the critical design process errors. This proposal essentially means performing two distinct FMEA’s – one product and one process FMEA. This is particularly useful, for example, if a Scenario-based FMEA is already part of the design process procedure, as it can be used as a starting point. By first performing the Scenario-based FMEA on the product that is being designed or developed, the data from it can then be fed into the Design Process FMEA. Having the QFD and FMEA groundwork laid out before the analysis starts not only facilitates execution but can also aid those not familiar with either the problem or the process speak with a common language and framework. The failure modes that occur could also be related to a scenario outlined in the Scenario-based FMEA. Because the Scenario-based FMEA, Product QFD, and Design Process FMEA are all connected, it is possible to work backwards and find potential design tasks where errors might have occurred – i.e., a root cause analysis. Since the QFD relationships can ultimately relate the design tasks to the failures (negative VOC’s), it also helps analysis of root causes of the failure to satisfy the customer requirements.

Another question is whether it would make sense to sum the weights from more than one depth, perhaps even to the end of the process. Certainly, an error in one task would likely affect tasks later. However, the link may not be very strong and possibly the error would not really affect meeting of customer needs. Considerations of how design reviews impact the propagation of error through the tasks and the usage of DSM to re-order the process complicates the analysis. For these reasons, the recommended scope is limited to local effects, namely the immediate engineering metrics affected. In addition, one can certainly link design tasks to product development partners, contractors, or other design teams. This gives us a quick estimate of seeing how different teams contribute to the product development process. If different teams are given specific responsibilities but have their own distinct processes, we can use QFD to unify the separate design processes. One option would be to have a matrix containing all the various types of design tasks. Another would be for each individual design process to be considered as a distinct i x 1 House IV’s, allows individual and separate analyses. 3.4. Using Expected Cost and QFD Both probability and cost data can be hard to obtain and estimate while RPN is criticized for its use of ordinal scales. However, the QFD can be used with them, rather than instead of. Illustrated in Figure 9, the method starts with performing a Traditional (Product-based) QFD House I on the customer requirements and engineering metrics of the product. House I will be important in estimating the expected costs of different failure modes. These failure modes can be used to estimate the costs of not meeting the customer requirements (negative VOC’s). The failures modes identified in the Scenario-based FMEA (Kmenta et al. 1999) can be compared with the customer requirements in the QFD to see how they could essentially be viewed as negative VOC’s. This connects QFD to expected costs. The QFD then identifies relationships with the design process tasks to identify where failures may occur.

4. PRODUCT DEVELOPMENT WORTH OF ELEMENTS Design process error-proofing research is also beginning to look at using these tools for resource allocation and selection, primarily through advanced product definition tools like Project Quality Function Deployment. More so than quantifying design errors, the quantification of the worth of solution elements would benefit from a stronger tie in with monetary values, particularly the bottom line of an organization, including Net Present Value or Return on Investment.

Scenario-Based

Design Process

FMEA

FMEA

Figure 9.

4.1. Product development metrics Product development metrics are important because “you can only manage what you can measure.” Effective metrics must be simple, minimum, practical based on business objectives and business process. There are four types: process, program/project, product, and enterprise metrics. For example, the “number of customer needs identified” or “verification percentage” could reflect requirements and specifications, process capability could reflect product assurance, and actual versus planned staffing could reflect program management [5]. Cooper (1985) identified key dimensions, including product quality and uniqueness; overall project/company resource compatibility; market need, growth,

Expected Cost

Cost

Probability

EM Effects

QFD I

Task Error

Expected Cost

Cost

Probability

End Effects

Negative VOCs


Engineering Metrics

USING QFD TO LINK SCENARIO-BASED FMEA WITH DESIGN PROCESS FMEA

9


Challenges and Methods in the Quantification of Design Errors and Solution Elements Chao and Ishii and size; and market competitiveness. Griffin (1993) believed that the “best measures of product success in the market place would be a combination of market share, profitability, and customer satisfaction. It is rare, however that these data are available.”

QFD to a “Project Risk QFD” so that House II relates the metrics to project risk generated in Project FMEA, and House III maps the project resources to the project risks, thereby quantifying their risk reduction potential. In this correlation matrix, the potential of each project resource or solution element to reduce the risks is entered. Though the risk reduction potential can be plotted as a raw score, it could also be scaled without affecting the order. Figure 10 shows a simple way to look at how solution elements benefit the project or reduce the risk is by creating a three dimensional cost-worth graph that simply maps the benefit against the potential. By keeping the benefit and the risk reduction potential separate, no transformations are necessary. This visual representation is a simple way of comparing the solution elements.

4.2. Solution element prioritization Other than design errors, there are a number of other elements in product development where customer-based prioritization and quantification can be helpful. QFD can be used to prioritize different elements related to design errorproofing, including design tasks, design errors, and project resources. Because product definition is so important in the development process, it must be done thoroughly and accurately, and for this reason, Qualify Function Deployment is a significant design methodology. Project QFD (Chao and Ishii 2004) is a variation of the traditional QFD which focuses on the design team and their project priorities, product definition, and strategic alignment. This method connects organizational, project, or mission requirements to different project and engineering metrics for a systematic, transparent, and quantifiable analysis. The organization’s Project Requirements are mapped against Project Metrics like project budget and time-to-market. In House II, the Project Metrics are mapped against Project Resources, notably solution elements and organizational tools. This analysis can help assist decisions when limited time and resources allow only limited implementation of solution elements. Through marketing and business case analysis as well as scorecarding, the fulfillment of project requirements can be tied to the Net Present Value of a project. In addition to bringing benefit to an organization through increasing the NPV, they also can prevent or reduce risks. These resources are there to prevent problems or “negative” Voice of the Customer. Likely, many tools can impact multiple project enhancements or multiple risk reductions.

Element Cost: Usage or implementation cost or time

Project Benefit: Project QFD worth in terms of VOC’s

Risk Reduction Potential: – RPN

Figure 10. 3D PROJECT COST-WORTH CHART

The worth used in traditional Project QFD is not tied directly to Net Present Value of the project nor risk reduction potential. Usually, when one thinks of the value of a solution element they look at how it enhances the project. This often can be correlated with increased earnings due to the usage of this tool, but it is not always easy to look at the value of the risk reduction. If one can cite the impact of the tool in terms of NPV, by plotting the project enhancement and risk reduction side-by-side in a two-dimensional cost-worth chart, there is a concise representation of the solution element cost versus NPV. However, putting these different measures on the same scale is not a simple transformation. Unfortunately the conversion between RPN to cost is not one-to-one. Kmenta (1999) confirmed that the relationship between RPN and expected cost is not monotonic. Scorecarding (Ishii 2004) is one method of converting the impact of the solution elements in terms of NPV. With Project Scorecarding, the goal is to see the effects of solution elements on the project objective (Biggest Y), most likely NPV, ROI, IRR, and/or market share. The objective measures (Y and y) are the project metrics, while the control factors (Vital X) are the project solution elements. The noise factor’s (V’s) include project risks, market demand, and variance in the solution elements themselves. Finally, the transfer function will relate the VOC fulfillment of these solution elements to NPV. This allows analysis of the potential to change to NPV through different solution elements.

4.3. Project Cost-Worth Cost-worth analysis is a tool to organize and determine the benefit of different design elements, like parts or resources. The cost of implementing a solution element is a major factor in determining whether it will be used. Another factor is the benefit it can bring. As part of Project QFD, preliminary Project Cost-Worth analyses have been done using the “worth” calculated in House II and the “cost” to implement the element. In these calculations, the worth was determined by how the solution elements met project requirements, but they may also prevent losses or other negative factors from occurring. Using QFD, the project benefit could be summed together to one relative percentage or weight. The obvious corollary would be to sum the RPN’s that the elements affect and plot that as the third axis. However, with FMEA, rank-ordered lists of project failures should not be added. One solution is to continue to perform ordinal operations by modifying Project

10


Challenges and Methods in the Quantification of Design Errors and Solution Elements Chao and Ishii [J] Griffin, A., 1993, “Metrics for Measuring Product Development Cycle Time.” Journal of Product Innovation Management. March, 10(2): 112. [K] Herkert, J.R., 1994, “Ethical Risk Assessment: Valuing Public Perceptions.” IEEE Technology and Society 13(1):4-10. [L] Hollocker, C.P., 1986, “Finding the Cost of Software Quality.” IEEE Transactions on Engineering Management, Vol.EM-33, No.4, November, 223-227. [M] Ishii, K., (ed.), 2004, ME317 Design for Manufacturability Course Reader, Stanford University, Stanford, CA. [N] Juran, J.M., et al., 1974, Quality Control Handbook, McGrawHill, New York. [O] Keeney, R.L., and von Winterfeldt, D., 1991, “A prescriptive risk framework for individual health and safety decisions.” Risk Analysis 11(3): 523-534. [P] Kmenta, S., Fitch, P. and Ishii, K., 1999, “Advanced Failure Modes and Effects Analysis of Complex Process,” Proceedings of the ASME DETC: DFM. Las Vegas, NV. [Q] Leveson, N.G., 1995, Safeware: System Safety and Computers, Addison-Wesley Publishing, Reading, MA. [R] McRobb, M., 1979, “Lost Costs: The Costs of Detecting an Error.” Purchasing & Quality. Marcel Dekker, Inc. [S] Mock, T.J., and Grove, H.D., 1979, Measurement, accounting, and organizational information. Wiley, New York. [T] NUREG/CR-1278, 1980, Handbook of Human Reliability Analysis with Emphasis on Nuclear Power Plant Applications. [U] Reason, J., 1990, Human Error, Cambridge University Press, Cambridge, England. [V] Reason, J., 1997, Managing the Risks of Organizational Accidents, Ashgate Publishing Ltd., Aldershot, England. [W] Sarbacker, S., and Ishii, K., 1998, "Application of a Framework for Evaluating Risk in Innovative Product Development," International Journal of Agile Manufacturing. [X] Stamatis, D.H., 1995, Failure Mode and Effect Analysis: FMEA from Theory to Execution, American Society for Quality. [Y] Williams, J.C., 1986, “HEART: a proposed method for assessing and reducing human error,” Proceedings of the Ninth Advances in Reliability Technology Symposium, University of Bradford. [Z] WASH-1400 (NUREG-75/014), 1975, Reactor Safety Study— An Assessment of Accident Risks in U.S. Commercial Nuclear Power Plants.

5. CONCLUSIONS AND FUTURE WORK The goal of design process error-proofing is developing strategies and supporting methodologies to predict and prevent errors in product development. An important step in implementing change is the ability to quantify and prioritize elements of design, like errors and project resources, for analysis on the engineering level. Already methodologies like Design Process FMEA and Project QFD help identify both design errors and solution elements for design. RPN and expected cost have their advantages and disadvantages, but QFD seems to be a good approach for error quantification in terms of ease of application and practicality in the “triage” of error-proofing priorities. The use of Error Priority Number derived from QFD-based measures of importance and relative likelihoods of occurrence and detection of errors means no ordinal multiplications but a less “universal” meaning. These scales are admittedly not meant to be precise values but rough approximations with low granularity scaled to a 0-9 range. In addition, Design Structure Matrix can be a helpful in allowing compact representation of the numerous relationships between different tasks and possibly with the houses of QFD. The cost and benefit of solution elements is important given our definition of design errors and the desire to eliminate unnecessary redesign and rework loops. Future research will systematize the procedure for Project Scorecarding to facilitate transformations into Net Present Value. Due to the scope of product development projects, the effort required in ascertaining the NPV is worthwhile to make proper decisions.

REFERENCES [A] Automotive Industry Action Group (AIAG), 1996, Potential Failure Mode and Effects Analysis (FMEA) Reference Manual, February. [B] Chao, L.P., and Ishii, K., 2003a, “Design Process Error-Proofing: Development of Automated Error-Proofing Information Systems,” Proceedings of the ASME DETC: DAC, Chicago, IL. [C] Chao, L.P., and Ishii, K., 2003b, “Design Process Error-Proofing: Failure Modes and Effects Analysis of the Design Process,” Proceedings of the ASME DETC: DFM, Chicago, IL. [D] Chao, L.P., and Ishii, K., 2004, “Design Process Error-Proofing: Project Quality Function Deployment,” Proceedings of the ASME DETC: DFM, Salt Lake City, UT. [E] Clausing, D., 1994, Total Quality Development. ASME Press. New York, NY. [F] Clemens, P.L., 2002, “Human Factors and Operator Errors,” 2nd edition, Jacobs Sverdrup, February. [G] Cohen, L., 1995, Quality Function Deployment: How to Make QFD Work for You, Addison-Wesley Publishing Company, Reading, MA. [H] Cooper, R.G., 1985, “Selecting Winning New Product Projects: Using the NewProd System.” Journal of Product Innovation Management. 2:34-44. [I] Dewar, D., ed., 2003, “Timely Tips for Teams.” QCI International, June, Cottonwood, CA.

WEB REFERENCES [1] Design Defects of the Ford Pinto: Engineering Disaster http://www.fordpinto.com/blowup.htm

[2] September 11th Victim Compensation Fund of 2001 http://www.usdoj.gov/victimcompensation/

[3] MSNBC.com/Steven Brill, “A Tragic Calculus” http://www.msnbc.com/news/676986.asp

[4] CNN.com/AP, “9/11 fund deadline approaching” http://www.cnn.com/2003/US/Northeast/05/26/attacks.victims.fund.ap/

[5] Kenneth Crow, “Product Development Metrics” http://www.npd-solutions.com/pdmetrics.html

[6] Bhopal Survivors Sue Union Carbide http://www.getipm.com/articles/bophal.htm

[7] CNN.com, “Daniel Pearl’s wife denied money from 9/11 fund” http://www.cnn.com/2004/US/Northeast/03/30/pearl.widow.ap/index.html

11
