Exploring Differences in Preference ... - NCSU COE People

Proceedings of the ASME 2011 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference IDETC/CIE 2011 August 28-31, 2011, Washington, DC, USA

DETC2011-48596 EXPLORING DIFFERENCES IN PREFERENCE HETEROGENEITY REPRESENTATION AND THEIR INFLUENCE IN PRODUCT FAMILY DESIGN Eric Sullivan Graduate Research Assistant North Carolina State University Raleigh, North Carolina, USA [email protected]

1

Scott Ferguson Assistant Professor North Carolina State University Raleigh, North Carolina, USA [email protected]

ABSTRACT When using conjoint studies for market-based design, two model types can be fit to represent the heterogeneity present in a target market, discrete or continuous. In this paper, data from a choice-based conjoint study with 2275 respondents is analyzed for a 19-attribute combinatorial design problem with over 1 billion possible product configurations. Customer preferences are inferred from the choice task data using both representations of heterogeneity. The hierarchical Bayes mixed logit model exemplifies the continuous representation of heterogeneity, while the latent class multinomial logit model corresponds to the discrete representation. Product line solutions are generated by each of these model forms and are then explored to determine why differences are observed in both product solutions and market share estimates. These results reveal some potential limitations of the Latent Class model in the masking of preference heterogeneity. Finally, the ramifications of these results on the market-based design process are discussed.

Joseph Donndelinger Staff Research Engineer General Motors Warren, Michigan, USA [email protected]

2. BACKGROUND The authors‟ work on combining market-based design and system optimization began by exercising a homogeneous market model to perform multidisciplinary design optimization of a single product [2,3]. The S-model [4,5] was used in this work to assess the overall marketplace competitiveness of a preliminary vehicle design. While useful for demonstrating the integration of market simulation and design optimization, the practical applicability of this multidisciplinary design framework was limited by the homogeneous nature of the market simulator, only permitting the design of a single vehicle. In subsequent work, the authors developed a heterogeneous version of the S-Model [6] and later employed a hierarchical Bayes mixed logit model [1] allowing for extension beyond optimal design of single products to optimal design of product families. In this work, the differences between applying latent class multinomial logit (LC-MNL) and hierarchical Bayes mixed logit (HB-ML) market simulation models are explored. The model forms for these market simulators are discussed in Sections 2.1 and 2.2. Section 2.3 covers assessment of design commonality within product lines designed to maximize share in markets with heterogeneous preferences. These are the technical underpinnings for the work presented in this paper.

1. INTRODUCTION In market-based design, it is necessary to use a heterogeneous market simulation model to represent the inherent differences that exist between people within a given market. In previous work by the authors, both continuous and discrete representations of this heterogeneity were used to design an optimal product line [1]. However, significant differences between the solutions were observed affecting the designer‟s choice of products to be brought to market. This work aims to explore the differences between the two ways of representing heterogeneity. In doing so, the authors hope to identify the shortcomings and advantages of each model and evaluate their effects on product line design.

2.1 Discrete Representations of Heterogeneity of Preference Stated most generally, and in keeping with terminology used by Train [7], heterogeneity in customer preferences can be defined as variation in taste across individuals. Discrete representations of heterogeneity have been most common in earlier engineering design applications [8-10], perhaps because their development and interpretation are more intuitive. The fundamental approach is to subdivide a sample of respondents

1

Corresponding Author

1

Copyright © 2011 by ASME and General Motors

x1 3

x2 2

x3 5

x4 6

x5 2

x6 3

Table 1: Number of levels for each product feature x7 x8 x9 x10 x11 x12 x13 x14 3 2 4 2 3 2 4 3

into discrete groups and to represent the tastes within each group using scalar parameters. McConville and Cook [11] observed striking differences in self-explicated willingness to pay for certain automotive features and consequently segmented the market into smaller groups with more uniform preference structures. To capture market heterogeneity, latent class multinomial logit models classify respondents into groups with similar preferences. This analysis is performed by simultaneously estimating respondents‟ probabilities of membership in each class and part worth utilities representing the preferences within each class, as shown in Equation 1. Here, Q(s) is the probability of a respondent belonging to class s, (i|s) is the probability of the respondent choosing i conditioned on belonging to class s, and P(i) is the overall probability of choice for alternative i.

x16 4

x17 4

x18 3

x19 2

assumptions concerning stochastic utility terms, because of the computational expense involved in both fitting and evaluating these models [15]. However, mixed logit models have been applied to engineering design problems by Shiau et al. [18], Ferguson and Donndelinger [1], Chen [19], and Michalek et al. [20]. Donndelinger et al. [21] demonstrated that results of a vehicle configuration design problem can be highly sensitive to the specification of the market demand model. More notably, Shiau and Michalek [22] established that when using multinomial logit to model customer choice while simultaneously perturbing more than one alternative in a design optimization problem, every perturbed design alternative will converge to the same solution. As opposed to the latent class model, a hierarchical Bayes mixed logit model represents preferences at the respondent level. However, it is not usually possible to estimate preferences for any single respondent from that respondent‟s choice task results alone. In the upper level of the hierarchical Bayes mixed logit model, it is assumed that all individuals‟ preferences conform to a multivariate normal distribution [23]. The mean of this distribution is updated with choice task data using a Markov Chain Monte Carlo process to generate posterior estimates of preferences at respondent level. This lower level is characterized by a multinomial logit that governs each respondent‟s probability of selecting an alternative. One of the challenges in applying the hierarchical Bayes mixed logit model form is that its smooth mixing distribution does not readily lend itself to customer segmentation. Thus, there is no natural by-product of the hierarchical Bayes model estimation process that may be used for customer segmentation, as there is from the class structure of a Latent Class model.

S

P(i)   (i | s)Q( s)

x15 3

(1)

s 1

In latent class analysis, a multinomial logit model is fit for each of the classes (which could be interpreted to represent market segments). An advantage of this formulation is that respondents are not constrained to belong to only one class; instead, each respondent may have non-zero probabilities of membership in multiple classes. Additionally, as preferences are represented by multinomial logit models within each class, there exists a single globally optimal product design for each class. Typically, latent-class models employing a main-effects additive utility function provide an adequate representation of customer preferences. However, the burden of assessing the propriety of the model form rests on the designer, as it may or may not be appropriate to include interaction terms in the utility function. Furthermore, some prior knowledge of market or customer segmentation is critical in this analysis. The challenge of identifying the most suitable degree of segmentation must be addressed before proceeding to product design.

2.3 Assessing Product Line Commonality Commonality within a product line benefits a product producer through decreased manufacturing costs [24]. However, in a market with heterogeneous customer preferences, excessive commonality in a product line could be detrimental, as described by Miller [25]. Research into the management of commonality within product families led to development of metrics for commonality [24]. These metrics are essential for quantifying tradeoffs between manufacturer costs of product development, and production and market coverage with customer populations exhibiting highly heterogeneous preferences. In this work, the Commonality Index proposed by Martin and Ishii will be used to assess commonality within each product line solution [26, 27]. This metric uses the number of unique features between products as input and returns a percentage of common features as output, as in Equation 2.

2.2 Continuous Representations of Heterogeneity Models with continuous representations of heterogeneity have recently become more prevalent in market-based engineering design. By nature, all random utility models in the generalized extreme value family include continuous representations of heterogeneity in some form through their stochastic terms. This includes the popular multinomial logit model applied by Wassenaar and Chen [12], Wassenaar et al. [13], Michalek et al. [14], and Kumar et al. [15] as well as the nested logit models used by Kumar et al. [16,17]. Researchers in the engineering design community have avoided random coefficient models, despite their much less restrictive

(2)

2


Here, u represents the total number of available product features, pj represents the number of features in product variant j, and vn is the total number of product variants offered. For the purposes of this study, the Commonality Index is used to measure the variety within the product line solutions developed by each of the demand models. Further, it is used to determine the similarity of products across product line solutions. Having discussed the background and theoretical nature of the LC-MNL and HB-ML market simulation models along with the method for measuring product commonality, the optimal designs of a product family may now be explored. The next section covers the approach to comparing the two representations of heterogeneity; Section 4 contains product line solutions to the problem.

Table 2: Minimum allowable prices for levels of product feature x4 Feature Level Minimum Price 1 $143 2 $384 3 $402 4 $545 5 $786 6 0 („none‟ selection) A choice-based conjoint study was previously fielded to capture the heterogeneous preferences of consumers for the product features considered in this study. The data consists of responses from 2275 respondents to a battery of 19 choice tasks, yielding a total of 43,225 observations. The CBC/Web System for Choice Based Conjoint from Sawtooth Software [28] was used to collect the choice task data. HB-ML models were fit using the Sawtooth Software CBC/HB module [23]. Additionally, LC-MNL models were fit using the Sawtooth Software CBC Latent Class module [29]. The objective of this paper is classifying the differences in customer representation and product line solution when using LC-MNL and HB-ML model fits. An approach has been developed to tackle this objective. The first step in this approach is identifying the most suitable number of latent classes for the population of respondents, which in turn determines the size of the product family. This is followed by finding product line solutions with each market simulation model. These solutions are then compared to gain insight into each model‟s representation of heterogeneity in customer preferences through the use of different choice rules and correlation metrics.

3. MOTIVATION AND APPROACH In this paper, we consider the following scenario. A company is bringing new products to market with a number of available features. Each feature is offered at a number of different levels representing either removal of the feature from the product or upgrades from basic feature functionality. The motivation for modeling product features in this manner is twofold: 1) customers are apt to upgrade or downgrade their preferred product configuration based on available variants and the number and types of various feature packages available, and 2) unforeseen changes in technology, supply chains, or other factors introduce substantial uncertainty in feature offerings, possibly resulting in cost increases for features or lack of feature availability. To capture the greatest possible market share, a reasonable strategy for the company would be to design products specifically targeted toward large pockets of customers with similar preferences. This is achieved by determining the optimal segmentation of the population followed by the design of a product line containing a number of products equal to the number of segments identified by the segmentation process. A product line is designed using each of the two market simulation models to provide a comparative basis for the two representations of heterogeneity. Now, consider that products in this product line are designed as combinations of 19 unique features, as shown in Table 1. Additionally, minimum allowable prices for incorporating each feature into a product have been determined based on feature costs and various business goals. Examples of minimum prices for available levels of product feature x4 are shown in Table 2. In this work, the overall price of the product is determined by summation of the minimum allowable prices for each feature level present in the product. This pricing model is acceptable for this work as it is a common industrial practice in which this research is being conducted. Within this industry, it is also typical that each feature level has the same price adjustment across all products offered in the product line. Amendments to this pricing scheme as part of future work are presented and discussed in Section 7.

4. SOLUTIONS FOR HB-ML & LC-MNL MODELS To provide reference points for comparison of the two representations of heterogeneity, product line solutions are generated using each of the market simulation models. To determine the number of products to include in the product lines, it is necessary to determine an appropriate number of classes for the latent class model. The process for comparing goodness of fit for latent class solutions with different numbers of classes is covered in Section 4.1. Following this, the solutions for the latent class and hierarchical Bayes models are presented in Section 4.2. 4.1 Determining the Number of Latent Classes Sawtooth Software‟s CBC Latent Class module [29] allows users to specify anywhere from two to thirty latent classes. The software performs a maximum likelihood solution to assign respondents to classes. This solution is sensitive to initial conditions and must be repeated from various starting points. A number of statistical measures can be used to assess goodness of fit: Percent Certainty [30,31], the Consistent Akaike Information Criterion (CAIC) [32, 33], the Chi Squared statistic, and the Relative Chi Squared statistic. These statistics, however, can provide conflicting information on the optimal

3


number of classes in the model [33]. Typically, there is a greater probability of identifying too many classes, rather than too few. Designing too many products because customer segmentation is believed to be finer than it actually is detracts from a manufacturer‟s profitability by increasing the cost of manufacturing and creating product differentiation issues with negligible offsetting gains in revenue or market share. There is neither a single statistic nor a particular combination of these statistics that guarantees identification of the appropriate number of classes; the modeler determines an appropriate number of classes by examining trends across all of the available statistics. The goodness of fit statistics for the latent class model fits in this work are shown in Table 3. There is a large decrease in CAIC (smaller is better) when going from 2 to 3 classes, with diminishing decreases in CAIC for larger numbers of classes. Similar trends are seen for changes in Percent Certainty (larger is better) and Chi Square (larger is better) statistics. A contrary trend is seen in the Relative Chi Square (larger is better) statistic, which suggests that a 2 class model provides the best fit.

Classes 2 3 4 5 6 7 8

Table 3: Goodness of Fit Statistics for Latent Class Models Percent CAIC Chi Sq. Certainty 26.15 89755.36 31338.92 30.14 85605.74 36118.95 31.93 84092.75 38262.34 33.26 83123.69 39861.81 34.31 82494.37 41121.54 34.91 82411.88 41834.44 35.30 82571.50 42305.22

Figure 1: Normalized Goodness of Fit Statistics for Latent Class Models Based on the consistency of the trends observed in CAIC, Percent Certainty, and Chi Squared statistics, and on the low rates of change in these statistics for models with 6 classes or more, the latent class model fit with 5 classes was selected for use in this work. Because there may only be 1 optimal product design for each class, this choice also determines that the product lines to be designed in this work will contain 5 products. Further verification of this decision is provided by Turner et al [34] with the same choice task data set utilized in this paper. Using product line solutions from the hierarchical Bayes model with a varying number of products, the logit transform of preference share achieves an optimal marginal improvement in preference share with five products. Product line solutions may now be found for both the latent class model and the hierarchical Bayes model. The process for finding these solutions, as well as the solutions themselves, are presented in the next section.

Rel. Chi Sq. 292.89 224.34 177.96 148.19 127.31 110.97 98.16

Continuing with latent class determination, Figure 1 presents a graphical representation of these statistics normalized in the direction of best fit. This figure verifies the correlation among the CAIC, Percent Certainty, and Chi Squared statistics. Again, it is noticed that the Relative Chi Squared statistic trends in the opposite direction as the other three. The normalized Chi Squared and the Percent Certainty statistics are almost identical, as indicated by the aligned curves in Figure 1.

4.2 Product Line Solutions The Sawtooth Software Advanced Simulation Module [35] is used to conduct product searches to create optimal product lines with both demand models. This software allows the user to specify the search desired to find the optimal product line. For the cases presented in this work, the genetic algorithm search was chosen with an initial pool of 1,000 products, creating 500 offspring products each generation, and has a relative mutation rate of 0.5. To search for these optimal product lines, an objective function must be specified. Sawtooth Software has developed a share of preference statistic that decreases the share of preference a product captures proportional to its similarity with other products [35]. Use of this statistic for the objective function rather than the traditional share of preference yields results that are much less vulnerable to independence from irrelevant alternatives. While Sawtooth Software recommends Randomized First Choice as the choice rule for this type of search, the computational expense associated with this rule is significant. All results presented in this paper, however, are verified using the Randomized First

4


simulation models have a very comparable amount of variety. This result also shows that both demand models agree on a baseline amount of variety that should be present in a fiveproduct product line. When comparing the specific product results from each of these models, some similarities are noticed. For example, one interesting result is that a common product was generated for both models. Products HB1 and LC1 match levels at each of the nineteen feature attributes and correspondingly share the same price. This product also represents the „low-end‟ of the product line, featuring sixteen attributes where the „none‟ level was selected. Another similarity noticed is that each product search generated two „high-end‟ products to offer within the given product line. The „high-end‟ products are characterized by the absence of the „none‟ option from all nineteen features and prices close to the model upper price bound of $12,000. Product Line HB offers Products HB4 and HB5 as its „highend‟ products, while Products LC3 and LC4 are the clear „highend‟ products in Product Line LC. When comparing levels of each feature across all four of these products, there are some interesting results to note. First, for Feature 1, the „high-end‟ products in Product Line LC agree on the lowest level, while Products HB4 and HB5 both feature the highest level. A similar result is found in Feature 4 with Product Line HB containing higher levels than Product Line LC. However, to not violate the price constraint, Products HB4 and HB5 have generally lower levels for Features 17 and 18, because the more expensive options were selected on the Features 1 and 4. For the most part, the remaining fifteen features are mostly consistent between the four products. When considering the remaining two unique products in each product line, notice that Product Line HB presents two products very different from each other in selected levels and price. Product HB2 can be defined as a mostly „low-end‟ product priced toward the lower bound of price. It includes eight features where the „none‟ level is selected. Product HB3 represents a „mid-level‟ product priced squarely between the

Choice Rule. The advantages of Randomized First Choice as a choice rule are further discussed in Section 5.2 and [35]. This product line optimization process is first completed using the HB-ML model to create Product Line HB. The products included in Product Line HB are presented in Table 4, organized from the least expensive to the most expensive. The same optimization process was performed using the LC-MNL model with five latent classes. The product line solution resulting from this search is referred to as Product Line LC and is presented in Table 5. The order of the products in Table 5 is determined by each product‟s similarity with the products in Table 4. This similarity is measured by the Commonality Index presented earlier. For example, the Commonality Index between Product HB1 and each of the five products in Product Line LC is calculated. Whichever product in Product Line LC that generates the highest Commonality Index is labeled Product LC1. This process is repeated with comparing Product HB2 to the remaining, unlabeled four products in Product Line LC to determine which product should be labeled Product LC2. This is continued so that each product has a corresponding product in the opposite solution. While we understand that the large number of product features included in Tables 4 and 5 may prove difficult to digest, it is necessary to present this data to demonstrate the extent of variability that is present in the product lines. To help simplify the comprehension of these tables, features that have the most variability of levels present are identified. For Product Line HB, Features 4, 7, 9, 13, 15, 16 and 17 all show a diverse range of levels. Similarly, Features 4, 9, 13, 15, 16, 17 and 18 have a high variety of levels in Product Line LC. It is interesting that six of the seven features identified as the most diverse are common between both models. For the most part, the greatest product differentiation occurs within the same features between both product lines. Continuing on that note, the product line created using the HB-ML model has a Commonality Index of 0.618 while Product Line LC has a Commonality Index of 0.605. This proves that the product lines created by the two different market

Table 4: Base Product Line Solution when Products are Designed Concurrently using a HB-ML Model Product x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 x15 x16 x17 x18 x19 Price Prod HB1 1 2 5* 6* 2* 3* 3* 2* 4* 2* 3* 2* 4* 1 3* 4* 4* 3* 2* $320 Prod HB2 1 2 5* 1 1 1 1 2* 4* 2* 3* 2* 1 1 3* 1 4* 1 2* $1,973 Prod HB3 1 2 4 3 1 1 1 2* 1 2* 3* 1 3 2 1 1 4* 1 1 $7,802 Prod HB4 3 1 4 4 1 1 1 1 3 1 1 1 3 2 2 3 1 1 1 $11,742 Prod HB5 3 1 4 5 1 1 2 1 2 1 1 1 1 1 2 3 2 1 1 $11,934 Note: * mean that the „none‟ option was selected for this feature Table 5: Base Product Line Solution when Products are Designed Concurrently using a LC-MNL Model Product x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 x15 x16 x17 x18 x19 Price Prod LC1 1 2 5* 6* 2* 3* 3* 2* 4* 2* 3* 2* 4* 1 3* 4* 4* 3* 2* $320 Prod LC2 1 2 4 2 1 1 1 2* 1 1 3* 1 1 1 1 1 4* 1 2* $7,258 Prod LC3 1 2 4 2 1 1 1 1 3 1 1 1 1 2 2 1 2 2 1 $11,990 Prod LC4 1 1 4 1 1 2 2 1 2 1 1 1 1 1 2 3 3 1 1 $11,760 Prod LC5 1 1 5* 4 1 1 1 1 1 2* 1 2* 3 2 2 3 1 1 2* $7,514 Note: * mean that the „none‟ option was selected for this feature

5


price bounds and containing four features that have the „none‟ level selected. On the other hand, Product Line LC features two „mid-level‟ products that are very similar in price to Product HB3 and each introduce four features set at the „none‟ level. However, Products LC1 and LC5 only have the same levels for six of the nineteen attributes and neither one necessarily matches Product HB3 consistently. This occurs because of the idea that there are much more combinations of feature levels that can result in a price in this range. The „lowend‟ products have to be more common because to be „low-end‟ products, a major portion of the features must be set at the „none‟ level. Similarly, „high-end‟ products will have inherent similarity because features are set to the more expensive levels. From these observations of the differences between the two product line designs, it is obvious that the two models differ in their representation of respondent preferences. Section 5 aims to investigate the differences in how these models represent a heterogeneous market using Product Lines HB and LC as examples.

Instead of using this simulation to compare overall market capture, the results are investigated at the individual respondent level to determine if a respondent‟s „first choice‟ changes when the opposite demand model is applied to the same product line. Table 6 gives the breakdown of the respondents‟ first choices in the HB-ML model for both product lines, while Table 7 does so using the LC-MNL model. The breakdown of the respondents‟ first choices in Table 6 allows some conclusions to be made about which products are being selected the most. For Product Line HB, Products HB4 and HB5 are obvious favorites in capturing the most market share. Looking back to Table 4, realize that these two products are the two „high-end‟ products as they have no attributes with the „none‟ level selected and are at the upper end of the price bounds. The other three products capture additional market share at the lower to middle end of the market. Among Product Line LC, Product LC4 is a clear favorite that, like Products HB4 and HB5, represents a higher-end model than most of the products in Product Line LC.

5. DISCUSSION OF MODEL DIFFERENCES There are distinct differences between the solutions of each demand model, as shown in Tables 4 and 5. While these two solutions will capture multiple pockets of the heterogeneous market, the pockets that each product line captures may vary between the two demand models. This section aims to compare the degree of heterogeneity captured by the two demand models. In Section 5.1, a First Choice rule is used to track how a respondent‟s highest-utility product changes between the two demand models. Such a tracking of an individual‟s consumer selection is not possible when using a share of preference rule. A Randomized First Choice simulation of the base solutions, which provides a more accurate prediction of overall market shares, is then used as a motivating factor for exploring the difference in utility among chosen products in Section 5.2. Finally, correlation coefficients are used in Section 5.3 to explore the heterogeneity captured by each model type.

Table 6: First Choice Simulation in HB-ML Model First Choice First Choice Product Product Share Share HB1 9.01% LC1 12.13% HB2 9.19% LC2 6.15% HB3 6.73% LC3 10.55% HB4 17.36% LC4 17.05% HB5 17.89% LC5 10.07% Line HB 60.18% Line LC 53.95% Considering Table 7, multiple products from both product lines are not chosen a single time by any of the 2275 respondents when evaluated using the LC-MNL model. The result is a significantly lower expected market share for Product Line HB. This is most likely caused by the products in both lines falling outside of a specific latent class‟s preferences. Therefore the products are unable to achieve a utility high enough to beat the other four products or an outside good.

5.1 Change in Respondent’s First Choice between Models To understand how a respondent‟s highest-utility product changes when analyzed in each model, a First Choice rule is used. It is important to note that the First Choice simulation has limitations because of its extremity. Even if the „chosen‟ product has a utility only slightly greater than the product with the next highest utility, the First Choice method assumes that the higher utility product will be chosen 100% of the time. While theoretically this simulation is proper for representing rational consumers, determining market share in this manner is only valid when we are confident that each respondent‟s utilities are an exact representation of his/her preferences. Due to the stochastic nature of the model fit, it has to be acknowledged that there is some noise in the quantitative utilities. Furthermore, the demand models generated by these 2275 respondents are being used to represent a market that is of a global scale. To suggest that the respondents captured in this study perfectly represent the entire market is imprudent.

Table 7: First Choice Simulation in LC-MNL Model First Choice First Choice Product Product Share Share HB1 0% LC1 0% HB2 0% LC2 0% HB3 0% LC3 10.42% HB4 27.08% LC4 23.78% HB5 20.75% LC5 22.33% Line HB 47.82% Line LC 56.53% The differences between shares captured in the opposite model are important to companies that are targeting heterogeneous markets because incorrectly modeling the demand could cause major losses. Further, it is also important

6


models were ones that had selected the „low-end‟ HB1 in the HB-ML model. Also, there were 835 people that decided not purchase any of the offered products in either model. The same process is conducted for Product Line LC by simulating each respondent‟s first choice utilizing the HB-ML model. The breakdown of these choices is given in Table 9. Again, it is noticed that Products LC1 and LC2 aren‟t chosen by any of the 2275 respondents in the LC-MNL. Product LC1 again represents the „low-end‟ product with most of the features set to the „none‟ level. Product LC2 is a „mid-level‟ product that shares a similar price range with Product LC5. From this fact, one might expect the people that Product LC2 in the HBML model to select Product LC5 in the LC-MNL model. Instead, of the 140 that selected Product LC2 in the HB-ML model, only 21 select Product LC5 in the LC-MNL model. Of the remaining 119 respondents, 107 are split between the two „high-end‟ products and 12 select the outside good. It is again noticed that a low percentage, only 404 of the 1120, of respondents consistently chose a product in both models. The product that retained the most respondents between models was Product LC4, one of the „high-end‟ products. Also, 836 respondents chose not to select any of the offered products in either model.

Table 8: Breakdown of Respondents’ Choices on Product Line HB Between Models LC-MNL Model Prod HB1 HB2 HB3 HB4 HB5 O.G. Totals HB1 0 0 0 35 21 149 205 HB2 0 0 0 95 52 62 209 HB3 0 0 0 32 58 63 153 HB4 0 0 0 207 153 35 395 HB5 0 0 0 179 185 43 407 O.G. 0 0 0 68 3 835 906 Totals 0 0 0 616 472 1187 2275 The fact that Products HB4 and HB5 were the only ones selected when using the LC-MNL model is again reflected in Table 8. We also remember that these two products were classified as the „high-end‟ for Product Line HB. Only 392 of the 1017 respondents that selected a product in both models remained with the same product in each model. Of the 407 respondents that picked Product HB5 using the HB-ML, nearly half of them decided on the other „high-end‟ product, HB4, in the LC-MNL model. Another interesting insight into the results of this table is how the 205 respondents that had selected the „low-end‟ Product HB1 made decisions in the LC-MNL model. Of these 205 respondents, 21 switched their choice to Product HB5, 35 switched to Product HB4, and the remaining 149 decided to not buy any of the offered products. This suggests that although the respondents preferred a „low-end‟ product in the HB-ML model, some of them actually favor the „high-end‟ product in the LC-MNL model, indicating that they may belong to a latent class that they don‟t necessarily agree with in terms of feature preferences. To continue, notice that by using the LC-MNL model we capture 71 respondents that did not select one of our offered products in the HB-ML model. On the other hand, a total of 352 respondents that selected one of the products in the HB-ML model were not captured in the LC-MNL. These respondents are spread among each of the five products in the HB-ML model with the exception of Product HB1. As mentioned earlier, nearly half of the respondents that were lost between

HB-ML Model

HB-ML Model

to understand these differences at the respondent level. To investigate this idea - how each respondent‟s choice changes or remains the same – the simulations are run through the opposite demand model and we keep track of the decisions made within different models. It can then be determined how many respondents made choices that were immune to the difference in models. Determining which respondents were predicted to select a different product in the opposite model can be made along with identifying any respondents that were lost or gained from the outside good due to the change in demand models. Table 8 breaks down the results of this analysis on Product Line HB by recording the respondents‟ HB-ML choices in each row and their LC-MNL choices by column.

Table 9: Breakdown of Respondents’ Choices on Product Line LC Between Models LC-MNL Model Prod LC1 LC2 LC3 LC4 LC5 O.G. Totals LC1 0 0 50 35 64 127 276 LC2 0 0 43 64 21 12 140 LC3 0 0 51 132 52 5 240 LC4 0 0 25 221 139 3 388 LC5 0 0 10 81 132 6 229 O.G. 0 0 58 8 100 836 1002 Totals 0 0 237 541 508 989 2275

From the results presented in Tables 8 and 9, it is repeatedly observed that the LC-MNL model seems to favor the „high-end‟ products. Continuing, the consistency of the choices that the respondents make between each of the models is much lower than had been anticipated. The preference utilities that are generated in each of the models are clearly different. Now that the differences between the two demand models have been investigated using the First Choice choice rule, results will be observed using a different choice rule allowing us to gain more insight into the models‟ fundamental differences. Section 5.2 continues the comparison by exploring the differences in utility between chosen products. Understanding these differences will hopefully lead to a conclusion of which demand model best represents the entirety of a heterogeneous market.

7


shares from Table 7 to the Randomized First Choice shares presented in Table 10 it is hard to find much similarity. The only real similarity that is identifiable between the two sets of data is the rank order of share captured for each of the five products. Some of the disparities include a total product line share difference of 4.73% and each product‟s share is different by an average of 6.89%. However, the most glaring dissimilarity emerges as Products LC1 and LC2 don‟t capture any share in the First Choice simulation, yet they capture 9.27% and 7.76%, respectively, in the Randomized First Choice scenario. This exemplifies the exact opposite characteristics as Product Line HB. For Product Line LC, we would expect the overall product utilities to be much closer than Product Line HB causing the Randomized First Choice simulation to yield product utility values that don‟t match the order of the First Choice simulation. To mathematically prove that this interpretation of the two simulation scenarios is correct, we conduct a study of the first choice utilities of Products HB3 and LC2. We will start with Product HB3 by evaluating the utilities of all products in Product Line HB for the first respondent. We then determine which product is chosen by the respondent by taking the product with highest utility. Next, the difference between the utility of this highest-utility product and Product HB3 is calculated and recorded. This process is repeated for all 2275 respondents yielding distribution of utility differences for Product HB3. Recall that for 153 respondents, Product HB3 was actually the highest-utility product. The data for these respondents is disregarded because it is useless to compare the utilities of the same product in the same simulation scenario and will skew the data toward a zero difference in utility. A histogram of all 2122 points of utility difference is presented in Figure 2. By completing the same process with Product LC2, we will be able to create a comparison between the differences of utility. Remember that none of the 2275 respondents chose Product LC2 in the First Choice simulation, so every respondent records a utility difference between the highestutility product and Product LC2. The distribution of these utility differences is plotted alongside the results from the Product HB3 study in Figure 2.

5.2 Finding the Utility Difference between Chosen Products While the previous section outlined the use of the First Choice choice rule to compare each respondent‟s product decisions in the two demand models, a more robust model that is still independent of IIA challenges is needed to compute preference share. The Randomized First Choice choice rule provides a platform to accomplish this goal, and yields results that clarify the reasons Products HB1, HB2, HB3, LC1 and LC2 were never selected using the First Choice choice rule. The Randomized First Choice method alleviates the problems associated with the First Choice method by accounting for this noise in utilities. Randomized First Choice, while computationally expensive, is considered the best choice rule because it runs numerous First Choice iterations while methodically varying the utility values within a range of the calculated utility [35]. The average of the iterations is presented and gives the designers a better idea of how their products will fair. Table 10 breaks down the shares captured by each product line at the product level. For this table each product is simulated in the model by which it was created. Table 10: Randomized First Choice Simulation Product RFC Share Product RFC Share HB1 9.83% LC1 9.27% HB2 9.79% LC2 7.76% HB3 6.77% LC3 12.98% HB4 17.34% LC4 18.18% HB5 17.67% LC5 13.07% Line HB 61.39% Line LC 61.26% Here, notice that both product lines perform at a very similar level within their respective models, but the real objective of this analysis is not to compare between the two models again. Starting with the Product Line HB simulation, look back to Table 6 to determine the differences in estimated product share between the two simulation scenarios. Realize here that there is very little difference between the overall market shares with the First Choice simulation estimating 60.18% and the Randomized First Choice yielding 61.39%. Even at the product level, the largest difference between estimated shares of the same product in the two simulations is 0.82% for Product HB1. In fact, Products HB3 and HB4 are within 0.04% of estimated share between the two simulation scenarios. With the understanding that the Randomized First Choice choice rule varies the respondent utilities, this indicates that the individual product utilities of Product Line HB must be well differentiated from one another. If they were closer together, we would expect to notice a significant change in share between the two choice rules. To illustrate an example where we expect to see very close product utilities, we continue on to examine the two simulations of Product Line LC. First it is necessary to refer back to Table 7 to gather the shares estimated by the First Choice Simulation scenario. When comparing the First Choice

Figure 2: Distribution of Utility Differences

8


Figure 2 clearly indicates that the utility differences between Product LC2 and the rest of Product Line LC are much lower than those between Product HB3 and Product Line HB. In fact, it was found that on average Product HB3 recorded 9.38 units of utility less than the highest-utility product in Product Line HB. On the other hand, the average utility difference across all 2275 respondents for Product LC2 is determined to be 2.48 units of utility. Furthermore, the utility difference of Product LC2 never exceeds 5 units of utility compared to the 1680 respondents that indicate a utility difference greater than 5 units of utility for Product HB3. With this study, it is safe to assume that a similar pattern is present across all of Product Lines HB and LC. Since the utility differences of Product Line HB are significantly higher than Product Line LC, it is understood that the Randomized First Choice simulations of Product Line LC are fluctuating the utilities enough to cause the shares to alter considerably. However, the usefulness of this study is not limited to a comparison between the simulation scenarios. This study also provides some valuable insight into the difference between the HB-ML and LC-MNL demand models. The much narrower distribution of utility difference in Product Line LC may indicate a lack of variety in the respondent utilities from the LC-MNL model. To verify this, a study of the correlation within the two demand models is conducted in Section 5.3.

Latent Class

Table 11: Correlation Matrix for Five Latent Classes Latent Class 1 2 3 4 5 1

1.00

2

0.903

1.00

3

0.775

0.644

1.00

4

0.472

0.289

0.548

1.00

5

0.889

0.958

0.683

0.263

1.000

The high correlation among latent classes could mean one of two things. First, it is possible that respondents in these classes do have similar preferences and prefer the same types of products. On the other hand, the LC-MNL model may be masking the heterogeneity of the market by trying to group people into latent classes that fail to capture the true diversity of the population. The only way to determine which case is represented in this scenario is through comparison of the LCMNL model correlation to the HB-ML correlation. From Sawtooth Software‟s Latent Class and HierarchicalBayes Modules, the individual respondent utilities from each model can be exported to measure correlation. By definition of model form, individual respondent utilities are directly calculated in a HB-ML model. For a LC-MNL model, a multinomial logit model is fit for each class, and respondents are assigned a probability of belonging to each class. Utilities can be calculated at the respondent level in a LC-MNL model by summing the product of class membership by each class‟s multinomial logit utilities. Proceeding in this manner allows for direct comparison of individual utilities between a HB-ML and LC-MNL model. As presenting a correlation matrix is impractical for a data set of 2,275 respondents, a histogram is constructed that displays the distribution of correlation measurements within each model. This histogram is presented in Figure 3. Notice here that the LC-MNL correlation is mostly clustered to the upper end of the correlation axis. In fact, 55% of the correlation statistics measured within the LC-MNL model have a correlation greater than 0.90. On the other hand, the HB-ML model predicts that only 0.10% of comparisons result in a correlation above 0.90. Further, correlations for the HB-ML are noticeably more spread across the range in the histogram, with 13.9% of the correlations being negative. This is in stark contrast to the LC-MNL model which has zero negative correlations between respondents. Furthermore, 21.4% of the correlations between respondents in the LC-MNL model have a value of unity (HBML model has zero correlations of unity). This suggests that the LC-MNL model is grouping people such that their correlations with respondents in other groups are extremely similar. This result provides evidence that the LC-MNL model may not capture the true heterogeneity that is present in this problem.

5.3 Demand Model Correlation From the results provided in the two previous subsections, it is hypothesized that the respondent utilities within the LCMNL model are much more similar than those in the HB-ML. This hypothesis can be further verified through the exploration of the correlation within each model. Higher correlations within the model suggest a lack of variety in the utilities. For the results presented in this paper, correlations are constrained to include only the utilities associated with product attributes. In depth analysis of the utilities associated with price, at both the level of the individual (HB-ML) and class (LC-MNL), show that the changes in part-worth utilities are typically monotonic. As the direction of this monotonic behavior is generally the same for all respondents (larger prices yield less positive utilities), correlation for this attribute tends to bias overall model correlation statistics. Further, as the outside good is quantified and scaled in relation to an individual‟s selection of the outside good in the choice tasks, this component is excluded from the correlation calculation. This analysis is started by examining the correlation between the latent classes identified in the LC-MNL model as shown in Table 11. The high correlation among Classes 1, 2, and 5 indicates a significant amount of similarity and raises serious questions about the heterogeneity of the model fit, as these three classes represent 71.2% of the market.

9


other product design scenarios. Section 6 addresses the affects of this conclusion on the product design process. 6. IMPLICATIONS FOR PRODUCT FAMILY DESIGN The dramatic differences observed in diversity of model parameters in the hierarchical Bayes and Latent Class models have significant ramifications for product family design problems. The motivation for designing a product family is to offer a broad enough variety of products to satisfy the diverse tastes of a population of customers. The broad spectrum of correlation coefficients observed for respondent utilities in the hierarchical Bayes model indicates that there is a high degree of heterogeneity of preference among the customers sampled in this work. The high levels of positive correlation observed for class utilities in the Latent Class model suggest that the heterogeneity of preference in this sample of customers is being masked in the Latent Class model; apparently the latent variable estimates in the Latent Class model cannot adequately represent the heterogeneity of preference for this sample of customers and the vast majority of variation in taste is being absorbed by the stochastic utility terms in the multinomial logit models fit for each class. This manifests itself first as strong positive correlation between class-level utilities and subsequently in relatively high levels of similarity between optimal product line solutions found using the Latent Class model. These results suggest that examination of correlations between utilities of agents (either respondents or classes) may be a valuable diagnostic measure when choice models are to be used for product family design problems. If correlation levels are similar, either model may be capable of yielding valid solutions. However, when strong positive correlations are observed, further investigation may be required to ensure that the model represents heterogeneity of preference well enough to generate product family design solutions with adequate levels of differentiation between products. In this work, the results of investigating the Latent Class model behavior suggest that the product line solutions found using the Latent Class model are not sufficiently differentiated to meet the business objective of maximizing penetration into a market with diverse customer preferences.

Figure 3: Distribution of Correlation Statistics Further investigation of the LC-MNL model can be conducted to investigate how the diversity of heterogeneity is impacted. Table 12 shows the number of respondents that belong to a latent class, or a combination of two latent classes, with a probability greater than or equal to 0.98. These results indicate that 66.7% of the respondents belong solely to one latent class. Additionally, Table 11 showed significant correlation between Classes 1, 2 and 5. The number of respondents in one of these classes, or a combination of these classes, represents 60.3% of the market.

Latent Class

Table 12: Group Memberships in Five Latent Classes with Probability >= 0.98 Latent Class 1 2 3 4 5 1

348

2

111

381

3

106

64

180

4

120

0

37

250

5

0

172

24

0

360

7. SCOPE OF FUTURE WORK The research presented in this work allows for interesting expansion in the market-based design field. To continue investigating the different demand models, the number of times each respondent chose the outside good can be explored as a way of classifying respondents‟ utilities. Also, a comparison of the utilities that were generated when no outside good is present can be explored. Continuing the investigation of the two demand models, a source of future work involves determining which model better predicts a set of known consumer part-worths. This idea has been explored in the area of marketing research by Andrews et al. [36], but further empirical evidence is needed before a final confusion can be drawn.

To adequately represent the diversity of a heterogeneous population, it would be expected that: 1) the number of respondents with full membership in one of the latent classes would not represent 2/3 of the total market; 2) over 60% of the market would not exist in a combination of latent classes with a correlation of 0.9 or greater; and 3) that there would be a larger number of respondents spread across multiple classes (or across two classes with very low, or negative, correlations). In this model, only 122 respondents have a probability of membership that extends beyond two latent classes. Although we strongly suspect that the LC-MNL does not adequately represent the heterogeneity that is present in this problem, further work is necessary to assess its practicality in

10


Competition,” Journal of Mechanical Design, 124(3): pp. 385-92. [10]Wassenaar, H.-J, Chen, W., Cheng, J. and Sudjianto, A., 2004, “An Integrated Latent Variable Choice Modeling Approach to Enhancing Product Demand Modeling,” Proceedings of the 30th ASME Design Automation Conference, Salt Lake City, UT, DETC2004/DTM-57487. [11] McConville, G., and Cook, H., 1997, “Evaluating Mail Surveys to Determine the Value of Vehicle Options,” SAE Transactions, Journal of Passenger Cars, 106: pp. 1290-7. [12] Wassenaar, H.-J. and Chen, W., 2003, “An Approach to Decision Based Design with Discrete Choice Analysis for Demand Modeling,” Journal of Mechanical Design, 125(3): pp: 490-7. [13] Wassenaar, H.-J, Chen, W., Cheng, J. and Sudjianto, A., 2004, “Demand Modeling for Decision-Based Design of Vehicle Engine,” Transactions of the SAE, Paper No. 2004-01-1535, SAE World Congress, Detroit, MI. [14] Michalek, J., Feinberg, F. and Papalambros, P., 2005, “Linking Marketing and Engineering Product Design Decisions via Analytical Target Cascading,” Journal of Product Innovation Management, 22: pp. 42-62. [15] Kumar, D., Kim, H. and Chen, W., 2006, “Multi-level Optimization for Enterprise-Driven Decision-Based Product Design,” Decision Making in Engineering Design (Lewis, K., Chen, W., and Schmidt, L. eds), pp. 203, ASME Press. [16] Kumar, D., Chen, W. and Simpson, T., 2009, “A MarketDriven Approach to the Design of Platform- based Product Families,” International Journal of Production Research, 47(1): pp. 71-104. [17] Kumar, D., Hoyle, C., Chen, W., Wang, N., Gomez-Levi, G. and Koppelman, F., 2009, “A Hierarchical Choice Modeling Approach for Incorporating Customer Preferences in Vehicle Package Design,” International Journal of Product Development, 8(3): pp. 228-251. [18] Shiau, C.-S., Tseng, I., Heutchy, A. and Michalek, J., 2007, “Design Optimization of a Laptop Computer Using Aggregated and Mixed Logit Demand Models with Consumer Survey Data,” Proceedings of the 33rd ASME Design Automation Conference, Las Vegas, NV, DETC2007/DAC-34883. [19] Chen, W., He, L., Hoyle, C.,Wang, J., and Yannou, B., 2010, “A Framework for Choice Modeling in Usage Context-Based Design,” Proceedings of the 36th ASME Design Automation Conference, Montreal, QU, DETC2010-28490. [20] Michalek, J., Feinberg, F., Ebbes, P., Adigüzel, F. and Papalambros, P., 2008, “Enhancing Marketing with Engineering: Optimal Product Line Design for Heterogeneous Markets,” International Journal of Research in Marketing, 28: pp. 1-12. [21] Donndelinger, J. A., Robinson, J. A., and Wissmann, L. A., 2008, “Choice Model Specification in Market-Based Engineering Design,” Proceedings of the 34th ASME

Another issue to be further developed is the pricing scheme of the products in this problem. Because of industry practice, the additive pricing method presented in the paper is the best method of representing price schemes for these products. However, future work can focus on developing an optimization process that finds the optimal price of each attribute level considering both market share and average contribution margin. Finally, in the market simulation of designed products, it is beneficial to investigate strategies for a staggered product launch. This will allow for designers to decide the best strategy for bringing the product line to market considering that they may not have the means to launch a large number of products simultaneously. Further, the addition of market competition to this simulation will increase its accuracy in representing an actual market. 8. ACKNOWLEDGMENTS We gratefully acknowledge support from the National Science Foundation through NSF Grant No. CMMI-0969961. Any opinions, findings, and conclusions presented in this paper are those of the authors and do not necessarily reflect the views of the National Science Foundation. 9. REFERENCES [1] Ferguson, S., and Donndelinger, J., 2010, “A Multidisciplinary Approach to Market Segmentation and Product Family Definition," 13th AIAA/ISSMO Multidisciplinary Analysis Optimization Conference, Fort Worth, TX, AIAA-2010-9228. [2] Donndelinger, J., and Fenyes, P., 2004, “Application of Math-Based Marketing and Financial Tools in an Automated Parametric Design Framework,” 2004 ASME IDETC, Salt Lake City, Utah, DETC2004 / DAC-57437. [3] Fenyes, P., Donndelinger, J., and Bourassa, J.- F., 2002, “A New System for Multidisciplinary Analysis and Optimization of Vehicle Architectures,” Proceedings of the 9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, Atlanta, GA, AIAA-2002-5509. [4] Cook, H. and DeVor, R., 1991, “On Competitive Manufacturing Enterprises I: The S-Model and the Theory of Quality,” Manufacturing Review, 4(2): pp. 96-105. [5] Donndelinger, J.A., Cook, H.E., 1997, “Methods for analyzing the value of automobiles,” SAE Special Publications, 1266: pp. 17-35. [6] Donndelinger, J., Ferguson, S., and Lewis, K., 2006, "Exploring Mass Trade-Offs in Preliminary Vehicle Design using Pareto Sets," 11th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Portsmouth, VA, AIAA-2006-7056. [7] Train, K.., 2003, Discrete Choice Methods with Simulation, Cambridge University Press, New York, NY. [8] Li, H. and Azarm, S., 2000, “Product Design Selection Under Uncertainty and With Competitive Advantage,” Journal of Mechanical Design, 122(4): pp. 411-8. [9] Li, H. and Azarm, S., 2002, “An Approach for Product Line Design Selection Under Uncertainty and

11


Design Automation Conference, Brooklyn, NY., DETC2008-50071. [22] Shiau, C.-S. and Michalek, J. J., 2008, "Should Designers Worry about Market Systems?" Journal of Mechanical Design, Vol. 131:011011: pp. 1-9. [23] Sawtooth Software, 2009, “The CBC/HB System for Hierarchical Bayes Estimation Version 5.0 Technical Paper,” Sawtooth Software, Inc., Sequim, WA, http://www.sawtoothsoftware.com/download/techpap/ hbtech.pdf. [24] Thevenot, H.J., and Simpson, T.W., 2006, “Commonality Indices for Product Family Design: A Detailed Comparison.” Journal of Engineering Design, 17(2): pp. 99-119. [25] Miller, S., 1999, “VW sows confusion with common pattern for models- investors worry profits may suffer as lines complete.” Wall Street Journal, p. A.25. [26] Martin, M.V., and Ishii, K., 1996, “Design for variety: a methodology for understanding the costs of product proliferation.” Proceedings of the ASME Design Engineering Technical Conferences, DETC1996/DTM1610. [27] Martin, M.V., and Ishii, K., 1997, “Design for variety: development of complexity indices and design charts.” Proceedings of the ASME Design Engineering Technical Conferences, DETC97/DFM-4359. [28] Sawtooth Software, 2008, “CBC v.6.0”, Sawtooth Software, Inc., Sequim, WA, http://www.sawtoothsoft ware.com/download/techpap/cbctech.pdf. [29] Sawtooth Software, 2004, “The CBC Latent Class Technical Paper, Version 3,” Sawtooth Software, Inc., Sequim, WA, http://www.sawtoothsoftware.com/ download/techpap/lctech.pdf. [30] Hauser, J. R., 1978, “Testing and Accuracy, Usefulness, and Significance of Probabilistic Choice Models: An Information-Theoretic Approach,” Operations Research, 26(May-June): pp. 406-421. [31] Ogawa, K., 1987, “An Approach to Simultaneous Estimation and Segmentation in Conjoint Analysis,” Management Science, 6(Winter): pp. 66-81. [32] Bozdogan, H., 1987, “Model Selection and Akaike‟s Information Criterion (AIC): The General Theory and its Analytical Extensions,” Psychometrika, 52: pp. 345-370. [33] Ramaswamy, V., DeSarbo, W. S., Reibstein, D. J., and Robinson, W. T., 1993, “An Empirical Pooling Approach for Estimating Marketing Mix Elasticities with PIMS Data,” Marketing Science, 12(Winter): pp. 103-124. [34] Turner, C., Ferguson, S., and Donndelinger, J., 2011, “Exploring Heterogeneity of Customer Preference to Balance Commonality and Market Coverage,” ASME 2011 International Design Engineering Technical Conferences, Washington, DC., DETC2011-48581. [35] Sawtooth Software, 2003, “Advanced Simulation Module (ASM) for Product Optimization Technical Paper, Version 1.5”, Sawtooth Software, Inc., Sequim, WA,

http://www.sawtoothsoftware.com/download/techpap/asmt ech.pdf. [36] Andrews, R. L., Ainslie, A., and Currim, I. S., 2002, “An empirical comparison of logit choice models with discrete versus continuous representations of heterogeneity,” Journal of Marketing Research, 39(Nov.): pp. 479-487

12
