An application of the exploratory structural equation

0 downloads 0 Views 652KB Size Report
and Muthén, 2004) with a robust weight least square (WLSMV) estima- tor. The WLSMV ... 36.98 8.63 42.80 7.69 t(390) = −14.39, p b 0.001 29.67%. Ag. ..... Stark, S., Chernyshenko, O. S., Chan, K. Y., Lee, W. C., & Drasgow, F. (2001). Effects of ...
Personality and Individual Differences 119 (2017) 220–226

Contents lists available at ScienceDirect

Personality and Individual Differences journal homepage: www.elsevier.com/locate/paid

An application of the exploratory structural equation modeling framework to the study of personality faking Philseok Lee a,⁎, Kevin T. Mahoney a, Sunhee Lee b a b

South Dakota State University, United States Chungnam National University, South Korea

a r t i c l e

i n f o

Article history: Received 11 May 2017 Received in revised form 19 July 2017 Accepted 21 July 2017 Available online xxxx Keywords: Personnel election Personality measure Faking Ideal employee factor Exploratory structural equation modeling Bi-factor exploratory structural equation modeling

a b s t r a c t This study compared the suitability of the restrictive framework of independent cluster model (ICM) and a more flexible framework of exploratory structural equation model (ESEM) to a personality instrument in a faking study. We proposed and answered five research questions using the model-testing procedures described by Morin, Arens, and Marsh (2016). More specifically, we compared the fit of ICM-CFA and ESEM, and ESEM and bi-factor ESEM, and we investigated the patterns of factor correlations and the presence of cross-loadings in these models. In our faking condition, we found the ESEM applications provided the better representation of the data, and the adverse effects of the strict assumptions of ICM-based models to be most apparent. Limitations and practical implications were discussed. © 2017 Elsevier Ltd. All rights reserved.

1. Introduction Over the last two decades, there has been growing interest in the use of personality measures for personnel selection. This interest has been stimulated by meta-analytic evidence that personality predicts important occupational outcomes (e.g., Barrick, Mount, & Judge, 2001; Berry, Ones, & Sackett, 2007). In addition, personality tests can provide incremental validity over general cognitive ability tests (Ployhart & Holtz, 2008), and have been shown to have little or no adverse impact (Hough, Oswald, & Ployhart, 2001). Despite the popularity of personality measures, faking, also termed socially desirable responding or impression management, has been a continuing concern in high-stakes settings (Griffith & Robie, 2013). When personality measures are used for selection decisions, the construct validity of the resulting scores is a critical issue (Ellingson, Smith, & Sackett, 2001). For example, if job applicants fake their responses, their personality scores reflect not only individual differences in personality constructs, but also faking behavior that is unrelated to the target constructs (Reeder & Ryan, 2011). Accordingly, many researchers suggest that faking has detrimental effects on the construct validity of personality measures due to inflation of factor inter-correlations and distortion of conceptual factor

⁎ Corresponding author. E-mail address: [email protected] (P. Lee).

http://dx.doi.org/10.1016/j.paid.2017.07.029 0191-8869/© 2017 Elsevier Ltd. All rights reserved.

structure (e.g., Ellingson, Sackett, & Hough, 1999; Montag & Comrey, 1990). As Stark, Chernyshenko, Chan, Lee, and Drasgow (2001) noted, consistent personality factor structure in both non-motivated samples (i.e., norm groups used in test development contexts) and motivated samples (i.e., job applicants in selection contexts) is crucial in order for a personality measure to have the desired-construct validity. In a seminal study, Schmit and Ryan (1993) performed a correlated confirmatory factor analysis (CFA) on the NEO Big Five-Factor Inventory and compared model fit between job applicant and student samples. They found that the five-factor model fits best for the student sample, but the six-factor model fit better for the job applicant sample. The authors argued that personality items were explained not only by respective Big 5 personality factors, but also by the sixth factor in the applicant sample, which they labeled the “ideal-employee factor”. This suggests engaging in faking responses changed the pattern of covariation among indicators, which undermined the construct validity by introducing extraneous variance from faking into personality trait measurement (Stark et al., 2001). To investigate the ideal employee factor, previous research typically used correlated CFA (e.g., Ellingson et al., 1999; Ellingson et al., 2001; Montag & Comrey, 1990) or the bi-factor model (e.g., Biderman, Nguyen, Cunnningham, & Ghorbani, 2011; Klehe et al., 2012; Ziegler & Buehner, 2009). In the bi-factor model, all items are allowed to load not only on their corresponding specific personality factors, but also on a global (G) factor representing the ideal-employee factor (see Fig.

P. Lee et al. / Personality and Individual Differences 119 (2017) 220–226

1, Model 3). Research has consistently found much stronger factor loadings on the ideal-employee factor and weaker factor loadings on the respective personality trait factors in motivated samples as compared to non-motivated samples (Biderman et al., 2011; Klehe et al., 2012; Ziegler & Buehner, 2009). Methodologically, previous faking research of the motivated test context has exclusively relied on the framework of independent clusters models (ICM) that posits that all cross-loadings between items and non-target factors are constrained to be zero. This approach ignores potential item cross-loadings derived from construct-irrelevant multidimensionality (e.g., faking). When cross-loadings are not modeled in ICM-CFA or bi-factor model, the construct-irrelevant relationships between items and non-target factors appear through the latent factor correlations or G factor loadings (Marsh et al., 2009; Morin, Arens, & Marsh, 2016). Consequently, the strict assumption of no cross-loadings yields more inflated factor correlations in the motivated testing context, which may obscure conceptual structure and meaning of latent construct, thereby diminishing the discriminant validity of a personality measure to an even greater degree. In addition, in instances where there are potential cross-loadings, applying the traditional bi-factor model results in more inflated factor variance accounted for by the ideal employee factor. Therefore, it may be misleading to rely exclusively on traditional measurement models for faking evidence in these settings. In this vein, new research is needed to examine whether less strict assumptions influence construct validity evidence of personality measures in the motivated testing context. Recently, exploratory structural equation model (ESEM; Asparouhov & Muthén, 2009) and bi-factor ESEM (B-ESEM; Morin, Arens, et al., 2016) have been developed to address the overly restrictive assumptions of ICM-CFA and the bi-factor model. Surprisingly, despite growing interest in the ESEM approach within the field of personality research (e.g., Booth & Hughes, 2014; Furnham, Guenole, Levine, & Chamorro-Premuzic, 2013; Marsh, Nagengast, & Morin, 2013; Marsh et al., 2010; Perry, Clough, Crust, Earle, & Nicholls, 2013; Rosellini & Brown, 2011), researchers have yet to apply the ESEM approach to a faking study. The current study proposed and answered five research

221

questions in order to examine the suitability of ESEM and B-ESEM as alternatives to CFA and bi-factor model for studying faking in response to self-report personality instruments. 1.1. The application of ESEM and B-ESEM to faking research ESEM allows all factor loadings to be freely estimated. Thus, all items load on their own target factor and cross-load on all other factors (see Fig. 1 Model 2). Accordingly, the ESEM application addresses two main concerns associated with the ICM-CFA approach; a) general lack of model fit in complex models and b) generally inflated factor correlations (Booth & Hughes, 2014). Thus, ESEM applications are useful in the motivated testing context as they enable researchers to control overinflated factor correlations and investigate construct-irrelevant associations between items and nontarget constructs due to faking. As an extended application of ESEM, B-ESEM (Morin, Arens, et al., 2016), a combination of ESEM and the bi-factor model, was recently developed (see Fig. 1 Model 4). B-ESEM applications explore two sources of “construct-relevant psychometric multidimensionality related to (a) the hierarchical nature of the constructs being assessed (i.e., the co-existence of global and specific components within the same measurement model) and (b) the infallible nature of indicators which tend to include at least some degree of association with non-target constructs” (Morin, Arens, et al., 2016, p. 30). B-ESEM application may be useful for faking research, as it allows researchers to assess not only construct-irrelevant associations, but also the co-existence of the ideal employee factor along with specific personality factors within the ESEM framework. That is, B-ESEM framework provides a more realistic evaluation of the ideal employee factor in the motivated testing context by freely estimating cross-loadings. 1.2. Research questions The current study aims to investigate whether ESEM-based model applications are more appropriate than ICM-based model applications

Fig. 1. Graphical representation of four measurement models. Note. Due to the space limit, F2 and F4 were not represented. G = global factor; F1–F5 = Big Five personality factors; X1–X60 = 60 items; curve lines between factors (F1–F5) represent factor covariances/correlations; bidirectional arrows for each factor represent factor variances; unidirectional arrows linked to items represent item uniqueness; solid unidirectional arrows between target factors and items represent factor loading; dotted unidirectional arrows between items and non-target factors represent cross-loadings.

222

P. Lee et al. / Personality and Individual Differences 119 (2017) 220–226

in the motivated testing context. Toward this purpose, we formulated five research questions as follows: (1) Is there more inflation of factor correlations manifested for the motivated testing context (referred to as faking condition in the present study) in ICM-CFA than ESEM? (2) Is there larger model fit improvement for ESEM than ICM-CFA in the motivated testing context than the non-motivated testing context (referred to as honest condition)? (3) In ESEM, are cross-loadings more apparent in the faking condition than in the honest condition? (4) In B-ESEM, are loadings for the ideal employee factor larger under the faking condition than the honest condition? (5) Under the faking condition, is the variance accounted for by the ideal employee factor more inflated in the bi-factor model than in B-ESEM? 2. Method 2.1. Participants 491 college students at a national university in South Korea completed a 60-item Big-5 personality Likert measure. Participants were given extra credit for their participation. Males were 49.9% and the average age was 20.94 years (SD = 2.35). 2.2. Measures and procedures A Likert-type Big Five personality measure was constructed using the International Personality Item Pool (IPIP; Goldberg, 1992). Specifically, the Korean version of 100 items (5-point Likert) was developed through the standard translation and back-translation procedure (Brislin, 1970); then 60 items measuring 5 dimensions were selected based on psychometric properties. A within-subject design was used. This approach is more appropriate for our purposes because it accounts for the variability of faking behavior among examinees and made it possible to classify fakers in the sample (Griffith, Chmielowski, & Yoshita, 2007). Half of the participants completed the personality test under “fake good” instructions in Time 1, and then four weeks later with “honest responding” instructions in Time 2. The other half began with “honest responding” instructions, and switched to “fake good” instructions after four weeks. Instructions for each condition were adapted from Mueller-Hanson, Heggestad, and Thornton (2006). For the honest condition, students were asked to answer by describing the way they truly see themselves and not how they wish to be seen by others. For the faking condition, students were asked to imagine that they are applying for their dream job. 2.3. Analytical strategies All measurement models were estimated using Mplus 7.2 (Muthén and Muthén, 2004) with a robust weight least square (WLSMV) estimator. The WLSMV is a robust estimator which does not assume normally distributed variables and provides the best option for modeling categorical ordered data (Brown, 2014). Oblique target rotation was used in ESEM to ensure cross-loadings were as close to zero as possible (Asparouhov & Muthén, 2009). Orthogonal bi-factor target rotation was used in B-ESEM to estimate specific and G factor loadings (Morin, Arens, et al., 2016). For model testing, we followed the procedure proposed by Morin, Arens, et al. (2016). First, we compared ICM-CFA and ESEM to investigate the presence of cross-loadings and the patterns of factor correlations. If ESEM provided better representation of the data, then we proceeded to model comparison between ESEM and B-ESEM to identify the presence of the ideal employee factor. For the model fit evaluation, we followed established fit interpretation guidelines (e.g., Hu & Bentler, 1999); N0.95 for the comparative fit index (CFI)

and Tucker-Lewis index (TLI), b 0.06 for root-mean-square error of approximation (RMSEA); and b 1.0 for weighted root-mean-square residual (WRMR). Following the guidelines for nested model comparisons (Chen, 2007), the simpler model was supported over the complex model when a change in CFI (Δ CFI) ≥ 0.005 and a change in RMSEA (ΔRMSEA) was ≥0.01. In order to examine whether the factor loadings were equivalent across samples from honest and faking conditions, we also conducted measurement equivalence/invariance (ME/I) tests (for details see Millsap & Yun-Tein, 2004). To determine the measurement invariance, goodness of fit statistics for comparable constrained and unconstrained models were examined, and if the results were not significantly different, then the measurement invariance is supported. For model difference testing, the Mplus DIFFTEST (MDΔχ2), ΔCFI (change in CFI relative to the preceding model), ΔTLI, and ΔRMSEA were examined. A p-value N 0.05 for the DIFFTEST indicates a non-significant difference between the models. ΔCFI value over 0.01, ΔTLI value over 0.01 and ΔRMSEA of 0.015 were used as cutoff values (Cheung & Rensvold, 2002; Morin, Boudrias, Marsh, Madore, & Desrumaux, 2016). For the evaluation of cross-loading pattern coefficients from ESEM, two criteria were considered following previous research (e.g., Myers, Martin, Ntoumanis, Celimli, & Bartholomew, 2014; Stenling, Ivarsson, Hassmén, & Lindwall, 2015). First, the standardized coefficients should be statistically significant (p b 0.05). Second, the absolute values of the standardized coefficients should be N0.20 (|λ| ≥ 0.20). To evaluate how much variance in personality scores was due to the ideal employee factor, omega hierarchical coefficient (ωH; for details, Reise, 2012) was computed by taking into account cross-loadings with non-target factors. ωH represents the variance accounted for by the ideal employee factor controlling for specific personality factors in the faking condition. 3. Results We first checked if the honest and faking conditions were successfully manipulated. The Table 1 showed significantly higher scale means for the faking condition than the honest condition across all five dimensions (p b 0.001). However, the proportions of fakers were lower in the current study as compared to previous research (e.g., Arthur, Glaze, Villado, & Taylor, 2010). Proportion of fakers were computed using Griffith et al.'s (2007) strategy (see Table 1). Table 2 displays the factor correlations of CFA and ESEM models in the honest and faking conditions. Stronger factor correlations between the Big Five dimensions were found for ICM-CFA than ESEM. In the honest condition, factor correlations ranged from − 0.35 to 0.30 for ICMCFA and from −0.22 to 0.25 for ESEM, respectively. In the faking condition, factor correlations ranged from −0.65 to 0.60 and from −0.42 to 0.47 for ICM-CFA and ESEM, respectively. The change in factor correlations between conditions was more apparent in ICM-CFA (average Δ r = 0.30) than in ESEM (average Δ r = 0.19). Thus, we

Table 1 Manipulation check with comparison between honest and faking conditions. Dimension

Op. Con. Ex. Ag. Neu.

Honest condition

Faking condition

Mean

SD

Mean

SD

42.42 36.85 36.98 43.24 34.71

7.32 7.68 8.63 5.63 8.74

45.12 45.87 42.80 46.28 29.01

6.58 8.59 7.69 5.58 7.69

Paired samples t-test

% of fakers

t(390) = −0.16, p b 0.001 t(390) = −18.73, p b 0.001 t(390) = −14.39, p b 0.001 t(390) = −11.27, p b 0.001 t(390) = 14.45, p b 0.001

19.95% 31.96% 29.67% 16.62% 19.43%

Note. Op. = Openness; Con. = Conscientiousness; Ex. = Extraversion; Ag. = Agreeableness; Neu. = Neuroticism % of fakers were computed using Griffith et al.'s (2007) strategy. If an individual's score from the faking condition exceed the upper bound of the 95% confidence interval (1.96 × SEM), the individual was classified as a faker. A standard error of measurement (SEM) was obtained from the honest condition.

P. Lee et al. / Personality and Individual Differences 119 (2017) 220–226 Table 2 Factor intercorrelations of CFA and ESEM model for both honest and faking samples. Conditions

Dimensions

Op.

Con.

Ex.

Ag.

Neu.

Honest

Openness Conscientiousness Extravertness Agreeableness Neuroticism Openness Conscientiousness Extravertness Agreeableness Neuroticism

– 0.10 0.12 0.11 0.01 – 0.40 0.41 0.41 −0.28

0.11 – 0.30 0.07 −0.31 0.27 – 0.60 0.50 −0.65

0.12 0.25 – 0.23 −0.35 0.32 0.47 – 0.51 −0.52

0.05 0.00 0.10 – −0.15 0.24 0.32 0.31 – −0.41

0.03 −0.20 −0.22 −0.04 – −0.10 −0.42 −0.33 −0.22 –

Faking

Note. Op. = Openness; Con. = Conscientiousness; Ex. = Extraversion; Ag. = Agreeableness; Neu. = Neuroticism. Factor correlations from ESEM model and ICM-CFA are displayed above and below the diagonal, respectively.

concluded more inflation of factor correlations manifested for the faking condition in ICM-CFA than ESEM (RQ1). Next, we examined whether fit improvement of ESEM over ICM-CFA is larger in the faking condition compared to the honest condition. Table 3 showed that ESEM fit the data better than ICM-CFA in both conditions, but the improvement of ESEM was larger in the faking condition (Δ CFI = 0.046, Δ TLI = 0.039, Δ RMSEA = 0.013) than in the honest condition (ΔCFI = 0.016, ΔTLI = 0.001, and ΔRMSEA = 0.001). Thus, we concluded that although ESEM provided better model fit than ICM-CFA in both conditions, the model fit improvement was more apparent in the faking condition (RQ2). Next, we proceeded to examine whether measurement invariance hold between the honest and the faking conditions in ESEM. The model of configural invariance provided a good model fit, χ2(3015) = 6481.37, RMSEA = 0.056, CFI = 0.924, TLI = 0.911. We then progressively tested metric invariance by constraining the factor loadings. Although ΔRMSEA indicates metric invariance (ΔRMSEA = 0.010), other results showed the metric invariance was not supported; MDΔχ2(324) = 580.03, p b 0.001, ΔCFI = 0.017, ΔTLI = 0.026. Accordingly, we concluded that metric invariance between the honest condition and the faking condition was not tenable in ESEM. Next, we evaluated how the patterns of cross-loadings in ESEM differed across honest and faking conditions (see Table 4). Under the honest condition, the ESEM solution showed 27 cross-loadings with significant (p b 0.05) and meaningful (| λ | ≥ 0.2) pattern coefficients. In contrast, under the faking condition, ESEM yielded nearly three times the number of cross-loadings (78 cross-loadings) with significant and meaningful pattern coefficients. In addition, 21 items were crossloaded under the honest condition, whereas 41 items were crossloaded under the faking condition. Thus, cross-loadings are much more evident in the faking condition than the honest condition for ESEM (RQ3). To answer research question 4, we first compared the fit of ESEM and B-ESEM. Table 3 showed that B-ESEM fit the data better than ESEM in both conditions; Δ CFI = 0.027, Δ TLI = 0.028, Δ RMSEA = 0.007 for the honest condition; ΔCFI = 0.018, ΔTLI = 0.019, ΔRMSEA = 0.008 Table 3 Model fit for four measurement models of Big Five personality measure. Condition

Model

Chi-square

df

RMSEA

CFI

TLI

WRMR

Honest

ICM-CFA ESEM Bifactor B-ESEM ICM-CFA ESEM Bifactor B-ESEM

3897.984 3379.892 3467.056 2838.208 4472.317 3026.051 3431.111 2491.277

1700.00 1480.00 1650.00 1425.00 1700.00 1480.00 1650.00 1425.00

0.058 0.057 0.053 0.050 0.065 0.052 0.053 0.044

0.879 0.895 0.900 0.922 0.896 0.942 0.933 0.960

0.874 0.875 0.892 0.903 0.892 0.931 0.928 0.950

1.911 1.244 1.712 1.034 1.862 1.023 1.476 0.852

Faking

Note. ICM-CFA = independent cluster model-confirmatory factor analysis; ESEM = exploratory structural equation model; B-ESEM = bifactor exploratory structural equation model.

223

for the faking condition. Additionally, we also found B-ESEM fit the data better than bi-factor in both conditions; Δ CFI = 0.022, Δ TLI = 0.011, Δ RMSEA = 0.003 for the honest condition; Δ CFI = 0.027, Δ TLI = 0.022, Δ RMSEA = 0.009 for the faking condition. Next, we proceeded to the measurement invariance test between the honest and the faking conditions for B-ESEM. The model of configural invariance provided a good model fit, χ 2 (2904) = 5396.67, RMSEA = 0.048, CFI = 0.945, TLI = 0.933. Next, we tested for metric invariance. Although ΔRMSEA indicates metric invariance (ΔRMSEA = 0.011), other fit indices indicated that metric invariance was not obtained; MDΔχ2 (324) = 580.03, p b 0.001, ΔCFI = 0.011, ΔTLI = 0.019. Thus, we concluded that metric invariance was not supported between the honest and faking conditions for B-ESEM, meaning factor loadings in B-ESEM were not equivalent across conditions. We further investigated whether loadings for the G factor (ideal employee factor) were larger in the faking condition in B-ESEM (see Table 5). An average of G factor loadings of all 60 items was | 0.26 | (min = |0.02|, max = |0.49|) in the honest condition. In contrast, the average of G factor loadings in the faking condition were | 0.48| (min = |0.06|, max = |0.75|). Although the magnitude of G factor loadings varied across personality dimensions as well as across items, we generally found much stronger loadings on the G factor and relatively weaker loadings on specific factors in the faking condition than in the honest condition. The results suggest the ideal employee factor is more apparent under the faking condition than the honest condition when B-ESEM is applied (RQ4). Finally, we examined whether the variance accounted for by the ideal employee factor was larger for B-ESEM than the bi-factor model in the faking condition. For B-ESEM, ωH was 0.60, which means approximately 60% of the personality scores were accounted for by the ideal employee factor after controlling specific factors. On the other hand, ωH was.83 for the bi-factor model. Thus, the variance explained by the ideal employee factor was larger in the traditional bi-factor model compared to B-ESEM (RQ5). To further examine the advantage of B-ESEM over bi-factor model in the faking condition, we calculated correlations of factor scores obtained from the bi-factor model and B-ESEM (see Table 6). High correlations between the two models in the honest condition (0.91 to 0.99) indicate factor scores are stable regardless of inclusion of cross-loadings. In contrast, the correlations were much lower (0.46 to 0.69) in the faking condition, indicating this is where allowing cross-loadings made the most difference. Taken together, the findings demonstrate that the benefits of B-ESEM over the bi-factor model are more apparent in the faking condition compared to the honest condition. 4. Conclusion and discussion ESEM applications are expanding considerably in personality research, but no attempt has been made to use ESEM to study faking. We believe this research is the first study of faking using ESEM and BESEM. In the answers to our five research questions, we found the flexible exploratory structural equation model framework to be more suitable for studying faking in response to a personality measure, than the more traditional framework of independent cluster models. Specifically, inflation of factor correlations for the faking condition compared to the honest condition was larger in ICM-CFA than ESEM (RQ1). Also, model fit improvement of ESEM over ICM-CFA was larger in the faking condition than in the honest condition (RQ2). In ESEM, cross-loadings were more apparent in the faking condition than the honest condition (RQ3). In B-ESEM, loadings for the ideal employee factor were larger under the faking condition than the honest condition (RQ4). For the faking condition, the variance accounted for by the ideal employee factor was larger in the bi-factor model than in B-ESEM (RQ5). Although the body of faking research accumulated thus far has certainly provided us with useful information, the research domain has exclusively relied on traditional measurement models that have highly

224

P. Lee et al. / Personality and Individual Differences 119 (2017) 220–226

Table 4 Standardized factor loadings of ESEM across honest and faking samples. Items

Item1 Item2 Item3 Item4 Item5 Item6 Item7 Item8 Item9 Item10 Item11 Item12 Item13 Item14 Item15 Item16 Item17 Item18 Item19 Item20 Item21 Item22 Item23 Item24 Item25 Item26 Item27 Item28 Item29 Item30 Item31 Item32 Item33 Item34 Item35 Item36 Item37 Item38 Item39 Item40 Item41 Item42 Item43 Item44 Item45 Item46 Item47 Item48 Item49 Item50 Item51 Item52 Item53 Item54 Item55 Item56 Item57 Item58 Item59 Item60

Openness Honest

Faking

0.75 0.72 0.54 0.72 0.77 0.53 0.40 0.69 0.79 0.61 0.54 0.48

0.80 0.69 0.43 0.60 0.76 0.49 0.39 0.62 0.82 0.57 0.52 0.49

Conscientiousness

Extraversion

Honest

Honest

Faking

Agreeableness Faking

0.36

0.27 0.22

0.22 0.35

0.28 0.20

Neuroticism

Honest

Faking

0.35

0.18

Honest

Faking 0.24 −0.23

0.21 −0.24

0.24 0.59 0.74 0.76 0.58 0.81 0.68 0.72 0.85 0.61 0.72 0.56 0.59

0.21 0.26 0.31 0.78 0.84 0.88 0.74 0.86 0.76 0.80 0.87 0.74 0.73 0.70 0.67 0.24 0.20 0.31 0.45 0.27 0.28 0.33 0.33 0.28 0.32 0.24 0.29

0.31 0.22

0.27

0.24 0.22

0.21

0.34

0.20 0.24 0.57 0.89 0.78 0.64 0.80 0.90 0.84 0.64 0.61 0.48 0.56 0.77

0.55 0.84 0.72 0.58 0.77 0.77 0.77 0.67 0.42 0.45 0.51 0.75 0.24 0.24

0.22

0.25

0.36

0.23

0.23 0.24

0.38

−0.23 −0.22

0.20 0.20 0.21

0.36

0.27 −0.38 −0.40 −0.41 −0.38 −0.21 −0.41 −0.35

−0.24

0.34 0.22 0.37

−0.20

−0.25 −0.20

0.26

−0.32 −0.21 −0.20

−0.44

−0.20 −0.21

−0.26 −0.22 −0.22

−0.26 −0.26 −0.26 −0.22

0.19

0.21

−0.20 0.20 0.61 0.61 0.54 0.46 0.41 0.61 0.40 0.35 0.65 0.58 0.41 0.55

0.23 −0.31 0.24

0.61 0.60 0.46 0.43 0.46 0.60 0.45 0.49 0.73 0.67 0.39 0.69

−0.42

0.21 −0.20

−0.23 −0.21 −0.22

−0.24

−0.24 −0.20

−0.27

−0.21

0.20

0.22

0.22 −0.20

0.86 0.87 0.70 0.70 0.28 0.67 0.79 0.42 0.76 0.61 0.70 0.58

−0.22 0.83 0.78 0.65 0.64 0.24 0.59 0.68 0.43 0.68 0.48 0.59 0.46

Bold in the table represents specific factor loadings corresponding each dimension.

restrictive assumptions. Our study reinforces the earlier findings that construct irrelevant multidimensionality is far more apparent in the motivated testing context. In that case, conventional applications that do not account for potential cross-loadings would result in additional inflation in both factor correlations and the variance accounted for by the ideal employee factor. In this vein, ESEM applications are particularly useful for the motivated testing context, because they enable researchers to control overinflated factor correlations, evaluate construct-irrelevant multidimensionality due to faking. Additionally,

B-ESEM applications allow researchers to examine the co-existence of the ideal employee factor along with specific factors while taking into account cross-loadings. Therefore, the current findings suggest that the ESEM and B-ESEM applications contribute more realistic discriminant validity evidence and may provide a better idea of the way the ideal employee factor functions in real-world settings. Although they are not the focus of our study, we'd like to address two unexpected findings. First, items still load on G factor in B-ESEM even in the honest condition (mean G factor loading = | 0.26 |).

P. Lee et al. / Personality and Individual Differences 119 (2017) 220–226 Table 5 B-ESEM standardized factor loadings of ideal employee factor across samples. Items

Item1 Item2 Item3 Item4 Item5 Item6 Item7 Item8 Item9 Item10 Item11 Item12 Item13 Item14 Item15 Item16 Item17 Item18 Item19 Item20 Item21 Item22 Item23 Item24 Item25 Item26 Item27 Item28 Item29 Item30 Item31 Item32 Item33 Item34 Item35 Item36 Item37 Item38 Item39 Item40 Item41 Item42 Item43 Item44 Item45 Item46 Item47 Item48 Item49 Item50 Item51 Item52 Item53 Item54 Item55 Item56 Item57 Item58 Item59 Item60

225

Table 6 Correlations of factor scores derived from bi-factor and bifactor ESEM.

Conditions

Dimension

Honest

Faking

0.16 0.43 0.24 0.33 0.41 0.33 0.18 0.00 0.06 0.07 0.26 0.11 0.38 0.26 0.31 0.36 0.27 0.15 0.31 0.30 0.18 0.19 0.17 0.16 0.41 0.37 0.39 0.38 0.43 0.40 0.49 0.41 0.26 0.25 0.26 0.29 0.11 0.31 0.09 0.12 0.49 0.38 0.43 0.16 0.26 0.29 0.11 0.23 −0.39 −0.39 −0.17 −0.13 −0.05 −0.13 −0.25 0.07 −0.14 −0.43 −0.44 −0.33

0.06 0.44 0.55 0.41 0.30 0.52 0.45 0.19 0.27 0.24 0.44 0.44 0.70 0.68 0.69 0.75 0.63 0.58 0.69 0.60 0.65 0.58 0.51 0.52 0.59 0.55 0.59 0.60 0.61 0.58 0.62 0.64 0.36 0.41 0.52 0.57 0.30 0.55 0.59 0.41 0.57 0.51 0.54 0.44 0.17 0.17 0.22 0.38 −0.59 −0.59 −0.46 −0.48 −0.29 −0.44 −0.44 −0.06 −0.41 −0.60 −0.54 −0.56

Although one cannot conclusively say what these G factor loadings in the honest condition reflect, they may reflect a response bias. One such response bias is a self-deception, an unintentional tendency to respond in a socially desirable manner (Tett & Simonet, 2011). Nevertheless, the current findings in the faking condition provided convincing evidence for the “ideal employee factor,” with Conscientiousness items having the highest loadings (λConscientiousness ¼ :63Þ , Openness items having the lowest (λOpenness ¼ :36Þ, and Agreeableness items falling in between (λAgreeableness ¼ :40 ). Although further research with

Openness Conscientiousness Extraversion Agreeableness Neuroticism

Bi-factor vs. B-ESEM Honest condition

Faking condition

0.99⁎⁎ 0.95⁎⁎ 0.91⁎⁎ 0.94⁎⁎ 0.94⁎⁎

0.68⁎⁎ 0.45⁎⁎ 0.65⁎⁎ 0.57⁎⁎ 0.62⁎⁎

⁎⁎ p b 0.001.

actual job applicants is recommended, we note that this pattern of results is consistent with previous studies (e.g., Birkeland, Manson, Kisamore, Brannick, & Smith, 2006; Holden, Wood, & Tomashewski, 2001). Secondly, we found less distortion of the Big-Five ICM-CFA model fit in the faking condition than in previous studies (e.g., Schmit & Ryan, 1993). A possible reason could be fewer fakers in the current study as compared to some previous studies. Kim (2011) suggested that faking “alters the factor structure of content tests if enough applicants fake enough scale items” (p. 258). Thus the proportion of fakers is a significant issue for examining factor structure. Although there were significant mean differences between honest and faking conditions, we found less fakers (about 10% less) in our study as compared to previous research (Arthur et al., 2010; Griffith & Robie, 2013). However, as of yet, no hard and fast criteria has been established as to the effect the proportion of fakers has on the Big Five factor structure in the motivated test condition. This seems a topic ripe for further research. In closing, this research presented promising alternative measurement models (ESEM and B-ESEM) over the traditional measurement models to investigate the effect of faking on the factor structure of personality tests. We suspect ESEM and B-ESEM applications may emerge as even more advantageous when there is a greater prevalence of faking, as in a real-world setting. We hope that practitioners and researchers are aware that having strict measurement assumptions (as in traditional measurement models) can lead to misleading validity evidence of personality measures as well as overestimating the magnitude of faking in high-stakes settings. In order to control for such limitations, applying less restrictive measurement models (ESEM and B-ESEM) may be a more suitable strategy for motivated testing contexts. References Arthur, W., Glaze, R. M., Villado, A. J., & Taylor, J. E. (2010). The magnitude and extent of cheating and response distortion effects on unproctored internet-based tests of cognitive ability and personality. International Journal of Selection and Assessment, 18, 1–16. Asparouhov, T., & Muthén, B. (2009). Exploratory structural equation modeling. Structural Equation Modeling, 16, 397–438. Barrick, M. R., Mount, M. K., & Judge, T. A. (2001). Personality and performance at the beginning of the new millennium: What do we know and where do we go next? International Journal of Selection and Assessment, 9, 9–30. Berry, C. M., Ones, D. S., & Sackett, P. R. (2007). Interpersonal deviance, organizational deviance, and their common correlates: A review and meta-analysis. Journal of Applied Psychology, 92, 410–424. Biderman, M. D., Nguyen, N. T., Cunnningham, C. J. L., & Ghorbani, N. (2011). The ubiquity of common method variance: The case of the Big Five. Journal of Research in Personality, 45, 417–429. Birkeland, S. A., Manson, T. M., Kisamore, J. L., Brannick, M. T., & Smith, M. A. (2006). A meta-analytic investigation of job applicant faking on personality measures. International Journal of Selection and Assessment, 14, 317–335. Booth, T., & Hughes, D. J. (2014). Exploratory structural equation modeling of personality data. Assessment, 21, 260–271. Brislin, R. (1970). Back translation for cross-cultural research. Journal of Cross-Cultural Psychology, 1, 185–216. Brown, T. A. (2014). Confirmatory factor analysis for applied research. Guilford Publications. Chen, F. (2007). Sensitivity of goodness of fit indices to lack of measurement invariance. Structural Equation Modeling, 14, 464–504. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233–255. Ellingson, J. E., Sackett, P. R., & Hough, L. M. (1999). Social desirability corrections in personality measurement: Issues of applicant comparison and construct validity. Journal of Applied Psychology, 84, 155–166.

226

P. Lee et al. / Personality and Individual Differences 119 (2017) 220–226

Ellingson, J. E., Smith, D. B., & Sackett, P. R. (2001). Investigating the influence of social desirability on personality factor structure. Journal of Applied Psychology, 86, 122–133. Furnham, A., Guenole, N., Levine, S. Z., & Chamorro-Premuzic, T. (2013). The NEO Personality Inventory–Revised: Factor structure and gender invariance from exploratory structural equation modeling analyses in a high-stakes setting. Assessment, 20, 14–23. Goldberg, L. R. (1992). The development of markers for the Big-Five factor structure. Psychological Assessment, 4, 26–42. Griffith, R. L., Chmielowski, T., & Yoshita, Y. (2007). Do applicants fake? An examination of the frequency of applicant faking behavior. Personnel Review, 36, 341–355. Griffith, R. L., & Robie, C. (2013). Personality testing and the “f-word”: Revisiting seven questions about faking. In N. Christiansen, & R. Tett (Eds.), Handbook of personality at work (pp. 253–280). New York: Taylor & Francis. Holden, R. R., Wood, L. L., & Tomashewski, L. (2001). Do response time limitations counteract the effect of faking on personality inventory validity? Journal of Personality and Social Psychology, 81, 160–169. Hough, L. M., Oswald, F. L., & Ployhart, R. E. (2001). Determinants, detection and amelioration of adverse impact in personnel selection procedures: Issues, evidence and lessons learned. International Journal of Selection and Assessment, 9, 152–194. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. Kim, B. H. (2011). Deception and applicant faking: Putting the pieces together. In G. P. Hodgkinson, & J. K. Ford (Eds.), International review of industrial and organizational psychology, Vol. 26. (pp. 239–292). Chichester: Wiley-Blackwell. Klehe, U. C., Kleinmann, M., Hartstein, T., Melchers, K. G., König, C. J., Heslin, P. A., & Lievens, F. (2012). Responding to personality tests in a selection context: The role of the ability to identify criteria and the ideal-employee factor. Human Performance, 25, 273–302. Marsh, H. W., Lüdtke, O., Muthén, B., Asparouhov, T., Morin, A. J., Trautwein, U., & Nagengast, B. (2010). A new look at the big five factor structure through exploratory structural equation modeling. Psychological Assessment, 22, 471–491. Marsh, H. W., Muthén, B., Asparouhov, T., Lüdtke, O., Robitzsch, A., Morin, A. J., & Trautwein, U. (2009). Exploratory structural equation modeling, integrating CFA and EFA: Application to students' evaluations of university teaching. Structural Equation Modeling, 16, 439–476. Marsh, H. W., Nagengast, B., & Morin, A. J. (2013). Measurement invariance of big-five factors over the life span: ESEM tests of gender, age, plasticity, maturity, and la dolce vita effects. Developmental Psychology, 49, 1194–1219. Millsap, R. E., & Yun-Tein, J. (2004). Assessing factorial invariance in ordered-categorical measures. Multivariate Behavioral Research, 39, 479–515. Montag, I., & Comrey, A. L. (1990). Stability of major personality factors under changing motivational conditions. Journal of Social Behavior and Personality, 5, 253–262. Morin, A. J., Arens, A. K., & Marsh, H. W. (2016). A bi-factor exploratory structural equation modeling framework for the identification of distinct sources of construct-

relevant psychometric multidimensionality. Structural Equation Modeling: A Multidisciplinary Journal, 23, 116–139. Morin, A. J., Boudrias, J. S., Marsh, H. W., Madore, I., & Desrumaux, P. (2016). Further reflections on disentangling shape and level effects in person-centered analyses: An illustration exploring the dimensionality of psychological health. Structural Equation Modeling: A Multidisciplinary Journal, 23, 438–454. Mueller-Hanson, R. A., Heggestad, E. D., & Thornton, G. C. (2006). Individual differences in impression management: An exploration of the psychological processes underlying faking. Psychology Science, 48, 288–312. Muthén, L., & Muthén, B. (1998–2014). Mplus 7.2. Los Angeles, CA: Muthén & Muthén (computer software). Myers, N. D., Martin, J. J., Ntoumanis, N., Celimli, S., & Bartholomew, K. J. (2014). Exploratory bi-factor analysis in sport, exercise, and performance psychology: A substantive-methodological synergy. Sport Exercise and Performance Psychology, 3, 258–272. Perry, J. L., Clough, P. J., Crust, L., Earle, K., & Nicholls, A. R. (2013). Factorial validity of the Mental Toughness Questionnaire-48. Personality and Individual Differences, 54, 587–592. Ployhart, R. E., & Holtz, B. C. (2008). The diversity–validity dilemma: Strategies for reducing racioethnic and sex subgroup differences and adverse impact in selection. Personnel Psychology, 61, 153–172. Reeder, M. C., & Ryan, A. M. (2011). Methods for correcting for faking. In M. Ziegler, C. MacCann, & R. D. Roberts (Eds.), New perspectives on faking in personality assessment (pp. 131–150). New York, NY: Oxford University Press. Reise, S. P. (2012). The rediscovery of bi-factor measurement models. Multivariate Behavioral Research, 47, 667–696. Rosellini, A. J., & Brown, T. A. (2011). The NEO Five-Factor Inventory: Latent structure and relationships with dimensions of anxiety and depressive disorders in a large clinical sample. Assessment, 18, 27–38. Schmit, M. J., & Ryan, A. M. (1993). The Big Five in personnel selection: Factor structure in applicant and nonapplicant populations. Journal of Applied Psychology, 78, 966–974. Stark, S., Chernyshenko, O. S., Chan, K. Y., Lee, W. C., & Drasgow, F. (2001). Effects of the testing situation on item responding: Cause for concern. Journal of Applied Psychology, 86, 943–953. Stenling, A., Ivarsson, P., Hassmén, P., & Lindwall, M. (2015). Using bi-factor exploratory structural equation modeling to examine global and specific factors in measures of sports coaches' interpersonal styles. Frontiers in Psychology, 6, 1–12. Tett, R. P., & Simonet, D. V. (2011). Faking in personality assessment: A “multisaturation” perspective on faking as performance. Human Performance, 24, 302–321. Ziegler, M., & Buehner, M. (2009). Modeling socially desirable responding and its effects. Educational and Psychological Measurement, 69, 548–565.