Validity Evidence for a Croatian Version of the Conditional Reasoning ...

International Journal of Selection and Assessment

Volume 22 Number 4 December 2014

Validity Evidence for a Croatian Version of the Conditional Reasoning Test for Aggression Zvonimir Galic´*, Kelly T. Scherer** and James M. LeBreton*** *Department of Psychology, University of Zagreb, Luciceva 3, 10 000 Zagreb, Croatia. [email protected] **Purdue University, West Lafayette, IN, USA ***Penn State University, State College, PA, USA

The Conditional Reasoning Test for Aggression (CRT-A) is based on the idea that aggressive individuals use motive-based cognitive biases to see their behavior as reasonable and that those biases can be measured with inductive reasoning tasks. Although the initial validation efforts for the CRT-A in the United States have been reasonably successful, there has been no attempt to determine if the evidence of validity and reliability generalizes to other cultural contexts. In this paper, we describe four studies designed to systematically accumulate validity evidence for the CRT-A using Croatian participants. Our analyses revealed that the Croatian adaptation of the CRT-A yielded psychometric characteristics that were similar to those obtained on the US samples (Study 1). CRT-A scores that predicted counterproductive work behaviors occurrence beyond self-reported personality (Study 2) were independent from general mental ability as measured with an abstract reasoning test (Study 3), and not susceptible to faking (Study 4).

1. Introduction

J

bs_bs_banner

ames and his associates (James et al., 2005; James & LeBreton, 2010, 2012) recently introduced the Conditional Reasoning Test for Aggression (CRT-A) that assesses aspects of implicit personality and has important implications for understanding and predicting counterproductive work behaviors (CWBs; Wu & LeBreton, 2011). The test is based on the assumption that individuals with a strong motive to aggress reason differently than the majority of people who have socially adaptive values and prosocial ideologies. The reasoning of aggressive individuals is characterized by frequent use of justifications that help them to resolve the conflict between the motive to aggress (i.e., the motive to harm) and the motive to hold a favorable view of self. James and colleagues (James et al., 2005; James & LeBreton, 2010, 2012) identified six justification mechanisms that they believed aggressive individuals use to enhance the logical appeal of aggressive behaviors. These mechanisms and their definitions are listed in Table 1. James and his associates suggested that these justification mechanisms are essentially cog-

nitive biases that impact how information is perceived, processed, and used to make decisions. Consequently, they argued that it is possible to measure these motive-based cognitive biases using a special form of inductive reasoning task that they called conditional reasoning (see James & LeBreton, 2010, 2012 for example items). In the review of the studies conducted on large samples of US respondents, James and LeBreton (2012) reported satisfactory levels of internal consistency and test–retest stability for CRT-A scores. Moreover, their studies garnered support for the construct and convergent validity of the CRT-A. The factor analysis of tasks revealed a structure coherent with the theory, and CRT-A scores did not show significant relationships with cognitive abilities or self-reported aggression (James & LeBreton, 2010, 2012). Importantly, LeBreton, Barksdale, Robin, and James (2007) showed that the CRT-A is resistant to faking when the indirect nature of measurement is preserved. Finally, the CRT-A scores predicted aggressive behaviors/CWBs in both laboratory and field settings. Meta-analytical estimates of the uncorrected validities of CRT-A scores in predicting ag-

© 2014 John Wiley & Sons Ltd, 9600 Garsington Road, Oxford, OX4 2DQ, UK and 350 Main St., Malden, MA, 02148, USA

344

Zvonimir Galic´, Kelly T. Scherer and James M. LeBreton

Table 1. Justification mechanism for motive to aggress (James & LeBreton, 2012) Justification mechanism

Definition

Hostile attribution A tendency to see hostile and malevolent bias intention in the behavior of others even when that behavior is friendly Derogation of A tendency to derogate the targets of target bias aggression as evil, immoral, and untrustworthy Potency bias A tendency to see interactions with others as a contest for dominance that pit the strong/brave against the weak/cowardly Retribution bias A tendency to favor revenge and retaliation over acts of reconciliations Victimization by A tendency of aggressive individuals to powerful see themselves as victims of powerful others who exploit and oppress them Social discounting A tendency to perceive social norms as bias repressive and restrictive of free will

gressive behaviors differed between sources. Whereas first estimates indicated high validities (.44; James et al., 2005), recent meta-analyses revealed more modest numbers (validity of CRT-A scores in predicting CWBs was .28 in James & LeBreton, 2012, and .16 in Berry, Sackett, & Tobares, 2010; .08 in Banks, Kepes, & McDaniel, 2012). The variability in these meta-analytic estimates is driven largely by differences in the subjective decision each team of researchers made in conducting their meta-analysis (Wanous, Sullivan, & Malinak, 1989). Nevertheless, across these meta-analyses, there is evidence that scores on the CRT-A may be predictive of important organizational criteria (e.g., CWBs).

1.1 The present study The work on the conditional reasoning approach to measurement of aggression implies that motive-based cognitive biases that aggressive individuals use to see their behavior as reasonable and logical are universal and should operate in any cultural context. But, international research on the CRT-A is scarce. For example, all three meta-analyses that tested the usefulness of this approach to personality assessment relied exclusively on the US samples (Banks et al., 2012; Berry et al., 2010; James & LeBreton, 2012). The purpose of the current study is to explore the extent to which the justification mechanisms identified by James and associates may generalize to a substantially different societal context. Indeed, it is quite possible that certain justification mechanisms are not operative in certain cultures. For example, one might argue that the victimization by powerful others bias or the social discounting bias (i.e., the proclivity to frame social

International Journal of Selection and Assessment Volume 22 Number 4 December 2014

norms as repressive and restrictive of free will) might not work as justification mechanisms for aggressive behaviors in societies that are less libertarian and more collectivistic than the United States. In less individualistic societies with larger power distances (Hofstede, 2001), unequal distribution of power is expected and therefore these biases may not be used to justify aggressive behaviors. Moreover, the selection of justification mechanisms for aggression might depend on cultural experiences with aggressive behaviors. For example, in countries with a recent broad experience of aggression (e.g., due to a recent war), the hostile attribution bias might to a certain extent reflect a veridical perception of social interactions and not a justification mechanism used by aggressive individuals. Dependence of the CRT-A on cultural experience could seriously limit its usefulness in international contexts. In our study, we tested the CRT-A’s validity in Croatia. We believe that Croatia represents an interesting context for examining the CRT-A’s assumptions at least for two reasons. First, Croatia represents a different cultural context characterized by lower individualism and higher power distance than the United States (Hofstede, 2001). Second, the experience of war (1991– 1995) and the difficult transition from a socialistic to a free-market country (Tanner, 2010) gave Croatian citizen an extensive (and compared with US samples, a unique) experience with acts of aggression (and the accompanying justifications for those acts). In this paper, we report the results of the four studies where data were collected using a Croatian version of the CRT-A. In the first study, we explored the psychometric evidence (i.e., reliability estimates, item total correlations, and factor structure) of the Croatian adaptation of CRT-A in a large heterogeneous sample of Croatian respondents. In the second study, we explored the relationship between CRT-A scores and scores on a measure of CWBs, and we also tested whether scores on the CRT-A provided incremental prediction of CWBs above and beyond commonly used self-report personality inventories. In the third study, we examined the relationship between scores on the CRT-A, and scores on a test of general mental ability (GMA). Finally, in the fourth study, we examined whether scores on the CRT-A changed with demands for positive presentations by comparing scores across applicant, incumbent, and student samples faking good.

2. Study 1 Within the first study, we tested psychometric properties of the CRT-A problems with data collected from a heterogeneous sample of participants consisting of students, incumbents, and personnel selection applicants. Per the recommendations of James and McIntyre (2000),

© 2014 John Wiley & Sons Ltd

Conditional Reasoning Test for Aggression all respondents were instructed that they were solving a reasoning test and to complete the testing battery to the best of their ability.

2.1. Method 2.1.1. Participants In total, data obtained on 988 respondents were included in the study (642 students, 196 incumbents, and 150 applicants in an actual selection situation). In accordance to the test manual instructions (James & McIntyre, 2000), participants who endorsed five or more illogical answers were omitted from the database. In total 21 of 988 participants were omitted (2.22%). Thus, the factor analyses were conducted on the data from 967 participants. The participants were 47.7% male, with an average age of 26.28 (standard deviation [SD] = 9.05). The final sample size, gender structure, and average age according to sample type were: students (n = 628; 46.4% male, Mage = 21.91, SDage = 3.61), incumbents (n = 192; 46.7% male, Mage = 36.61, SDage = 11.88), and applicants (n = 147) 54.4% male, Mage = 30.93, SDage = 7.17). 2.1.2. Instrument The CRT-A consists of 25 inductive reasoning problems – 22 of these problems are conditional reasoning problems designed to reveal the justification mechanisms associated with aggression in a respondent’s reasoning (James & McIntyre, 2000; James & LeBreton, 2010, 2012). Each of these 22 problems has four response options: an inductively logical aggressive response, an inductively logical response based on socially adaptive ideology and reasoning, and two illogical responses. Three of the test problems (Problems 1, 2, and 6) are ‘classic’ inductive reasoning problems containing a single inductively valid solution. These three items are not scored as part of the CRT-A, but instead are included early in the test to further enhance its face validity as a reasoning test and thus maximize the indirect nature of assessment. 2.1.3. Translation and adaptation process The translation and adaptation of the CRT-A to Croatian was consistent with the recommendation for a cross-cultural test adaptation (Geisinger, 1994; Hui & Triandis, 1985). Three Croatian researchers proficient in English independently translated the CRT-A into Croatian. Considering that all translators judged that some information included in the CRT-A problems were culture specific, certain adaptations had to be made in the Croatian version. Most of the changes were minor and related to changes in the names of individuals and places from American names/places to Croatian


345 names/places. Only the problem exploring reasons of recent improvements of the American car industry underwent a major change such that ‘American car industry’ was replaced with ‘Croatian industry of refrigerators’ in order to appeal to Croatian respondents. The response options for this problem in the Croatian version were formulated so that the same aggression justification mechanisms would be operative as were in the original problem. The adapted version of the CRT-A was then reviewed by two psychologists experienced in personality assessment and familiar with the conditional reasoning approach. The Croatian version of the CRT-A was finalized after incorporating a few minor suggestions raised by these two psychologists. 2.1.4. Procedure In all situations the CRT-A was administered according to the instructions provided in the test manual (i.e., as an inductive reasoning test; James & McIntyre, 2000). Data from students were collected during a regular class meeting. The students were motivated to complete the study package with the information that they would have the opportunity to gain practice by completing the employment selection tests commonly used by organizations to select graduate students for entry-level positions. The package contained various psychological instruments, including the CRT-A. Incumbents were recruited by students from an I/O psychology course who, as part of an extra credit assignment, were asked to identify individuals who worked more than 6 months, for 20 or more hours per week. The students were instructed to give the survey package to participants, who fit the above description, at their homes, and to make sure participants completed the CRT-A within the 25-min time limit. Both participants and students received instructions that the CRT-A represents an ordinary reasoning test, and were not aware of its measurement objective during the data collection period. The applicants were recruited in two mid-sized Croatian companies. In both firms, the CRT-A was administered as a part of a selection test battery for professional positions related to sales or marketing (e.g., the largest number of applicants was seeking employment as brand/product managers, key account managers, or marketing specialists).

2.2. Results and discussion 2.2.1. Descriptive statistics Average item ‘difficulties’ and item total correlations are reported in Table 2. The proportion of respondents who selected the aggressive answer varied between .03 (‘bosses and employees’) and .60 (‘good product at low price’). The average proportion of aggressive responses



Late for meetings Aggressively going after customers Generals An eye for an eye Bosses and employees New technology and workplace Girl scouts and boy scouts A homeless man Good product at a low price Duels with swords A new girl at the high school Permits to carry guns American cars (adapted to Croatian fridges) Store employees versus shoplifters Bonuses for employees Search on employees Gangs Wild animals Hold up victims Divorces Employee’s revenge Agreement between countries

3 4 5 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

.21 .19 .11 .05 .03 .22 .26 .10 .60 .13 .53 .12 .29 .15 .20 .22 .34 .14 .15 .29 .33 .26

p .26 .32 .40 .45 .43 .36 .35 .47 .36 .41 .41 .31 .46 .41 .45 .46 .42 .32 .24 .25 .52 .50

rb

Item characteristics F2 .23 .14 .12 .30 .30 .14 .42 −.09 .00 .70 .17 .07 −.13 −.02 .01 .08 .02 .49 −.21 −.16 .02 .02

F1 −.05 .05 .33 .30 .41 .14 −.12 .64 .06 .02 .09 .19 .41 .38 .43 .31 .38 −.12 .28 .25 .45 .49

Factor saturation

a

Note: aThe largest absolute structure correlations are given in bold. p = proportion of aggressive responses; rb = biserial correlation coefficient between item response and total score on the CRT-A.

Items

Problem no.

Our study (n = 967)

.31 .21 .05 .05 .07 .35 .06 .19 .37 .21 .26 .15 .13 .19 .04 .29 .22 .22 .08 .28 .14 .04

p

.35 .38 .41 .41 .38 .41 .51 .42 .49 .45 .49 .39 .39 .41 .46 .45 .45 .42 .24 .37 .46 .46

rb

Item characteristics

.04 .12 .38 .42 .30 .04 .45 .32 .19 .11 .28 .29 .48 .29 .56 .25 .36 .11 .44 .24 .53 .59

F1

.05 .21 .49 .63 .37 .19 .43 .13 .10 .43 .19 .22 .12 .22 .34 .23 .14 .35 .18 .17 .27 .40

F2

.23 .09 .08 −.05 .16 .28 .48 .23 .40 .20 .29 .06 .10 −.01 .25 .17 .25 .18 .05 −.09 .13 .11

F3

Factor saturationsa

James and LeBreton (2012; n = 4772)

Table 2. Comparison of psychometric characteristics for the items of the Conditional Reasoning Test for Aggression (CRT-A) obtained on the Croatian and US samples: item difficulties, item total correlations, and factor saturations

346 Zvonimir Galic´, Kelly T. Scherer and James M. LeBreton


Conditional Reasoning Test for Aggression was .22, slightly higher than the one reported on the US sample (.18; James & LeBreton, 2012). For only two problems (the ‘good product at low price’ [p = .60] and the ‘a new girl at the high school’ [p = .53]), the proportion of respondents who selected the aggressive alternative exceeded .50. Item total correlations calculated as biserial correlation coefficients (Lord & Novick, 1968) ranged between .24 (‘hold up victims’) and .52 (‘employee revenge’). The average biserial coefficient in our study was .39, which was very similar to the .42 reported by James and LeBreton (2012) in their review based on US samples. The internal reliability coefficient for the Croatian CRT-A calculated based on item total biserial coefficients (see Equation 21, p. 389; Gulliksen, 1950) was .73, which was quite similar to the value of .76 reported by James and LeBreton (2012).

2.2.2. Factor analyses In accordance with the instructions given in James and LeBreton (2012), for the purposes of factor analysis, we scored responses to the problems on a 3-point scale such that −1 (non-aggressive alternative), 0 (illogical or no response), and 1 (aggressive alternative). Using recoded responses, we conducted exploratory factor analysis on polychoric correlations analyses using fa.poly procedure in R (R Development Core Team, 2010). The procedure was used in order to take discrete categorical scoring into account (Joreskog & Sorbom, 2005) and to avoid underestimation due to skewed data (Olsson, Drasgow, & Dorans, 1982). The maximum likelihood was used as the model fitting procedure. Comparisons of one-, two-, and threefactor solutions revealed that the last two showed acceptable model fit (two factor: root mean square error of approximation [RMSEA] = .078; confidence interval [CI] = .073–.081; three-factor: RMSEA = .075; CI = .070–.078) whereas the fit of the one-factor model was above the .08 threshold (RMSEA = .086; CI = .082– .090). The two-factor solution converged with our subsequent interpretation of various solutions, which ultimately yielded a two-factor solution. The two factors were rotated obliquely using promax rotation. The correlation between the two factors was .26 (p < .001). The rotated factor structures obtained on the Croatian sample and the factor solution from a large US sample (n = 4772; James & LeBreton, 2012) are also shown in Table 2. James and LeBreton (2012) in their analysis labeled the first factor as the external justification bias because most of the problems that loaded on the factor indicated hostile attribution and victimization by powerful other biases. Essentially, people who have a high score on this factor justify their aggressive acts by others’ behavior. In the original analysis (James & LeBreton, 2012), the


347 second factor was labeled as internal justification because the problems on which it loads reflect the retribution and potency biases. Common to these two biases is an internal justification for aggressive behavior such that aggressive individuals see retribution as more appealing than forgiveness or reconciliation, or value interactions with others as a context for bravery. Finally, James and LeBreton (2012) interpreted the third factor obtained with US data as powerlessness because it was related to aggressive scores on conditional reasoning problems for which ‘a key source for building a false sense of rationality for aggression is anger and frustration with the lack of control over one’s life’ (James & LeBreton, 2012, p. 121). As might be seen from Table 2, Factor 1 from the Croatian data was reasonably similar to the external justification obtained on the US samples. All 11 problems that had primary loadings on the first factor in United States-based studies showed the largest correlations with the first factor for the Croatian sample. The most important difference was that the internal justification and powerlessness factors from the James and LeBreton (2012) factor analysis merged into a single factor in the Croatian sample. Most of the problems that loaded on the second and the third factor in the US sample had primary factor loadings on the second factor in the Croatian sample. This finding seems reasonably aligned with theory behind the instrument because justification for aggression on the conditional reasoning problems related to both the internal justification and powerlessness factors from James and LeBreton (2012) is found within aggressive individuals. Further differentiation of the internal justification factor into two factors observed in the James and LeBreton (2012) analysis could be related to the much smaller number of participants in our study. There are some exceptions in primary loadings between our analysis and earlier factor analyses but those differences could be traced either to the sampling or specific context of the data collection. In our analysis, the ‘generals’ and ‘bosses and employees’ problems loaded more strongly on the external justification, than in the original analysis in which it was more strongly saturated with the internal justification bias. However, it should be observed that both in our analysis and the original analyses (James & LeBreton, 2012) that problem had relatively high loadings on the first two factors. Therefore, the differences in primary projections may be related to simple sampling error. In our analysis the problem ‘good product at low price’ had low saturations with both factors revealing that it did not share variance with other problems. An interaction between the content of the conditional reasoning problem and the context of the data collection might have contributed to this finding. For the problem, respondents were asked to find the most logical conclusion that follows


348 the story about labor cost cuts (i.e., downsizing, wage reductions) to which organizations are forced in order to keep prices of their products low. The aggressive alternative ascribed ‘evil’ intention to the employing organizations that reflects hostile attribution bias. During the period of data collection, Croatian citizens experienced economic downturn followed by widely enforced layoffs and wage cuts that were not always justified by an economic necessity (Galic´ & Plecáš, 2012). This means that the selection of the aggressive alternative could reflect respondents’ actual experience and not a cognitive bias used by dispositionally aggressive individuals to justify hurting others. Regardless of minor differences, the psychometric properties of the Croatian adaptation of the CRT-A seemed reasonably similar to those obtained for the US sample and coherent with the theory behind the instrument.

3. Study 2 Within the second study, we examined the validity of the Croatian adaptation of the CRT-A in predicting selfreported CWBs. We also explored incremental validity of CRT-A scores over self-reports on the Big Five questionnaire and a self-report measure of aggression. These two instruments were selected because they represent the dominant conceptualization of personality for scientific and practical purposes (Big Five; Rothstein & Goffin, 2006) or represent a personality trait most often related to CWBs (anger; Douglas & Martinko, 2001; Hershcovics et al., 2007).

3.1. Method 3.1.1. Participants Participants in this study were full-time employees (n = 192) from a variety of occupations whose characteristics have already been described within Study 1 (the incumbents sample). 3.1.2. Instruments 3.1.2.1. Personality: conditional reasoning. The CRT-A measured the justification mechanisms for aggression. Internal consistency of the test was slightly lower than reported in previous studies with a value of 0.67. 3.1.2.2. Personality: self-report. The 50-item questionnaire from the International Personality Item Pool (IPIP50) measured the Big Five factors (Goldberg et al., 2006). In addition, we also used the IPIP 10-item angry hostility scale as our indicator of self-reported aggression. Reliability of all scales was satisfactory and ranged


Zvonimir Galic´, Kelly T. Scherer and James M. LeBreton between 0.77 (conscientiousness) and 0.88 (emotional stability). A Croatian version of this instrument was developed following the translation-back translation procedures. 3.1.2.3. Counterproductive work behaviors. Participants filled in a 34-item measure taken from the CWB checklist (Spector et al., 2006). The scale encompasses various CWBs directed toward other individuals and the organization. Respondents reported their frequency of CWBs on a 5-point scale for which 1 (never), 2 (once or twice), 3 (once or twice per month), 4 (once or twice per week), and 5 (every day). We aggregated all of the items in order to obtain a total score of CWBs. Reliability of the total score was 0.86. A Croatian version of this instrument was developed following the translation–back translation procedure. 3.1.3. Procedure The procedure of the recruitment and the incumbents’ CRT-A data collection is already given in the description of the procedure within Study 1. In addition to the CRT-A, the participants received a package containing the self-reported personality measures and the CWBs scale. After they completed all instruments, participants were instructed to put all of the questionnaires into an envelope, seal it, and give it to the student researchers who then brought the envelopes to the head researcher. All participants responded to the instruments anonymously, without giving any information about their identity.

3.2. Results and discussion The reported percentages revealed that CWBs were relatively common in our sample. For example, over two thirds of participants admitted that they took a longer break than they were allowed to take, and more than a half of the participants tried to look busy while doing nothing or ignored someone at work. A relatively high frequency was reported even for less desirable behavior (e.g., 42.7% of participants reported they purposely wasted their employer’s materials and supplies, and 35.4% had been nasty or rude to a client or customer). The percentages of reported CWBs suggest that participants were probably honest in reporting undesirable behaviors. In Table 3, we present the correlations among Study 2 variables. The CRT-A score had a significant positive correlation with CWBs total score (r = .22; p < .01), but also showed a significant relationship with gender (r = −.16), age (−.16), and tenure (−.14, all p < .05). No significant relationships were found between CRT-A score and self-reported personality, which was consistent with United States-based studies of the CRT-A


Conditional Reasoning Test for Aggression

349

Table 3. Descriptive statistics and intercorrelations of variables in Study 2 Variable

M (SD)

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 11.

1.54 (0.49) 36.71 (11.64) 1.48 (0.50) 14.42 (11.57) 33.75 (6.00) 38.99 (5.07) 37.47 (5.28) 33.10 (7.04) 35.40 (5.51) 27.00 (7.12) 4.57 (2.18) 49.81 (11.21)

.06 –

.06 −.03 –

.06 .94** −.15* –

.02 −.09 .04 −.08 –

.27** .04 .07 .07 .31** –

−.02 .21** .06 .24** −.10 .17* –

−.16* −.08 −.03 −.05 .21** .12 .13 –

−.13 −.18* .17* −.20 .37** .20** .04 .15* –

.13 .07 −.05 .04 −.05 −.15* −.15* −.83** −.14 –

−.16* −.19** −.05 −.16* .00 −.08 −.11 −.02 −.04 .02

−.14* −.12 .04 −.12 .13 −.21** −.25** −.18* .00 .19** .22** –

Gendera Age Educationb Tenure Extraversion Agreeableness Conscientiousness Emotional stability Intellect Angry hostility CRT-A CWB

Note: aMale = 1, female = 2. b1 = higher education, 2 = college education. CRT-A = Conditional Reasoning Test for Aggression; CWBs = counterproductive work behaviors; M = mean; SD = standard deviation.

(James & LeBreton, 2012). In addition to correlating with CRT-A score, the total score on the CWBs scale correlated with gender, agreeableness (negatively), conscientiousness (negatively), emotional stability (negatively), and angry hostility (positively). The finding that the CWBs scale correlated negatively with agreeableness, conscientiousness, and emotional stability and positively with the angry hostility scale was in accordance with the literature (Berry, Ones, & Sackett, 2007; Hershcovics et al., 2007). In order to test the potential incremental validity of the CRT-A scale in predicting CWBs, we conducted a hierarchical regression analysis wherein the CWBs scale total score was regressed first on self-report personality traits (entered in Step 2) before adding the CRT-A scores (entered in Step 3). Considering that age and gender showed a relationship with CRT-A and/or CWBs scale score, we treated them as covariates (entered in Step 1). By including these variables as we essentially allowed them to take credit for any variance they might have shared with CWBs and scores on the CRT-A. Thus, inclusion of these covariates provides a conservative test of the incremental validity of the CRT-A. Because of the high intercorrelation with age (.95, p < .001), tenure was not included in the regression analysis as a covariate. The angry hostility scale was highly correlated with the overall emotional stability scale (.83; p < .001). Thus, to minimize problems with multicollinearity (e.g., inflated standard errors; Cohen, Cohen, West, & Aiken, 2003) we opted to conduct two separate regression analyses. In the first, we tested the incremental validity of the CRT-A over the Big Five traits, and in the second analysis, over the angry hostility scale. The results of the two analyses are shown in Tables 4a and b, respectively.1 The results of Study 2 offer two important insights about the usefulness of Croatian adaptation of the CRT-A. First, CRT-A score showed a significant rela-


Table 4. (a) Summary of hierarchical regression analyses in predicting counterproductive work behaviors (CWBs): incremental validity of the Conditional Reasoning Test for Aggression (CRT-A) over the Big Five self-reports. (b) Summary of hierarchical regression analyses in predicting counterproductive work behaviors (CWBs): incremental validity of the Conditional Reasoning Test for Aggression (CRT-A) over angry hostility self-reports (a)

ΔR2

β

Predictor Step 1 Control variablesa Step 2 Extraversion Agreeableness Conscientiousness Emotional stability Intellect Step 3 CRT-A Total R2 n

.19** 192

(b)

ΔR2

Predictor Step 1 Control variablesa Step 2 Angry hostility Step 3 CRT-A Total R2 n

.03* .14** .22** −.18* −.15* −.21* .04 .02* .16*

β .03* .05** .22** .03* .18* .11**

Note: aControl variables included gender and age. *p < .05. **p < .01.

tionship with CWBs. The size of the correlation was relatively low (.22), but in accordance with the size of the correlations for the other personality constructs. Second, our data support the proposition that the CRT-A measures a construct that is unrelated to selfreported personality; the CRT-A score adds to the


350 prediction of CWBs above sociodemographic variables and self-reported personality. We should add that as an anonymous reviewer noted, the use of a self-report criterion as less than ideal, in part because previous research has documented that the optimal criteria against which to validate measures of implicit personality tend to be more objective behaviors (Bornstein, 1999; James & LeBreton, 2012). Objective criteria are often preferred because they circumvent potential concerns associated with response biases, faking, and the overall introspective accuracy (or lack thereof) that associated with some self-report measures (Haidt, 2001; Morgeson et al., 2007; Nisbett & Wilson, 1977). Indeed, a growing number of researchers (Bing, LeBreton, Davison, Migetz, & James, 2007; Bornstein, 1999, 2002; Frost, Ko, & James, 2007; James & LeBreton, 2010, 2012; McClelland, Koestner, & Weinberger, 1989; Spangler, 1992; Winter, John, Stewart, Klohnen, & Duncan, 1998) have found that measures of implicit and explicit personality often predict different types of criteria: (a) implicit measures are more closely linked to objective behavioral criteria whereas (b) explicit measures tend to be more closely related to subjectively measured criteria (e.g., selfreports or other reports). Thus, although the use of a self-report criterion measure in the current study was suboptimal, the most likely consequence of using such a criterion would be to decrease the likelihood of finding significant relationships with the CRT-A.

4. Study 3 One possible explanation for the predictive validity of the CRT-A in Study 2 was that the observed correlation with CWB may have resulted from a spurious relationship both variables might have with GMA. In the third study, we wanted to see if the finding that scores on the CRT-A are uncorrelated with measures of GMA (James & LeBreton, 2012) generalized to the Croatian adaptation of the test. In addition to an abstract reasoning test that served as our measure of GMA, we also included two additional variables known to be correlated with GMA: need for cognition (Cacioppo, Petty, & Kao, 1984) and cognitive reflection (Frederick, 2005). Although the need for cognition represents a well-known individual difference variable of inclination toward effortful cognitive activities (Cacioppo et al., 1984), the cognitive reflection test measures a tendency of respondents to inhibit a prepotent response alternative that is incorrect and to engage in more effortful reflection that leads to the correct response (Frederick, 2005). The relationship of the two variables with GMA served as a benchmark for our evaluation of the relationship between the CRT-A and GMA.



4.1. Method 4.1.1. Participants The study was conducted on a sample of undergraduate and graduate students from a Croatian business school (n = 100). Analogous to earlier described studies, participants who had five or more illogical answers were omitted from the database (n = 3, 3% of the sample). Thus, the analyses were conducted on the data from 97 participants. The participants were 48.5% male, with an average age of 23.80 (SD = 6.62).

4.1.2. Instruments Croatian versions of all instruments were developed using the translation-back translation procedures. 4.1.2.1. Personality: conditional reasoning. The CRT-A measured the justification mechanisms for aggression. The internal consistency reliability of the test was .75. 4.1.2.2. General mental ability. The abstract reasoning test from the Differential Aptitude Test battery (Bennet, Seashore, & Wessman, 1990) was used as a GMA measure. The internal consistency of the test using the KR 20 formula was .87. 4.1.2.3. Need for cognition. The need for cognition was measured using an 18-item (Cacioppo et al., 1984) scale. Sample items from the scale are ‘I would prefer complex to simple problems’ and ‘I only think as hard as I have to’ (negatively coded). The participants gave their responses on a 5-point scale ranging from 1 (completely disagree) to 5 (completely agree). The internal consistency of the scale in our study was .73. 4.1.2.4. Cognitive reflection. The Cognitive Reflection Test (Frederick, 2005) consists of three items that generate an intuitive incorrect answer and measures cognitive impulsivity, that is, the ability to resist reporting responses that come to mind. A sample item is ‘If it takes 5 machines to make 5 widgets, how long would it take 100 machines to make 100 widgets?’ The intuitive and incorrect answer is 100, and correct answer 5. The internal consistency of the test in this study was .55.

4.1.3. Procedure During the regular class, students were asked to complete a booklet consisting of three tests. They were instructed that all three tests measured aspects of cognitive functioning.


Conditional Reasoning Test for Aggression

351

Table 5. Descriptive statistics and intercorrelations between Study 3 variables

CRT-A Abstract reasoning Cognitive reflection Need for cognition

M (SD)

Abstract reasoning

Cognitive reflection

Need for cognition

4.86 (2.46) 30.10 (10.07) 0.50 (0.81) 57.83 (6.41)

.13 – – –

.27** .45** – –

.09 .25* .24* –

Note: *p < .05. **p < .01. CRT-A = Conditional Reasoning Test for Aggression; M = mean; SD = standard deviation.

4.2. Results and discussion The correlations given in Table 5 support our assumption that the CRT-A is not significantly related to GMA as measured with the DAT abstract reasoning test. The CRT-A also showed a nonsignificant correlation with the need for cognition but significantly correlated with the Cognitive Reflection Test. Considering that this observed relationship could result from sampling issues (e.g., restriction of range), we compared our correlations with those from another study containing the same three ‘cognitive’ variables (GMA, cognitive reflection, and need for cognition). The correlations reported by Frederick (2005) gave us confidence in our findings. In his study, the correlations between GMA (measured with the Wonderlic Personnel Test) and cognitive reflection, GMA and need for cognition, and cognitive reflection and need for cognition were almost identical to those obtained in our study (.43, .19, and .22, respectively). The significant positive correlation between the CRT-A score and cognitive reflection obtained in our study is somewhat surprising (especially given the relatively meager reliability of our measure of cognitive reflection). These results indicate that individuals who tended to rely on justification mechanisms linked to the motive to aggress were also less cognitively impulsive.

5. Study 4 Within the fourth study, we sought to test the assumption that the CRT-A is relatively immune to faking (LeBreton et al., 2007), at least when the indirect nature of measurement is preserved. We compared CRT-A data obtained from the sample of incumbents, with student respondents who were asked to fake good, and with actual job applicants who completed the CRT-A within a personnel selection test battery. Considering that incumbents anonymously responded to the CRT-A outside of their organizations, we believed that we enhanced the probability of honest responding. On the other hand, respondents faking good (i.e., those instructed to fill out the survey as if they were an ideal job candidate) and responding in a personnel selection


situation should have an incentive to improve their selfpresentation. In accordance with earlier studies (LeBreton et al., 2007), we expected that the Croatian adaptation of the CRT-A should show resistance to faking. However, in order to have a standard against which we can compare the ‘fakeability’ of the CRT-A, in the honest and ‘fake good’ conditions, we also collected data for the selfreport angry hostility scale. Unfortunately, angry hostility data were not available in the applicant situation.

5.1. Method 5.1.1. Participants In addition to incumbents and applicants described in Study 1, an independent sample of participants was asked to complete the CRT-A in a ‘fake good’ situation (n = 100). Because the number of illogical responses was 5 or higher, data from seven participants from the ‘fake good’ condition were omitted from further analyses. Participants in the ‘fake good’ condition were students from a large Croatian university (gender: 83.7% female, age: M = 22.41; SD = 1.42). Therefore, the final number of respondents in Study 3 was 192 in the honest condition, 93 in ‘fake good’ condition, and 147 in the personnel selection condition. 5.1.2. Instruments In all of the conditions, participants completed the Croatian adaptation of the CRT-A. In the honest condition and the ‘fake good’ condition, participants also completed the IPIP angry hostility scale described in Study 2. Internal consistencies of the CRT-A were .67 under honest condition, .65 under ‘fake good’, and .74 within selection. The reliability of the angry hostility scale were .89 under honest responding and .90 in ‘fake good’ situation. 5.1.3. Procedure The procedure of data collection from honest respondents and applicants was described earlier in Study 2. In other words, participants were instructed to take the CRT-A in accordance with the guidelines provided in


352


Table 6. Means and standard deviations on the Conditional Reasoning Test for Aggression (CRT-A) and the angry hostility scale in Study 4 situations Trait

Honest M (SD)

Fake good M (SD)

Selection M (SD)

CRT-A Angry hostility scale

4.57 (2.18) 2.71 (0.71)

4.59 (2.29) 1.62 (0.64)

4.31 (2.34) –

M = mean; SD = standard deviation.

the test manual (James & McIntyre, 2000). Participants in the ‘fake good’ condition were asked to complete the questionnaire anonymously during a regular class session. They received instructions to complete the surveys in a survey package consisting of the CRT-A and self-report personality questionnaires, as an ideal candidate for a nursing job. The nursing job was selected because aggressiveness (i.e., the motive to harm) represents an undesirable trait for a vocation focused on helping and healing others.

5.2. Results and discussion The descriptive statistics for the CRT-A in the three conditions and for the angry hostility scale from the honest responding and ‘fake good’ conditions are given in Table 6. The analysis of variance (ANOVA) results showed that differences in the CRT-A scores between situations were nonsignificant, F(2, 429) = 0.71, p = .49.2 At the same time, the difference between the angry hostility scale scores in the honest responding and ‘fake good’ conditions was highly significant, t (282) = 12.26; p < .001. The size of the difference on angry hostility scale (Cohen’s d = 1.61) revealed that our instruction was successful in creating a faking response set. The correlation between CRT-A score and angry hostility score was nonsignificant both under the honest responding (r = .02; p = .820) and ‘fake good’ instructions (r = .10; p = .350). Therefore, when the indirect nature of measurement is preserved, the CRT-A score did not show sensitivity to situational cues that are sometimes observed for selfreport personality measures. These results were consistent with the assumption that the CRT-A represents an indirect measure of cognitive biases.

6. General discussion To the best of our knowledge, construct and criterionrelated validity evidence for the inferences drawn from scores on the CRT-A has not been accumulated in a cultural context other than the United States where the instrument was developed. Our studies were conducted


in Croatia, a post-socialistic country that, during the 1990s, went through the war for independence and a turbulent postwar period (Tanner, 2010). Moreover, the context in which our participants and their families developed was marked by the transition from a socialistic planned economy to a democratic free-market society characterized by corruption, injustice, and growing social inequalities (Galic´ & Plecáš, 2012; Nestic´, 2002; Vojnic´, 1994). All of these characteristics created an interesting context for using the CRT-A because in many social situations, our participants likely experienced hostile intention behind others’ behavior (e.g., during the war), or were victimized by powerful others (e.g., corrupted politicians or ‘tycoons’). Thus, selecting ‘aggressive’ responses to conditional reasoning problems may be driven in part by extended exposure to the hostile intentions of others and being victimized by powerful others. If this were the case, we would expect to find limited validity evidence for the CRT-A in this setting. However, across four studies we accumulated validity and reliability evidence that was quite similar to that reported by James and his colleagues using samples from the United States. In Study 1, we observed a relatively coherent factor structure that could be meaningfully linked to the structure reported by James and LeBreton (2012). The results of Studies 2 and 3 indicated that scores on the CRT-A predicted CWBs and explained variance that was unrelated to that explained using an array of self-report personality surveys, or GMA tests. Finally, comparison of CRT-A scores between contexts differing in a need for positive self-presentation provided an initial test of the indirect nature of the CRT-A in Croatian settings and its general resistance to faking. Our study has several limitations and related suggestions for future research. We will mention three that seem the most salient. First, the main objection someone could give to this study is that the data were collected in a small European country, and, thus, the generalizability of obtained conclusions to other countries/populations is limited. Although this might be true, the goal of our study was to examine if the psychometric characteristics and the validity evidence for the CRT-A would generalize to samples obtained from a very diverse cultural context. We hope our findings encourage other international researchers to begin testing whether the CRT-A could be used in their countries. Second, the relationship between CRT-A scores and CWBs obtained in our study are, at best, modest, and the usefulness of the test might be seen as limited. The zero-order correlation between the CRT-A and CWBs was .22, which can be categorized as a small to medium effect size (Cohen, 1988). If the psychometric properties (i.e., reliability) of the test were improved with a longer set of inductive problems, and separate scales developed for specific forms of bias, and then related to specific forms of CWBs, we could expect higher correlations


Conditional Reasoning Test for Aggression (e.g., if victimization by powerful others is related with organizational deviance, and hostile attribution bias with interpersonal deviance). Finally, as first mentioned in the results section of Study 2, the fact that we used self-report surveys to assess CWBs may be seen as problematic by some researchers. For example, to the extent that respondents either engaged in socially desirable responding or simply lacked introspective insight into their behavior, the observed relationship between CWBs and CRT-A would be attenuated. Such attenuation would be consistent with previous research using the CRT-A. For example, James and LeBreton (2012) reported smaller criterionrelated validities when correlating the CRT-A with subjective data, especially self-report data. On the other hand, some researchers might view the use of self-reports of CWBs as advantageous. For example, the most recent meta-analysis by Berry, Carpenter, and Barratt (2012) showed that self-reports of CWBs might generally be a preferable alternative to other reports. Their analyses showed that self-raters revealed more CWBs than their coworkers/supervisors, and that other reports added little incremental variance over self-reported CWBs for the most often-studied covariates (e.g., personality traits, organizational justice). Although our results provide tentative support for the use of the CRT-A in a Croatian context, considering that this conditional reasoning represents a new technology of personality conceptualization and measurement, future research using additional criteria for aggressive behavior is certainly warranted.

Notes 1.

The substantive conclusions concerning the incremental prediction of the CRT-A did not change when a single regression analysis was conducted containing all six of the self-report surveys in Step 2. 2. Considering that groups differed in gender and age composition, and that age on the total sample was correlated with the CRT-A score, we also conducted analyses of covariance where age and gender were used as covariates. Substantive conclusions were the same; thus, we reported the more parsimonious ANOVA results.

References Banks, G. C., Kepes, S., & McDaniel, M. A. (2012). Publication bias: A call for improved meta-analytic practice in the organizational sciences. International Journal of Selection and Assessment, 20, 182–196. Bennet, G. K., Seashore, H. G., & Wessman, A. G. (1990). Differential aptitude test for selection – general abilities battery (GAB): Manual. New York: The Psychological Corporation. Berry, C. M., Carpenter, N. C., & Barratt, C. L. (2012). Do other-reports of counterproductive work behavior provide


353 an incremental contribution over self-reports? A metaanalytic comparison. Journal of Applied Psychology, 97, 613– 636. Berry, C. M., Ones, D. S., & Sackett, P. R. (2007). Interpersonal deviance, organizational deviance, and their common correlates: A review and meta-analysis. Journal of Applied Psychology, 92, 410–424. Berry, C. M., Sackett, P. R., & Tobares, V. (2010). A metaanalysis of conditional reasoning test of aggression. Personnel Psychology, 63, 361–384. Bing, M. N., LeBreton, J. M., Davison, H. K., Migetz, D. Z., & James, L. R. (2007). Integrating implicit and explicit social cognitions for enhanced personality assessment: A general framework for choosing measurement and statistical methods. Organizational Research Methods, 10, 346–389. Bornstein, R. F. (1999). Criterion validity of objective and projective dependency tests: A meta-analytic assessment of behavioral prediction. Psychological Assessment, 11, 48–57. Bornstein, R. F. (2002). A process dissociation approach to objective-projective test score interrelationships. Journal of Personality Assessment, 78, 47–68. Cacioppo, J. T., Petty, R. E., & Kao, C. F. (1984). The efficient assessment of need for cognition. Journal of Personality Assessment, 48, 306–307. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Mahwah, NJ: Lawrence Erlbaum. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.) Mahwah, NJ: Lawrence Erlbaum Associates. Douglas, S. C., & Martinko, M. J. (2001). Exploring the role of individual differences in the prediction of workplace aggression. Journal of Applied Psychology, 86, 547–559. Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19, 25–42. Frost, B. C., Ko, C. E., & James, L. R. (2007). Implicit and explicit personality: A test of a channeling hypothesis for aggressive behavior. Journal of Applied Psychology, 92, 1299– 1319. Galic´, Z., & Plecáš, M. (2012). Quality of working life during the recession: The case of Croatia. Croatian Economic Survey, 14, 5–41. Geisinger, K. F. (1994). Cross-cultural normative assessment: Translation and adaptation issues influencing the normative interpretation of assessment instruments. Psychological Assessment, 6, 304–312. Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. G. (2006). The international personality item pool and the future of public-domain personality measures. Journal of Research in Personality, 40, 84–96. Gulliksen, H. (1950). Theory of mental tests. New York: Wiley. Haidt, J. (2001). The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review, 108, 814–834. Hershcovics, M. S., Turner, N., Barling, J., Arnold, K. A., Dupré, K. E., Inness, M., et al. (2007). Predicting workplace aggression: A meta-analysis. Journal of Applied Psychology, 92, 228–238. Hofstede, G. H. (2001). Culture’s consequences: Comparing values, behaviors, institutions and organizations across nations. Thousand Oaks, CA: Sage.


354 Hui, C. H., & Triandis, H. C. (1985). Measurement in crosscultural psychology: A review and comparison of strategies. Journal of Cross-Cultural Psychology, 16, 131–152. James, L. R., & LeBreton, J. M. (2010). Assessing aggression using conditional reasoning. Current Directions in Psychological Science, 19, 30–35. James, L. R., & LeBreton, J. M. (2012). Assessing the implicit personality through conditional reasoning (1st ed.) Washington, DC: American Psychological Association. James, L. R., & McIntyre, M. D. (2000). Conditional reasoning test of aggression: Test manual. Knoxville, TN: Innovative Assessment Technology. James, L. R., McIntyre, M. D., Glisson, C. A., Green, P. D., Patton, T. W., LeBreton, J. M., & Mitchell, T. R. (2005). A conditional reasoning measure for aggression. Organizational Research Methods, 8, 69–99. Joreskog, K. G., & Sorbom, D. (2005). Lisrel 8.72. Chicago: Scientific Software International. LeBreton, J. M., Barksdale, C. D., Robin, J., & James, L. R. (2007). Measurement issues associated with conditional reasoning tests: Indirect measurement and test faking. Journal of Applied Psychology, 92, 1–16. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley. McClelland, D. C., Koestner, R., & Weinberger, J. (1989). How do self-attributed and implicit motives differ? Psychological Review, 96, 690–702. Morgeson, F. P., Campion, M. A., Dipboye, R. L., Hollenbeck, J. R., Murphy, K., & Schmitt, N. (2007). Reconsidering the use of personality tests in personnel selection contexts. Personnel Psychology, 60, 683–729. Nestic´, D. (2002). Ekonomske nejednakosti u Hrvatskoj 1973– 1998. (Economic inequalities in Croatia 1978–1998). Financijska Teorija I Praksa, 26, 595–613.


Zvonimir Galic´, Kelly T. Scherer and James M. LeBreton Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231–259. Olsson, U., Drasgow, F., & Dorans, N. J. (1982). The polyserial correlation coefficient. Psychometrika, 47, 337–347. R Development Core Team. (2010). R: A language and environment for statistical computing. Available at http://www.Rproject.org (accessed 22 September 2012). Rothstein, M. G., & Goffin, R. D. (2006). The use of personality measures in personnel selection: What does current research support? Human Resource Management Review, 16, 155–180. Spangler, W. D. (1992). Validity of questionnaire and TAT measures of need for achievement: Two meta-analyses. Psychological Bulletin, 112, 140–154. Spector, P. E., Fox, S., Penney, L. M., Bruursema, K., Goh, A., & Kessler, S. (2006). The dimensionality of counterproductivity: Are all counterproductive behaviors created equal? Journal of Vocational Behavior, 68, 446–460. Tanner, M. (2010). Croatia: A nation forged in war (3rd ed.) Yale: University Press. Vojnic´, D. (1994). European integration processes and the countries in transition – with special reference to Croatia and former Yugoslavia. Ekonomski Pregled, 9–10, 203–239. Wanous, J. P., Sullivan, S. E., & Malinak, J. (1989). The role of judgment calls in meta-analysis. Journal of Applied Psychology, 74, 259–264. Winter, D. G., John, O. P., Stewart, A. J., Klohnen, E. C., & Duncan, L. E. (1998). Traits and motives: Toward an integration of two traditions in personality research. Psychological Review, 105, 230–250. Wu, J., & LeBreton, J. M. (2011). Reconsidering the dispositional basis of counterproductive work behavior: The role of aberrant personality. Personnel Psychology, 64, 593–626.
