The Validity of Interpersonal Skills Assessment Via Situational ... - UGent

Journal of Applied Psychology 2012, Vol. 97, No. 2, 460 – 468

© 2011 American Psychological Association 0021-9010/12/$12.00 DOI: 10.1037/a0025741

RESEARCH REPORT

The Validity of Interpersonal Skills Assessment Via Situational Judgment Tests for Predicting Academic Success and Job Performance Filip Lievens

Paul R. Sackett

Ghent University

University of Minnesota, Twin Cities Campus

This study provides conceptual and empirical arguments why an assessment of applicants’ procedural knowledge about interpersonal behavior via a video-based situational judgment test might be valid for academic and postacademic success criteria. Four cohorts of medical students (N ⫽ 723) were followed from admission to employment. Procedural knowledge about interpersonal behavior at the time of admission was valid for both internship performance (7 years later) and job performance (9 years later) and showed incremental validity over cognitive factors. Mediation analyses supported the conceptual link between procedural knowledge about interpersonal behavior, translating that knowledge into actual interpersonal behavior in internships, and showing that behavior on the job. Implications for theory and practice are discussed. Keywords: interpersonal skills, situational judgment test, high-stakes testing, student selection, medical selection

Situational judgment tests (SJTs) are a step removed from direct observation, and are better viewed as measures of procedural knowledge in a specific domain (e.g., interpersonal skills). SJTs confront applicants with written or video-based scenarios and ask them to indicate how they would react by choosing an alternative from a list of responses (Christian, Edwards, & Bradley, 2010; McDaniel, Hartman, Whetzel, & Grubb, 2007). Widely used in employment contexts, there is also growing interest in the potential use of SJTs in student admission/selection settings (Lievens, Buyse, & Sackett, 2005a; Oswald, Schmitt, Kim, Ramsay, & Gillespie, 2004; Schmitt et al., 2009). The initial evidence for the use of SJTs in such high-stakes settings is promising. In particular, Lievens et al. (2005a) found that a video-based SJT predicted grade point average (GPA) in interpersonal skills courses. Although the results of these studies are promising, they do not address the key issue as to whether students’ interpersonal skills scores on a video-based SJT at the time of admission really achieve long-term prediction of performance in actual interpersonal situations. Such interpersonal interactions can be observed in some academic settings (e.g., in internships) and subsequently on the job after completion of a program of study. Therefore, we use a predictive validation design to examine whether an assessment of interpersonal skills via the use of SJTs at the time of admission predicts academic (internship performance) and postacademic (job performance) criteria. Our study is situated in the context of selection to medical school in Belgium. Medical admissions are a relevant setting for examining the value of interpersonal skills assessments for predicting academic and postacademic success because in many countries, calls have been made to include a wider range of skills in medical admission (Barr, 2010; Bore, Munro, & Powis, 2009; Powis, 2010). Our interest in predicting outcomes other than academic performance, particularly outcomes after students have completed their

Terms such as “personal characteristics,” “soft skills,” “noncognitive skills,” and “21st-century skills” are often used to refer to a wide array of attributes (e.g., resilience, honesty, teamwork skills) viewed as valuable in many settings, including work and higher education. There exists a long interest in measuring these noncognitive predictors for use in selection or admission as they enable to go beyond cognitive ability, predict various success criteria, and reduce adverse impact (Kuncel & Hezlett, 2007; Schmitt et al., 2009; Sedlacek, 2004). We focus in this article on one characteristic of broad interest, namely, interpersonal skills. This refers to skills related to social sensitivity, relationship building, working with others, listening, and communication (Huffcutt, Conway, Roth, & Stone, 2001; Klein, DeRouin, & Salas, 2006; Roth, Bobko, McFarland, & Buster, 2008). Over the years, a variety of approaches for measuring interpersonal skills have been proposed and examined. In workplace settings, methods that involve direct observation of interpersonal behavior are widely used (e.g., interviews, work samples, and assessment center exercises; Arthur, Day, McNelly, & Edens, 2003; Ferris et al., 2007; Huffcutt et al., 2001; Roth et al., 2008). In higher education admissions settings, such direct approaches are uncommon, as it is difficult to reliably and formally apply them in the large-scale and high-stakes nature of student admission contexts (Schmitt et al., 2009).

This article was published Online First October 3, 2011. Filip Lievens, Department of Personnel Management and Work and Organizational Psychology, Ghent University, Belgium; Paul R. Sackett, Department of Psychology, University of Minnesota, Twin Cities Campus. We thank Tine Buyse for her help in collecting the data. Correspondence concering this article should be addressed to Filip Lievens, Department of Personnel Management and Work and Organizational Psychology, Ghent University, Henri Dunantlaan 2, 9000 Ghent, Belgium. E-mail: [email protected] 460

INTERPERSONAL SKILLS AND SITUATIONAL JUDGMENT TESTS

studies and entered their chosen profession, fits in a growing trend in graduate and professional education (Kuncel & Hezlett, 2007; Kuncel, Hezlett, & Ones, 2004; Schultz & Zedeck, 2008). This trend has also generated some discussion. Specifically, two differing perspectives on the appropriate role of tests used in the admission process can be distinguished. One perspective posits that predicting academic outcomes (e.g., GPA) is an insufficient basis for the justification of the use of admission tests, as academic outcomes are not of interest in and of themselves. From this perspective, links to postacademic (employment) performance are essential for justifying test use. A competing perspective is that the goal of admission testing is to identify students who will develop the highest level of knowledge and skill in the curriculum of interest, as such knowledge and skill is the foundation for effective practice in a profession. From this perspective, research linking admissions tests to postacademic outcomes is not critical to the justification of test use. This debate on the appropriate role of admissions tests reflects value judgments, and thus cannot be resolved by research. Nonetheless, regardless of whether links to postacademic outcomes are or are not viewed as critical to justifying test use, we believe that an investigation of these linkages helps to better understand what tests can and cannot accomplish. By linking a test given as part of an admission process with job performance subsequent to degree completion, this study also spans the education and employment worlds.

Interpersonal Skills and SJTs Recent taxonomies of interpersonal skills (Carpenter & Wisecarver, 2004; Klein et al., 2006) make a distinction between two metadimensions, namely, building and maintaining relationships (e.g., helping and supporting others) and communication/ exchanging information (e.g., informing and gathering information). In the medical field, the “building and maintaining relationships” and “communication/exchanging information” have also been proposed as key dimensions for effective physician–patient interactions (Accreditation Council for Graduate Medical Education, 2010; Makoul, 2001). For example, the clinical skills examination checklist used by the National Board of Medical Examiners includes items such as listening carefully to patients or encouraging patients to express thoughts and concerns (indicators of building and maintaining relationships) and using words that patients can understand or using clear and precise speech (indicators of communication/exchanging information). In this study, students’ interpersonal skills in the physician– patient interaction were assessed via a video-based SJT. A higher fidelity video-based format is particularly relevant in this context as meta-analytic evidence has revealed a large validity difference between video-based and written SJTs (.47 vs. .27) for measuring interpersonal skills (Christian et al., 2010). The SJT contained video-based situations related to the dimensions of “building and maintaining relationships” and “communication/exchanging information,” which is consistent with the emphasis placed on those dimensions by interpersonal skills taxonomies and medical examination boards. For example, the SJT included situations dealing with showing consideration and interest, listening to patients, conveying bad news, reacting to patients’ refusal to take the prescribed medicine, or using appropriate language for explaining

461

technical terms. No medical knowledge was necessary to complete the SJT. The assumption underlying the use of an SJT in a medical admission context was that students’ scores on such interpersonally oriented SJT situations at the time of admission will predict their performance in future actual interactions with patients, as observed/rated during internships and many years later on the job. So, even though students at the time of admission might not have any experience as physicians with patient situations, we expect their answers on the video-based SJT situations to be predictive of their future internship and job performance. Conceptually, this expectation is based on the theory of knowledge determinants underlying SJT performance (Motowidlo & Beier, 2010; Motowidlo, Hooper, & Jackson, 2006). According to Motowidlo et al., an SJT is a measure of procedural knowledge, which can be broken down in job-specific procedural knowledge and general/nonjobspecific procedural knowledge. Whereas the former type of knowledge is based on job-specific experience, the latter accrues from experience in general situations. As SJTs used in admission exams do not rely on job-specific knowledge, only general procedural knowledge is relevant. Motowidlo defined this general procedural knowledge as the knowledge somebody has acquired about effective and ineffective courses of trait-related behavior in situations like those described in the SJT. Applied to this study’s interpersonally oriented SJT, this general procedural knowledge relates to students’ procedural knowledge about (in)effective behavior in interpersonal situations (with patients) as depicted in the SJT items. This procedural knowledge about costs and benefits of engaging in interpersonally oriented behavior is considered to be a precursor of actual behavior in future interpersonal situations (Lievens & Patterson, 2011; Motowidlo & Beier, 2010; Motowidlo et al., 2006). So, the assumption is that students with substantial procedural knowledge about effective behavior in interpersonal situations as assessed by an SJT will also show superior interpersonal performance when they later interact with actual patients as compared with students with inadequate procedural knowledge about effective interpersonal behavior.

Stability of Interpersonal Skills When considering the use of measures of procedural knowledge about interpersonal behavior to predict both academic and postacademic employment criteria that span multiple years, it is important to consider the stability of the interpersonal skills construct. Hypothesizing such relationships requires the assumption of some degree of stability in students’ interpersonal knowledge and skills as they progress through the curriculum and onto the job. In the vast literature on dynamic criteria (Alvares & Hulin, 1972; Barrett, Phillips, & Alexander, 1981; Campbell & Knapp, 2001; Deadrick & Madigan, 1990; Ghiselli, 1956; Schmidt, Hunter, Outerbridge, & Goff, 1988), this issue as to whether individuals change over time is captured by the “changing person” model. In the ability domain, the changing person explanation has now been largely rejected as postdictive validities appear to follow the same patterns of changes as predictive validities (Humphreys & Taber, 1973; Lunneborg & Lunneborg, 1970). Similar arguments of stability have been made for personality traits, as meta-analytic evidence (Fraley & Roberts, 2005; Roberts & DelVecchio, 2000) suggests that rank-order stability is high (Caspi, Roberts, & Shiner, 2005).

LIEVENS AND SACKETT

462

At first glance, one might expect a different pattern of findings in the interpersonal skills domain, as training programs and courses aim to change these skills, and are successful at doing so. Arthur, Bennett, Edens, and Bell’s (2003) meta-analysis of training program effectiveness reports mean ds for interpersonal skills of 0.68 for learning criteria and 0.54 for behavioral criteria. However, in considering the implications of this for the changing person model, it is important to consider the implications of different types of change. Here we consider four possible ways for interpersonal skills to be changed by intervention. First, an intervention might improve the skills of all individuals by a comparable amount, in which case the validity of a predictor of interpersonal skills would be unaffected. Second, an intervention might improve the skills of those with severe deficits, but have little impact on those with good skills. In this case, it is possible that rank order is unchanged; all that is seen is a tightening of the distribution, and the validity of an interpersonal skills predictor is also affected only to a limited degree. Third, the intervention might train all individuals to a common level of interpersonal skill, in which variance would be reduced to zero, and therefore validity of a predictor would also go to zero. Fourth, the intervention might be differentially effective, resulting in substantial change in the rank ordering of individuals in terms of their interpersonal skills, and thus in substantial reduction in validity. For example, Ceci and Papierno (2005) report that it is not uncommon for those with higher preintervention scores to benefit more from the intervention than those with lower scores. Thus, the first two possible forms of “changing abilities” pose no threat to validity, whereas the last two forms do pose a threat. However, we note that if either of these latter two forms were the true state of affairs, one would observe very low pretest–posttest correlations between measures of interpersonal skills. In contrast, a high pretest–posttest correlation would be strong evidence against these latter two forms. We find such evidence in a metaanalysis by Taylor, Russ-Eft, and Chan (2005) of behavioral modeling training programs aimed at interpersonal skills. They reported a mean pretest–posttest correlation of .84 across 21 studies for the effects of training on job behaviors, which is inconsistent with either the “training eliminates variance” or the “training radically alters rank order” perspectives. Thus, we expect that the forms of a “changing persons” argument that would lead to reduced validity can also be rejected in the interpersonal skills domain.

model. One might consider performance on an interpersonally oriented SJT, internship performance with patients, and job performance with actual patients as assessments of interpersonal skills that differ in their degree of fidelity. Hereby fidelity refers to the extent to which the assessment task and context mirror those actually present on the job (Callinan & Robertson, 2000; Goldstein, Zedeck, & Schneider, 1993). As noted above, an interpersonal SJT serves as an interpersonal skills assessment with a low degree of fidelity because it assesses people’s procedural knowledge about interpersonal behavior instead of their actual behavior. Next, internship performance can be considered a high-fidelity assessment (Zhao & Liden, 2011) because during internships, individuals have the opportunity to interact with real patients under close supervision. However, they work only with a small number of patients and have no responsibility for those patients. Finally, physicians’ demonstration of interpersonal skills with actual patients on the job is no longer a simulation as it constitutes the criterion to be predicted. All of this suggests a mediated model wherein procedural knowledge about interpersonal behavior (lowfidelity assessment) predicts internship performance (high-fidelity assessment), which in turn predicts job performance (criterion). So far, no studies have scrutinized these causal links between lowfidelity/high-fidelity samples and the criterion domain. We posit the following: Hypothesis 2 (H2): The relationship between procedural knowledge about interpersonal behavior (low-fidelity assessment) and job performance (criterion) will be mediated by internship performance (high-fidelity assessment). It should be noted that internship and job performance are multidimensional because in this study, performance on both involves a combination of interpersonal as well as technical knowledge/skills. So, internship and job performance are saturated with both interpersonal and technical/medical skills. Saturation refers to how a given construct (e.g., interpersonal skills) influences a complex multidimensional measure (e.g., ratings of job performance; Lubinski & Dawis, 1992; Roth et al., 2008). The multidimensional nature of internship and job performance makes it useful to examine the incremental contribution of procedural knowledge about interpersonal behavior over and above other factors (i.e., cognitive factors) influencing performance. In particular, we expect the SJT to offer incremental validity over cognitive factors in predicting internship and job performance. Thus,

On the basis of the above discussion, we posit the following hypotheses:

Hypothesis 3a (H3a): Procedural knowledge about interpersonal behavior (assessed by an SJT) will have incremental validity over cognitive factors for predicting internship performance.

Hypothesis 1a (H1a): Procedural knowledge about interpersonal behavior (assessed by an SJT at the time of admission) will be a valid predictor of internship performance.

Hypothesis 3b (H3b): Procedural knowledge about interpersonal behavior (assessed by an SJT) will have incremental validity over cognitive factors for predicting job performance.

Hypotheses

Hypothesis 1b (H1b): Procedural knowledge about interpersonal behavior (assessed by an SJT at the time of admission) will be a valid predictor of job performance. Using the behavioral consistency logic (Schmitt & Ostroff, 1986), these first hypotheses might be tested together in a mediated

Method Sample and Procedure This study’s sample consisted of 723 students (39% men and 61% women; average age ⫽ 18 years and 4 months; 99.5%


Caucasian) who completed the Medical Studies Admission Exam in Belgium, passed the exam, and undertook medical studies. This sample includes four entering cohorts of students who had completed the exam between 1999 and 2002. We focused on these four cohorts because criterion data for the full curriculum (7 years) were available from them. On average, the passing rate of the admission exam was about 30%. Candidates who passed the exam received a certificate that warranted entry in any medical university. Thus, there was no further selection on the part of the universities. This sample came from a population of 5,444 students (63% women, 37% men; average age ⫽ 18 years and 10 months; 99.5% Caucasian) who completed the examination between 1999 and 2002. Data on this total applicant pool will be used for range restriction corrections to estimate validity in the applicant pool (see below).

Predictor Measures Students’ scores on the predictor measures were gathered during the actual admission exam. Each year, the exam lasted for a whole day and was centrally administered in a large hall. To preserve the integrity of the tests, alternate forms per test were developed each year. Therefore, candidates’ test scores were standardized within each exam. Cognitive composite. In their meta-analysis, Kuncel, Hezlett, and Ones (2001) showed that a composite of general measures (e.g., Graduate Record Exam [GRE] verbal and numerical) combined with specific GRE subject-matter tests provided the highest validity in predicting academic performance. To provide the strongest test of the incremental validity of interpersonal skills, a cognitive composite was used that consisted of four science knowledge test scores (biology, chemistry, mathematics, and psychics) and a general mental ability test score. The science-related subjects consisted of 40 questions (10 questions per subject) with four alternatives. The general mental ability test consisted of 50 items with five possible answers. The items were formulated in verbal, numeric, or figural terms. In light of test security, we cannot mention the source of this test. Prior research demonstrated the satisfactory reliability and validity of this cognitive composite for a medical student population (Lievens et al., 2005a). Medical text. In this test, candidate medical students were asked to read and understand a text with a medical subject matter. Hence, this test can be considered a miniaturized sample of tasks that students will encounter in their education. The texts developed typically drew on texts from popular medical journals. Students had 50 min to read the text and answer 30 multiple-choice questions (each with four possible answers). Across the exams, the average internal consistency coefficient of this test equaled .71. Video-based SJT. The general aim of the video-based SJT was to measure interpersonal skills. As noted above, the videobased SJT focused on the “building and maintaining relationships” and “communication/exchanging information” components of interpersonal skills. First, realistic critical incidents regarding these domains were collected from experienced physicians and professors in general medicine. Second, vignettes that nested these incidents were written. Two professors teaching physicians’ consulting practices tested these vignettes for realism. Using a similar approach, questions and response options were derived. Third,

463

semiprofessional actors were hired and videotaped. Finally, a panel of experienced physicians and professors in general medicine developed a scoring key. Agreement among the experts was satisfactory, and discrepancies were resolved upon discussion. The scoring key indicated what response alternative was correct for each item (⫹1 point). In its final version, the SJT consisted of videotaped vignettes of interpersonal situations that physicians are likely to encounter with patients. After each critical incident, the scene froze, and candidates received 25 s to answer the question (“What is the most effective response?”). In total, the SJT consisted of 30 multiple-choice questions with four possible answers. Prior research attests to the construct-related validity of the SJTs developed as they consistently correlated with scores on interpersonally oriented courses in the curriculum (Lievens et al., 2005a). Prior studies also revealed that alternate form reliability of the SJTs was .66 (Lievens, Buyse, & Sackett, 2005b), which was consistent with values obtained in studies on alternate form SJTs (Clause, Mullins, Nee, Pulakos, & Schmitt, 1998). Operational composite. To make actual admission decisions, a weighted sum of the aforementioned predictors (cognitively oriented tests, medical text, and SJT) was computed. The weights and cutoff scores were determined by law, with the most weight given to the cognitive composite. A minimal cutoff was determined on this operational composite.

Criterion Measures Internship performance rating. Two of the authors inspected the descriptions of the medical curricula of the universities. To qualify as an “internship,” students had to work in a temporary position with an emphasis on on-the-job training and contact with patients (instead of with simulated patients). Interrater reliability (ICC 2,1) among the authors was ⬎ .90. In four of the seven academic years (first, fourth, sixth, and seventh year), internships were identified. Whereas in the first year the internship had a focus on observation, the other internships were clinical clerkships lasting 2– 4 months in different units. In their internships, students were evaluated on their technical (e.g., examination skills) and interpersonal skills (e.g., contact with patients) using detailed score sheets. Internship ratings of 606 students were obtained from archival records of the universities. Only the global internship ratings were available. Those ratings ranged from 0 to 20, with higher scores indicating better ratings. Given differences across universities, students’ internship ratings were standardized within university and academic year. As the four internship ratings were significantly correlated, a composite internship performance rating was computed. Job performance rating. Some of the medical students of this study (about 20%, n ⫽ 103) who ended their 7 years of education chose a career in general medicine and entered a General Practitioner training program of up to 2 years duration. During that program, they worked under the supervision of a registered General Practitioner in a number of general practice placements. At the end of the training program, all trainees were rated on a scale from 0 to 20 by their supervisor. None of the supervisors had access to the trainees’ admissions scores. Similar evaluation sheets (with a focus on technical and interpersonal skills) were used for internship performance ratings. Only the global job performance ratings

LIEVENS AND SACKETT

464

were available from the archives. As expected, internship and job performance were moderately correlated (corrected r ⫽ .40). As the above description refers to participants as “trainees,” a question arises as to whether this should be viewed as a measure of “training performance” rather than of “job performance.” We view this as “job performance” in that these graduates are engaged in full-time practice of medicine. They are responsible for patients while they work under supervision of a General Practitioner charged with monitoring and evaluating their work. GPA. Although no hypotheses were developed for students’ GPA, this criterion (average GPA across 7 years) was included by way of a comparison with prior studies. Given differences across universities, students’ GPA was standardized within university and academic year prior to making a composite.

Range Restriction In this study, students were selected on an operational composite of the various predictors, leading to indirect range restriction on each individual predictor. Given that indirect range restriction is a special case of multivariate range restriction, the multivariate range restriction formulas of Ree, Carretta, Earles, and Albert (1994) were applied to the uncorrected correlation matrix. As recommended by Sackett and Yang (2000), statistical significance was determined prior to correcting the correlations.

Results Table 1 presents the means, standard deviations, and correlations among the predictors. In Table 2, the correlations between the predictors and the criteria are shown. The values below the diagonal of Table 2 represent the corrected correlations between the predictors and performance. The values above the diagonal are the uncorrected correlations. Consistent with prior research, the corrected correlation between the cognitive composite and overall academic performance (GPA after 7 years) equaled .36. This was significantly higher than the corrected correlation (.15) between the SJT and GPA, t(720) ⫽ 4.31, p ⬍ .001. Our first hypotheses dealt with the validity of procedural knowledge about interpersonal behavior for predicting internship (H1a) and job performance (H1b). Table 2 shows that the corrected validities of the interpersonal SJT for predicting overall internship performance and supervisory-rated job performance were .22 and

.21. These results support H1a and H1b. Although the corrected correlations of the SJT with both criteria were higher than the corrected validities of the cognitive composite (.13 and .10, respectively), it should be noted that these differences were not statistically significant. H2 posited that a high-fidelity assessment of interpersonal behavior would mediate the relationship between a low-fidelity assessment of that behavior and job performance. All requirements for mediation according to Baron and Kenny (1986) were met (see Table 3). The effect of the interpersonal SJT on job performance dropped from .26 to .10 when the mediator (internship performance) was controlled for. To statistically test the mediating role of internship performance, we used the bootstrapping method (Preacher & Hayes, 2004; Shrout & Bolger, 2002). This was done on the uncorrected correlation matrix because it was not possible to run a bootstrapping procedure on a corrected matrix. Bootstrapping procedures showed a point estimate of .16 for the indirect effect of the interpersonal SJT on job performance through internship performance (95% CI [.04, .29]). Thus, H2 was supported. The third hypotheses related to the incremental validity of procedural knowledge about interpersonal behavior (as measured via an SJT) over cognitive factors for predicting internship and job performance. We conducted hierarchical regression analyses, with the matrices corrected for multivariate range restriction serving as input. The cognitive composite was entered as a first block because such tests have been traditionally used in admissions. As a second block, the medical text was entered. Finally, we entered the SJT. Table 4 shows the SJT explained incremental variance in internship (5%) and job performance (5%), supporting H3a and H3b. It should be noted, though, that these results were obtained with the other two predictors not being significant predictors of the criteria.

Discussion This study focused on the assessment of interpersonal skills via SJTs. Key results were that procedural knowledge about interpersonal behavior as measured with an SJT at the time of admission was valid for both internship (7 years later) and job performance (9 years later). Moreover, students’ procedural knowledge about interpersonal behavior showed incremental validity over cognitive factors for predicting these academic and postacademic success criteria, underscoring the role of SJTs as “alternative” predictors in

Table 1 Means, Standard Deviations, and Correlations Among Predictors in the Sample Selectees (n ⫽ 723)

Applicant population (N ⫽ 5,439) Variable 1. 2. 3. 4.

Cognitive composite Written text SJT Operational composite

M

SD

1

2

11.68 15.17 18.35 20.66

2.65 4.74 3.08 5.29

— .36 .20 .91

— .24 .45

General practitioners (n ⫽ 103)

3

M

SD

M

SD

— .28

14.08 16.81 19.30 24.90

1.67 4.47 2.84 3.89

13.60 16.30 20.15 24.63

1.47 4.16 2.66 4.04

Note. Although all analyses were conducted on standardized scores, this table presents the raw scores across exams. The maximum score on each test was 30, with the exception of the operational composite (maximum score ⫽ 40). Both the selectees (i.e., medical students) and general practitioners are subsamples of the applicant sample. Correlations between the predictors in the applicant group are presented. All correlations are significant at p ⬍ .01. SJT ⫽ situational judgment test.


465

Table 2 Correlations Among Predictors and Criteria in Selected Sample Variable Predictors (N ⫽ 723) 1. Cognitive composite 2. Written text 3. SJT 4. Operational composite Criteria 5. GPA (N ⫽ 713) 6. Internship performance (N ⫽ 606) 7. Job performance (N ⫽ 103)

1

2

3

4

5

6

7

— .13 .03 .86

⫺.01 — .15 .28

⫺.06 .13 — .16

.75 .19 .11 —

.27 .05 .10 .23

.09 .01 .21 .09

.13 ⫺.12 .21 .00

.36 .13 .10

.10 .03 ⫺.12

.13 .22 .21

.34 .14 .01

— .61 .37

.60 — .40

.38 .40 —

Note. Correlations were corrected for multivariate range restriction. Uncorrected correlations are above the diagonal, corrected correlations are below the diagonal. Apart from the last row, correlations higher than .09 are significant at the .05 level; correlations higher than .12 are significant at the .01 level. For the last row, correlations higher than .20 are significant at the .05 level; correlations higher than .26 are significant at the .01 level. SJT ⫽ situational judgment test; GPA ⫽ grade point average.

analyses confirmed that the link between a low-fidelity assessment of criterion behavior (i.e., procedural knowledge about interpersonal behavior via an SJT) and criterion ratings (job performance) was mediated by a high-fidelity (internship) assessment of criterion behavior. So, as a key theoretical contribution, we found support for a conceptual link between possessing procedural knowledge about interpersonal behavior, translating that knowledge into actual interpersonal behavior in constrained settings such as internships, and showing that interpersonal behavior later on the job. This study is also the first to establish evidence of the long-term predictive power of an interpersonal skills assessment via SJTs. That is, an operational SJT administered at the time of application for admission retained its validity many years later as a predictor

selection. These findings speak not only to the potential role of interpersonal skills assessment via SJTs in higher education admissions but also to its relevance in the employment world. A contribution of this study was that we tested conceptual arguments why the assessment of interpersonal skills via SJTs is predictive of future academic and postacademic performance. On the basis of the theory of knowledge determinants underlying SJT performance (Motowidlo & Beier, 2010; Motowidlo et al., 2006), we hypothesized that students’ procedural knowledge about costs and benefits of engaging in specific interpersonally oriented behavior that is assessed at the time of admission via an SJT will be a precursor of future actual behavior in interpersonal situations as encountered during internships and on the job. Our mediation Table 3 Regression Results for Mediation Variable Direct and total effects Equation 1 Dependent variable ⫽ Internship performance Independent variable ⫽ Interpersonal SJT Equation 2 Dependent variable ⫽ Job performance Independent variable ⫽ Interpersonal SJT Equation 3 Dependent variable ⫽ Job performance Independent variable ⫽ Internship performance Independent variable ⫽ Interpersonal SJT

Indirect effect and significance using normal distribution Sobel

Bootstrap results for indirect effect Effect

b

SE

t

p

.32

.08

3.85

.001ⴱⴱⴱ

.26

.12

2.19

.03ⴱ

.52 .10

.14 .12

3.80 0.81

.001ⴱⴱⴱ .42

Value

SE

LL 95% CI

UL 95% CI

z

p

0.17

0.06

.04

.29

2.66

.01

M

SE

LL 95% CI

UL 95% CI

0.16

0.06

.07

.28

Note. N ⫽ 103. Results are based on the uncorrected correlation matrix. Values were rounded. Unstandardized regression coefficients are reported. Bootstrap sample size ⫽ 1,000. SJT ⫽ situational judgment test; LL ⫽ lower limit; UL ⫽ upper limit; CI ⫽ confidence interval. Listwise deletion of missing data. ⴱ p ⬍ .05. ⴱⴱⴱ p ⬍ .01.

LIEVENS AND SACKETT

466

Table 4 Summary of Hierarchical Regression Analyses of Predictors on Internship and Job Performance Internship performance (N ⫽ 606)

Job performance (N ⫽ 103)

Model

Predictors

␤

R2

⌬R2

␤

R2

⌬R2

1 2 3

Cognitive composite Reading text SJT

.13ⴱ ⫺.02 .22ⴱⴱ

.02ⴱ .02 .06

.02 .00 .05ⴱⴱ

.12 ⫺.17 .23ⴱ

.01 .03 .08

.01 .02 .05ⴱ

Note. The corrected correlation matrix served as input for the regression analysis. Parameter estimates are for the last step, not entry. Due to rounding, ⌬R2 differs .01 from the cumulative R2. SJT ⫽ situational judgment test. ⴱ p ⬍ .05. ⴱⴱ p ⬍ .01.

of academic and postacademic criteria. This finding has theoretical implications as it informs the discussion of whether it is useful to select people on skills that will be taught to them later on. As noted in the introduction, there are a number of possible patterns of change in interpersonal skills over the course of one’s training. Attempts to select on the basis of interpersonal skills would be inappropriate if training eliminated variance (e.g., all were trained to a common level of skill) or resulted in a change in the rank ordering of individuals that was unrelated to initial skill level. Both of these are ruled out by the present findings, namely that predictive relationships still applied for interpersonal criteria that were gathered many years later (i.e., up to 9 years later after SJT administration). This is an important finding as most prior SJT studies were concurrent in nature or relied on predictors gathered after the first 6 months. Moreover, our results demonstrate that selecting on interpersonal skills early on is worthwhile. Training on them later on does not negate the value of selection. The challenge of broadening both criteria and predictors with noncognitive factors runs as a common thread through scientific and societal discussions about selection and admission processes. At a practical level, our results demonstrate that SJTs might be up to this challenge as they provide practitioners with an efficient formal method for measuring procedural knowledge of interpersonal skills early on. Accordingly, this study lends evidence for the inclusion of SJTs in formal school testing systems when decision makers have made a strategic choice of emphasizing an interpersonal skills orientation in their programs. Presently, one commonly attempts to assess (inter)personal attributes through mechanisms such as interviews or letters of recommendation or personal statements, whereas the formal system focuses only on academic achievement in science domains and specific cognitive abilities (Barr, 2010). Hereby it should be clear that measures such as SJTs are not designed to replace traditional cognitive predictors. Instead, they are meant to increase the coverage of skills not measured by traditional predictors. In particular, we see two possibilities for implementing SJTs as measures of procedural interpersonal knowledge in selection practice alongside cognitive skills assessment. One is to use the SJT at the general level of broad admissions exams to college. Another option is to implement the SJT at the level of school-specific additional screening of students who have met prior hurdles in the admission process. However, a caveat that qualifies these practical implications is in order. A contextual feature worthy of note of this study is that

to the best of our knowledge, there was no flourishing commercial test coaching industry in Belgium focusing on the SJT at the time of these cohorts (1999 –2002). At that time, coaching mostly focused on the cognitive part of the exam. In more recent years, commercial coaching programs related to the interpersonal component have also arisen, and it will be useful to examine SJT validity under this changed field context. So far, only laboratory studies on SJT coaching (Cullen, Sackett, & Lievens, 2006; Ramsay et al., 2003) have been reported (with results indicative of moderate effect sizes). Several limitations of this study should be mentioned. Like virtually all studies in the selection literature, this study reflects an examination of a single testing program in a single setting. We make no grand claims of generalizability; rather, we believe that it is useful to illustrate that an assessment of interpersonal skills via SJTs can predict performance during internships and in a subsequent work setting. Another limitation is the small sample size (N ⫽ 103) for the analysis of validity against job performance criteria. We also wish N were larger, but note that we are studying the entire population of these medical school graduates moving into general practice. The rarity of studies following individuals from school entry to subsequent job performance 9 years after administration of the predictor measure makes this a useful study to report, in our opinion, despite this limitation. Additional studies using this strategy are certainly needed before strong conclusions can be drawn. In terms of future research, we need more studies that integrate both education and work criteria as they provide a more comprehensive and robust view of the validity of admission/ selection procedures. Such research might provide important evidence to relevant stakeholders (e.g., students, admission systems, schools, organizations, general public) that the selection procedures used are valid for predicting both academic and job performance. In the future, the adverse impact of SJTs in student admissions should also be scrutinized. Along these lines, Schmitt et al. (2009) provided evidence that the demographic composition of students was more diverse when SJTs and biodata measures were used. However, that study was conducted in a research context. Therefore, studies in which the potential adverse impact reduction is examined via the use of noncognitive measures in actual admission and selection contexts are needed.


References Accreditation Council for Graduate Medical Education. (2010). ACGME general competencies and outcomes assessment for designated institutional officials. Retrieved from http://www.acgme.org/acWebsite/irc/ irc_competencies.asp Alvares, K. M., & Hulin, C. L. (1972). Two explanations of temporal changes in ability-skill relationships: A literature review and theoretical analysis. Human Factors, 14, 295–308. Arthur, W., Jr., Bennett, W., Jr., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 88, 234 –245. doi: 10.1037/0021-9010.88.2.234 Arthur, W., Jr., Day, E. A., McNelly, T. L., & Edens, P. S. (2003). A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56, 125–153. doi:10.1111/j.17446570.2003.tb00146.x Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182. doi:10.1037/0022-3514.51.6.1173 Barr, D. A. (2010). Science as superstition: Selecting medical students. The Lancet, 376, 678 – 679. doi:10.1016/S0140-6736(10)61325-6 Barrett, G. V., Phillips, J. S., & Alexander, R. A. (1981). Concurrent and predictive validity designs: A critical reanalysis. Journal of Applied Psychology, 66, 1– 6. doi:10.1037/0021-9010.66.1.1 Bore, M., Munro, D., & Powis, D. A. (2009). A comprehensive model for the selection of medical students. Medical Teacher, 31, 1066 –1072. doi:10.3109/01421590903095510 Callinan, M., & Robertson, I. T. (2000). Work sample testing. International Journal of Selection and Assessment, 8, 248 –260. doi:10.1111/ 1468-2389.00154 Campbell, J. P., & Knapp, D. J. (Eds.). (2001). Exploring the limits in personnel selection and classification. Mahwah, NJ: Lawrence Erlbaum Associates. Carpenter, T. D., & Wisecarver, M. M. (2004). Identifying and validating a model of interpersonal performance dimensions (ARI Technical Report No. 1144). Alexandria, VA: United States Army Research Institute for the Behavioral and Social Sciences. Caspi, A., Roberts, B. W., & Shiner, R. L. (2005). Personality development: Stability and change. Annual Review of Psychology, 56, 453– 484. doi:10.1146/annurev.psych.55.090902.141913 Ceci, S. J., & Papierno, P. B. (2005). The rhetoric and reality of gap closing: When the “have-nots” gain but the “haves” gain even more. American Psychologist, 60, 149 –160. doi:10.1037/0003-066X.60.2.149 Christian, M. S., Edwards, B. D., & Bradley, J. C. (2010). Situational judgment tests: Construct assessed and a meta-analysis of their criterionrelated validities. Personnel Psychology, 63, 83–117. doi:10.1111/ j.1744-6570.2009.01163.x Clause, C. S., Mullins, M. E., Nee, M. T., Pulakos, E., & Schmitt, N. (1998). Parallel test form development: A procedure for alternative predictors and an example. Personnel Psychology, 51, 193–208. doi: 10.1111/j.1744-6570.1998.tb00722.x Cullen, M. J., Sackett, P. R., & Lievens, F. (2006). Threats to the operational use of situational judgment tests in the college admission process. International Journal of Selection and Assessment, 14, 142–155. doi: 10.1111/j.1468-2389.2006.00340.x Deadrick, D. L., & Madigan, R. M. (1990). Dynamic criteria revisited: A longitudinal study of performance stability and predictive validity. Personnel Psychology, 43, 717–744. doi:10.1111/j.1744-6570.1990 .tb00680.x Ferris, G. R., Treadway, D. C., Perrewe´, P. L., Brouer, R. L., Douglas, C., & Lux, S. (2007). Political skill in organizations. Journal of Management, 33, 290 –320. doi:10.1177/0149206307300813 Fraley, R., & Roberts, B. W. (2005). Patterns of continuity: A dynamic

467

model for conceptualizing the stability of individual differences in psychological constructs across the life course. Psychological Review, 112, 60 –74. doi:10.1037/0033-295X.112.1.60 Ghiselli, E. E. (1956). Dimensional problems of criteria. Journal of Applied Psychology, 40, 1– 4. doi:10.1037/h0040429 Goldstein, I. L., Zedeck, S., & Schneider, B. (1993). An exploration of the job analysis-content validity process. In N. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations (pp. 2–34). San Francisco, CA: Jossey Bass. Huffcutt, A. I., Conway, J. M., Roth, P. L., & Stone, N. J. (2001). Identification and meta-analytic assessment of psychological constructs measured in employment interviews. Journal of Applied Psychology, 86, 897–913. doi:10.1037/0021-9010.86.5.897 Humphreys, L. G., & Taber, T. (1973). Postdiction study of the Graduate Record Examination and eight semesters of college grades. Journal of Educational Measurement, 10, 179 –184. doi:10.1111/j.17453984.1973.tb00795.x Klein, C., DeRouin, R. E., & Salas, E. (2006). Uncovering workplace interpersonal skills: A review, framework, and research agenda. In G. P. Hodgkinson & J. K. Ford (Eds.), International review of industrial and organizational psychology (Vol. 21, pp. 79 –126). New York, NY: Wiley & Sons. doi:10.1002/9780470696378.ch3 Kuncel, N. R., & Hezlett, S. A. (2007, February 23). Standardized tests predict graduate students’ success. Science, 315, 1080 –1081. doi: 10.1126/science.1136618 Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127, 162–181. doi:10.1037/0033-2909.127.1.162 Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2004). Academic performance, career potential, creativity, and job performance: Can one construct predict them all? Journal of Personality and Social Psychology, 86, 148 –161. doi:10.1037/0022-3514.86.1.148 Lievens, F., Buyse, T., & Sackett, P. R. (2005a). The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology, 90, 442– 452. doi: 10.1037/0021-9010.90.3.442 Lievens, F., Buyse, T., & Sackett, P. R. (2005b). Retest effects in operational selection settings: Development and test of a framework. Personnel Psychology, 58, 981–1007. doi:10.1111/j.1744-6570.2005.00713.x Lievens, F., & Patterson, F. (2011). The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. Journal of Applied Psychology, 96, 927–940. Lubinski, D., & Dawis, R. V. (1992). Aptitudes, skills, and proficiencies. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 3, 2nd ed., pp. 1–59). Palo Alto, CA: Consulting Psychologists Press. Lunneborg, C. E., & Lunneborg, P. W. (1970). Relations between aptitude changes and academic success during college. Journal of Educational Psychology, 61, 169 –173. doi:10.1037/h0029253 Makoul, G. (2001). Essential elements of communication in medicine encounters: The Kalamazoo Consensus Statement. Academic Medicine, 76, 390 –393. doi:10.1097/00001888-200104000-00021 McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & Grubb, W. L. (2007). Situational judgment tests, response instructions, and validity: A metaanalysis. Personnel Psychology, 60, 63–91. doi:10.1111/j.17446570.2007.00065.x Motowidlo, S. J., & Beier, M. E. (2010). Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test. Journal of Applied Psychology, 95, 321–333. doi:10.1037/a0017975 Motowidlo, S. J., Hooper, A. C., & Jackson, H. L. (2006). Implicit policies

468

LIEVENS AND SACKETT

about relations between personality traits and behavioral effectiveness in situational judgment items. Journal of Applied Psychology, 91, 749 – 761. doi:10.1037/0021-9010.91.4.749 Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, 187–207. doi:10.1037/0021-9010.89.2.187 Powis, D. (2010). Improving the selection of medical students. British Medical Journal, 340, 708. doi:10.1136/bmj.c708 Preacher, K. J., & Hayes, A. F. (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments, & Computers, 36, 717–731. doi:10.3758/ BF03206553 Ramsay, L. J., Gillespie, M. A., Kim, B. H., Schmitt, N., Oswald, F. L., Drzakowski, S. M., & Friede, A. J. (2003, November). Identifying and preventing score inflation on biodata and situational judgment inventory items. Invited presentation to the College Board, New York, NY. Ree, M. J., Carretta, T. R., Earles, J. A., & Albert, W. (1994). Sign changes when correcting for restriction of range: A note on Pearson’s and Lawley’s selection formulas. Journal of Applied Psychology, 79, 298 – 301. doi:10.1037/0021-9010.79.2.298 Roberts, B. W., & DelVecchio, W. F. (2000). The rank-order consistency of personality traits from childhood to old age: A quantitative review of longitudinal studies. Psychological Bulletin, 126, 3–25. doi:10.1037/ 0033-2909.126.1.3 Roth, P., Bobko, P., McFarland, L., & Buster, M. (2008). Work sample tests in personnel selection: A meta-analysis of Black–White differences in overall and exercise scores. Personnel Psychology, 61, 637– 661. doi:10.1111/j.1744-6570.2008.00125.x Sackett, P. R., & Yang, H. (2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85, 112–118. doi: 10.1037/0021-9010.85.1.112

Schmidt, F. L., Hunter, J. E., Outerbridge, A. N., & Goff, S. (1988). Joint relation of experience and ability with job performance: Test of three hypotheses. Journal of Applied Psychology, 73, 46 –57. doi:10.1037/ 0021-9010.73.1.46 Schmitt, N., Keeney, J., Oswald, F. L., Pleskac, T., Quinn, A., Sinha, R., & Zorzie, M. (2009). Prediction of 4-year college student performance using cognitive and noncognitive predictors and the impact of demographic status on admitted students. Journal of Applied Psychology, 94, 1479 –1497. doi:10.1037/a0016810 Schmitt, N., & Ostroff, C. (1986). Operationalizing the “behavioral consistency” approach: Selection test development based on a contentoriented strategy. Personnel Psychology, 39, 91–108. doi:10.1111/ j.1744-6570.1986.tb00576.x Schultz, M. M., & Zedeck, S. (2008). Identification, development, and validation of predictors of successful lawyering. Retrieved from http:// www.law.berkeley.edu/files/LSACREPORTfinal-12.pdf Sedlacek, W. E. (2004). Beyond the big test: Noncognitive assessment in higher education. San Francisco, CA: Jossey Bass. Shrout, P. E., & Bolger, N. (2002). Mediation in experimental and nonexperimental studies: New procedures and recommendations. Psychological Methods, 7, 422– 445. doi:10.1037/1082-989X.7.4.422 Taylor, P. J., Russ-Eft, D. F., & Chan, D. W. L. (2005). A meta-analytic review of behavior modeling training. Journal of Applied Psychology, 90, 692–709. doi:10.1037/0021-9010.90.4.692 Zhao, H., & Liden, R. C. (2011). Internship: A recruitment and selection perspective. Journal of Applied Psychology, 96, 221–229. doi:10.1037/ a0021295

Received April 8, 2011 Revision received July 18, 2011 Accepted August 25, 2011 䡲