How Do Respondents Construe Ambiguous ...

Journal of Personality and Social Psychology 2003, Vol. 85, No. 5, 956 –968

Copyright 2003 by the American Psychological Association, Inc. 0022-3514/03/$12.00 DOI: 10.1037/0022-3514.85.5.956

How Do Respondents Construe Ambiguous Response Formats of Affect Items? Susana Lloret Segura and Vicente Gonza´lez-Roma´

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

University of Valencia Ambiguous response formats predict correlations from ⫺.467 to ⫺1 between opposite items, depending on whether the respondent’s interpretation of the format is unipolar or bipolar. The authors present a procedure to investigate the proper interpretation in each case. It consists of applying nonparametric and parametric item response theory models (the Mokken and the graded response models) to pairs of opposite items in order to find the locations of the response options along the latent scale and, therefore, identify the response format construction. The authors tested this procedure on 4 samples (Ns ⫽ 142–1,150) and 2 item pairs (“relaxed”–“tense” and “optimistic”–“pessimistic”). The results revealed that respondents constructed the formats as bipolar and supported the bipolarity of the item pairs.

Diener’s (1999) claim about the structure of self-reports of affect clearly calls for new and more comprehensive lines of research. Probably, as he suggested, the key lies in the number of important empirical questions that have not been resolved yet. One of these unresolved empirical questions is the way respondents use the various types of scales and the effect this has on affect structure (Diener, 1999). Following this suggestion, this article analyzes two aspects of this question: (a) how respondents interpret response formats and (b) the effect of these interpretations on the correlations between opposite affects used in the evaluation of affect structure. The question of how people respond to items seems so obvious that when people’s responses do not follow the expected pattern, researchers tend to attribute this failure to the fact that items do not measure the expected trait or that the trait does not present the expected structure. The study by Remington et al. (2000) is a clear example of this reasoning. However, early studies in the field of personality inventories have shown that sometimes respondents fail to show the expected response patterns because researchers’ expectations, and not the response patterns, were incorrect. Recent contributions to the field of affect point in the same direction (Lloret & Gonza´lez-Roma´, 2000; Russell & Carroll, 1999a, 1999b; Yik, Russell, & Feldman Barrett, 1999). The early studies mentioned above investigated the acquiescence response style.2 In the 1950s and 1960s, the acquiescence response style (Cronbach, 1942, 1946, 1950) was threatening the validity of major personality inventories and attitude scales, such as the Minnesota Multiphasic Personality Inventory and the California F Scale (Adorno, Frenkel-Brunswik, Levinston, & Sanford, 1950). Many studies reported that when subjects were faced with the original and reversed forms of the same pool of items, most of them simultaneously endorsed or rejected both forms of the items (for a detailed exposition and a critical review, see Rorer, 1965).

The structure of affect has been the object of great debate for more than 40 years. Throughout these 4 decades, some controversies have been resolved and others have been forgotten, but the cornerstone of the polemic, that is, the bipolar or monopolar nature of the main dimensions of the affect structure, has remained the same. Diener (1999), in a recent special section on affect in the Journal of Personality and Social Psychology, summarized some of the points on which some consensus has been reached: that self-reports of affect fit an approximately circumplex model and that pleasantness– unpleasantness is an ever-present dimension in these reports. However, the results of recent research only “provide mixed support for contemporary versions of the circumplex model of affect” (Remington, Fabrigar, & Visser, 2000, p. 296), given that affective states are not consistently located as expected along the perimeter of the circumplex (see Figure 1), and correlations between opposite affects1 (that is, affects that are 180° apart from one another on the circle) are not as high as expected. In fact they range from ⫺.520 to ⫺.788, after removing random error of measurement (Remington et al., 2000, p. 296). In conclusion, the fit of the circumplex to affect data is not conclusive, and the correlations between supposed opposite affects are not high enough to support the bipolar conceptualization of affect or low enough to support the unipolar one. As Diener (1999) pointed out, “Despite a large amount of research in this area, the structure of self-reports of affect is still not fully understood” (p. 804).

Susana Lloret Segura and Vicente Gonza´lez-Roma´, Department of Methodology of Behavioral Sciences, Faculty of Psychology, University of Valencia, Valencia, Spain. This article is based on the doctoral dissertation of Susana Lloret Segura. We thank Wijbrandt van Schuur for his helpful comments on earlier stages of the research presented here. We also thank Abraham Buunk and Wilmar Schauffeli for their comments on a previous version of this article. This study was partially supported by Research Grant PB940997 from Direccioń General Interministerial de Ciencia y Tecnologıá. Correspondence concerning this article should be addressed to Susana Lloret Segura, Department of Methodology of Behavioral Sciences, Faculty of Psychology, University of Valencia, Av. Blasco Iban˜ez, 21, 46010, Valencia, Spain. E-mail: [email protected]

1

Bipolar affects are operationally defined as those affects 180° apart on the circle representing the structure of affect. See Russell and Carroll (1999a). 2 We thank one of the anonymous reviewers of this article for suggesting the inclusion of Rorer (1965). 956


AFFECT AND RESPONSE FORMAT CONSTRUCTION

Figure 1. Semantic model of the structure of affect. This figure was adapted with permission from the following two sources. From “Independence and Bipolarity in the Structure of Affect,” by L. Feldman Barrett and J. A. Russell, 1998, Journal of Personality and Social Psychology, 74, p. 970. Copyright 1998 by the American Psychological Association. From “On the Bipolarity of Positive and Negative Affect,” by J. A. Russell and J. M. Carroll, 1999, Psychological Bulletin, 125, p. 6. Copyright 1999 by the American Psychological Association.

This unexpected response pattern was interpreted as being consistent with the item wording instead of the item content. Thus, given that double rejections were more frequent than double endorsements, the term negativism as opposed to acquiescence was proposed (Christie, Havel, & Seidenberg, 1958; Peabody, 1961). However, the failure was not in subjects’ responses, and what seemed to be a pervasive bias in personality inventories was finally reinterpreted as a pervasive bias in researchers’ reasoning: “the great response-style myth” (Rorer, 1965). Rorer (1965) realized that “there is nothing at all inconsistent or contradictory about rejecting both the original and the reversing item” (p. 135). In fact, when carefully considered, the content of certain items (e.g., “Everybody must believe in God” and its logical reversal “Nobody must believe in God”) allows for these double rejections. Thus, Rorer uncovered the fact that some researchers operated under the following erroneous hypothesis: When a respondent is faced with an item and with its logical reversal, either the reversed item receives the reversed answer or the respondent is inconsistent with the item content, and, consequently, he or she is acquiescent. Therefore, low correlations between an item and its reversal are not always a sign of acquiescence. In light of item content, these correlations can be accounted for. Returning to the field of affect, the bipolarity hypothesis states that when a respondent is faced with an affect item (e.g., “Today I feel happy”) and with its polar opposite (“Today I feel sad”), either the opposite item receives the opposite score or both items are not bipolar opposites, and, consequently, their bipolarity is rejected. Analogous to Rorer’s (1965) insights, Russell and Carroll (1999a) claimed that both items may be rejected without compromising the bipolarity of the affects involved. These authors explained how item content and response format interaction can account for this fact. The key is in the item response format, which is often ambiguous. If the response option provided to reject the item “Today I feel happy” is simply 1. Not at all, then respondents are free to interpret this option as only meaning that they do not

957

feel the way the item indicates (that is, that they do not feel happy) or as meaning that what they feel is just the opposite (that is, sad). Russell and Carroll (1999a) called this response format ambiguous because it “leaves the respondent to define the anchors” (p. 8). They called the two aforementioned interpretations of this type of item response format strictly unipolar (see Figure 2A) and strictly bipolar (see Figure 2C), respectively.3 When the interpretation of the item response format of the item “Today I feel happy” and its polar opposite “Today I feel sad” is strictly unipolar, the rejection of both items will not contradict the bipolarity of the affects involved. Obviously, the correlation between the responses given to the item “Today I feel happy” and to its polar opposite “Today I feel sad” will differ according to the strictly unipolar or strictly bipolar interpretation of the response format. The theoretical correlation for the strictly unipolar format will approach ⫺.467 instead of ⫺1 (for more details, see Russell & Carroll, 1999a, 1999b), whereas the theoretical correlation for the strictly bipolar format will approach ⫺1. The theoretical correlation for other format interpretations that are neither strictly unipolar nor strictly bipolar (see Figure 2B) becomes “impossible to specify, except to say that it should fall between ⫺.467 and ⫺1.00. The more bipolar the format, the closer the correlation is to ⫺1.00” (Russell & Carroll, 1999a, p. 11). Analogously, the shape of the bivariate frequency distribution of responses to the item “Today I feel happy” and to its polar opposite “Today I feel sad” will differ according to the interpretation of the response format. The strictly unipolar format will yield L-shaped distributions of responses, the strictly bipolar format will yield linear and inverse distributions of responses, and the other format interpretations (neither strictly unipolar nor strictly bipolar) will yield triangular distributions of responses (Russell & Carroll, 1999a). An interesting amendment to the straightforward relationship between response format, distribution of responses, and correlations between opposite items was made in an earlier article (Lloret & Gonza´lez-Roma´, 2000). We pointed out that the empirical correlation between opposite items with strictly unipolar format can be higher than the theoretical value of ⫺.467, depending on the sample composition. The more polarized the sample (that is, the greater the number of subjects feeling extremely happy and extremely sad), the more negative the resulting correlation. This is because subjects feeling extremely happy will score high on the positive item and low on the negative item, whereas subjects feeling extremely sad will reverse these scores. In this case, the distribution of responses may still be L-shaped, but as the number of respondents located in the extremes of the distribution increases, so too will the negative correlation. In other words, the polarization of the sample increases the number of respondents endorsing an item and rejecting its opposite while it reduces the amount of 3

Notice that for unipolar response formats, “the positive item— happy—is being defined as a part of the full continuum, namely the part above neutral. Similarly, the negative item—sad—is defined as a part of the full continuum, namely the part below neutral” (Russell & Carroll, 1999a, p. 8). In the case of a bipolar response format, “the positive item is a dimension that extends all the way from the most extreme negative feeling through neutral to the most extreme positive feeling” (Russell & Carroll, 1999a, p. 7), and the reverse for the negative item.

´ LEZ-ROMA ´ LLORET SEGURA AND GONZA


958

Figure 2.

Relationship between the responses given to the item “happy” and to its polar opposite, “sad.”

respondents rejecting both items. All this will increase the observed correlation between both opposite items. In any case, once the impact of the response format has been considered, the expected correlation under the bipolarity hypothesis is no longer necessarily ⫺1. As Russell and Carroll (1999a, 1999b) illustrated, this is the expected correlation only for formats interpreted as strictly bipolar. And now it is clear that other interpretations make sense, too. When this is the case, correlations as low as ⫺.467 will not contradict the bipolarity of the affects involved. The importance of this finding for the defenders of the bipolar nature of affective dimensions is considerable, because it means that the low correlations that had been used to question this conceptual model can be accounted for within the model’s framework. All that is needed is a method to ascertain whether these low correlations appear between items whose response formats have been interpreted as unipolar. Or, in Russell and Carroll’s (1999a) own words, “the solution is research that establishes the properties of the response format used” (p. 11). For now, only the shape of the bivariate distribution of responses has been used to provide

support in this respect. However, better analytical methods are needed. Consequently, the relationship between response formats and correlations between opposite affects, although theoretically modeled and considered in some articles (Yik et al., 1999), remains formally untested. The main purpose of the current article is to propose some methods for carrying out this testing. In the present study, we demonstrate that the problem of the respondent’s construction of the ambiguous response format can be easily translated into the problem of the location of the item response options along the bipolar continuum. An ambiguous response format constructed as strictly unipolar is characterized by a particular location of the item response options: These response options tend to extend only from the neutral point to the positive or negative end of the continuum, depending on the item content (see Figure 2A). Two items like these, although bipolar opposites, will not overlap along the affective continuum. An ambiguous response format constructed as strictly bipolar is characterized by another particular location of the item response options: These response options tend to extend all the way from the negative end of the bipolar affective dimension to the positive end (see Figure



2C). Two opposite items like these will completely overlap along the affective continuum. Finally, less strict response formats will vary between the two former cases (an example is given in Figure 2B), so that the more bipolar the format construction, the more overlapping there is among response options. The key point here is that classical test theory methods cannot locate the response options on the underlying continuum, but item response theory (IRT) methods can.4 What is needed to deal with the response format is a psychometric model able to locate the response options of two items that are semantic opposites along a common underlying dimension. In the end, these locations will provide information about the respondents’ response format constructions. Once the problem of the item format is translated into the problem of the response option’s location along the underlying continuum, the analysis of item format becomes much simpler, and the evaluation of its impact on the observed correlation between pairs of opposite items becomes possible. In short, our first aim was to show how IRT models can be used to reveal how respondents construe ambiguous response formats. Our second aim was to investigate Russell and Carroll’s (1999a) ideas, which can be summarized in the following principle: When ambiguous response formats are in use, the correlation between opposite items will vary from ⫺.467 to ⫺1, depending on the response format construction, so that “the more bipolar the format, the closer the correlation is to ⫺1” (Russell & Carroll, 1999a, p. 11). We posit that the response format construction can be inferred from the overlapping among the response options of the items under study, as shown in Figure 2. This overlapping can be uncovered using IRT methods. Finally, we expected that this information would allow us to test the validity of Russell and Carroll’s principle.

Method Participants and Procedure This study was conducted across four independent Spanish samples. Samples 1 (N ⫽ 138) and 2 (N ⫽ 290) were obtained from the population of employees of a state administration agency who worked in two different cities. In both cases, the data-gathering procedure was the same. A battery of questionnaires, including the measures used in this study, was distributed among the employees by the agency’s technical personnel staff. Along with the battery, a letter from the research team guaranteeing anonymity and confidentiality and a letter from the top manager encouraging participation were distributed. The completed questionnaires were returned in a sealed envelope to the agency technical personnel staff. The response rate for Sample 1 was 45%. Fifty-three percent were women. The average age was 36.4 years (SD ⫽ 8.5). The response rate for Sample 2 was 43%. Fifty-seven percent were women. The average age was 35.6 years (SD ⫽8.6). Sample 3 was made up of 403 health professionals (family physicians, pediatricians, psychiatrists, psychologists, nurses, social workers, and auxiliary and administrative personnel) who worked in 33 health centers. In each center, one member of our research team organized the collective administration of a battery of questionnaires, including those used in this study. The response rate was 48.5%. Forty-three percent were men. The average age was 36.4 years (SD ⫽7.2). Sample 4 was made up of 1,150 university students. Data were gathered by means of collective administration of a battery of questionnaires, including those used in the present study. Participation was voluntary. Seventy-seven percent of participants were women. The average age was 22.2 years (SD ⫽ 5.2).

959

Measurement Instruments The items we analyzed present two different ambiguous response formats, one asking for the intensity of the affect described by the items and the other asking for their frequency. We considered these two formats because both are used in affect research, although intensity response scales tend to provide less support for the hypothesis of bipolarity than frequency response scales (Diener & Iran-Nejad, 1986). The respondents in Samples 1, 2, and 3 responded to the items from scales devised by Warr (1990) to measure the affective dimensions tense– calm and depressed– enthusiastic. Respondents were asked to think about the past few weeks and indicate to what extent they had been in the state indicated by the items. The response scale was 1 (never), 2 (occasionally), 3 (some of the time), 4 (much of the time), 5 (most of the time), and 6 (all of the time). Scores on the items that tapped the negative poles were reversed, so that the total score on each scale represented the level of positive affect in the corresponding affective dimension. The items on the tense– calm scale were “contented,” “calm,” “relaxed,” “tense,” “worried,” and “uneasy.” Those on the depressed– enthusiastic scale were “miserable,” “depressed,” “pessimistic,” “cheerful,” “optimistic,” and “enthusiastic.” Respondents in Sample 4 were asked about the intensity (or the strength) with which they experienced some affect words. They responded to the items of two scales that we devised to measure the same affective dimensions. Respondents were asked to think about the past few weeks and indicate to what extent they had been in the state indicated by the items. The response scale was 1 (Not at all), 2 (Just a little), 3 (A little), 4 (Somewhat), 5 (A lot), 6 (Quite a lot), and 7 (Entirely). Scores on the items that tapped the negative poles were reversed, so that the total score on each scale represented the level of positive affect in the corresponding affective dimension. The items on the tense– calm scale were “tranquil,” “calm,” “relaxed,” “tense,” “jittery,” and “anxious.” Those on the depressed– enthusiastic scale were “discouraged,” “sad,” “pessimistic,” “cheerful,” “optimistic,” and “lively.” All the scales were made up of items with ambiguous response formats. The criteria that we used for selecting the items analyzed in this study were that they should form pairs of semantic opposite items, like the usual “happy–sad” pair (Russell & Carroll, 1999a; Watson & Tellegen, 1999), and that they should be available in both scales. The pairs of opposite items common to both scales were “optimistic”–“pessimistic” and “tense”–“relaxed.”

Analysis First, we submitted data to the traditional analyses; that is, we examined the bivariate frequency distributions and calculated the correlations between each pair of opposite items. Next, following the recommendations of Stout (2001), we analyzed data using a combined nonparametric and parametric IRT approach: We first used the Mokken model (Mokken, 1971), a nonparametric IRT model for polytomous items (Molenaar, 1982, 1986, 1991), and then the GRM (Samejima, 1969, 1996), a parametric IRT model for polytomous items. We selected the GRM because in personality inventories the guessing parameter is meaningfulness (Chernyshenko, Stark, Chan, Drasgow, & Williams, 2001; Fraley, Waller, & Brennan, 2000). We selected the Mokken model for polytomous items because it is a nonparametric version of the GRM (Masters, 1982); that is, it is based on weaker assumptions than the GRM. As a result, (a) the Mokken model only yields ordinal

4

This strategy was used by Watson and Tellegen (1999). These authors applied the graded response model (GRM) to locate the response options of two unipolar items along the same continuum. But they only concluded that “two unipolar items clearly afford a more differentiated assessment across the bipolar continuum than either item could by itself” (Watson & Tellegen, 1999, p. 607).



960

locations of response options on the trait scale, whereas the GRM provides parameter estimates for these locations; and (b) the Mokken model tends to fit empirical data often, whereas GRM tends to fit empirical data less often. The advantage of using the Mokken model before the GRM model is that the former is a good tool for checking the validity with which the more stringent GRM can be applied to data (Stout, 2001). Besides, if the GRM showed a poor fit to data, the Mokken model would be a tool for having, at least, ordinal information about the locations of response options on the trait scale. The standard Mokken model is defined by three assumptions: (a) unidimensionality, which means that all test items measure the same latent trait so that persons are represented by one scalar parameter; (b) local independence, which states that item responses, given a specific latent trait level, are independent; and (c) monotonicity of the cumulative option response function (ORF), which is the function relating a person’s probability of responding above a particular option to the trait level. The ORF must be at least nondecreasing as the latent dimension increases; that is, the probability of responding above a particular response option cannot decrease as the latent dimension increases (for a detailed exposition on these topics, see Hambleton & Swaminathan, 1985). The standard GRM model is defined by the three aforementioned assumptions plus a particular mathematical function for the ORFs (Samejima, 1969, 1996). That is, the GRM fully parameterizes the shape of the ORFs (for a more detailed exposition, see Fraley et al., 2000). In the present study, Mokken analyses were carried out using the MSP program (Molenaar, Debets, Sijtsma, & Hemker, 1994; Molenaar & Sijtsma, 2000), and GRM analyses were carried out using MULTILOG (Thissen, 1991).

Results The analysis of the bivariate distributions of responses offers an initial idea about the way in which respondents have constructed our ambiguous response formats. No sample showed the L shape that would suggest a unipolar construction of the response format (Figures 3A and 3B show the most L-shaped bivariate distributions, obtained in Sample 1). On the contrary, two distributions clearly showed the linear shape congruent with a bipolar construction of the response format (see Figures 4A and 4B, obtained in Samples 4 and 3, respectively). The remaining distributions were only moderately linear shaped. The correlations between each item pair (see Table 1) offer the same picture: they are low in the two samples that present the most L-shaped bivariate distributions (⫺.39 for Sample 1 on both the item pairs “optimistic”–“pessimistic” and “relaxed”–“tense”), medium in the four samples with moderately linear-shaped distributions (⫺.438 and ⫺.497 for Samples 2 and 3 on the item pair “optimistic”–“pessimistic” and ⫺.494 and ⫺.593 for Samples 2 and 4 on the item pair “relaxed”–“tense”), and higher in the two samples with clearly linear-shaped distributions (⫺.70 for Sample 4 on the item pair “optimistic”–“pessimistic” and ⫺.614 for Sample 3 on the item pair “relaxed”–“tense”). Up to this point, only the results yielded by two samples seemed to support the bipolarity of the pairs of affects tested here (Sample 4, item pair “optimistic”–“pessimistic,” and Sample 3, item pair “relaxed”–“tense”). Clearly, these results are contradictory with the bipolarity of the affects involved. Russell and Carroll (1999a) pointed out that if the response format is adequately introduced in the analysis, then correlations as low as ⫺.467 do not contradict the bipolarity of affects. Does this principle explain these contradictory results? Surprisingly, our results do not point in this direction. According to the bivariate distributions obtained, the samples

that show correlations between the pairs of items approaching ⫺.467 (Samples 2 and 3 on the item pair “optimistic”– “pessimistic” and Samples 2 and 4 on the item pair “relaxed”– “tense”) do not appear to have interpreted the response formats as unipolar. We guess that this could be the case even for the other samples showing the lowest correlations (Sample 1 in both pairs of items). To verify this hypothesis, we needed to ascertain how respondents actually constructed the involved response formats. IRT analysis can provide information about the locations of the response options along the latent scale and, therefore, about the response format construction. First, we applied the Mokken model to each pair of opposite items in each sample (recall that this nonparametric IRT model serves to check the basic assumptions of the GRM). The scalability coefficient, H (for more information on this coefficient, see Molenaar, 1996), varied from medium (.4 ⱕ H ⬍ .5) to strong (.5 ⱕ H ⱕ 1), indicating that the scales were unidimensional enough to continue the analysis. Visual inspection of the ORFs as well as the count of violations of the monotonicity assumption revealed monotonic increasing ORFs. These results showed that the Mokken model adequately fit our data and that Samejima’s GRM could be meaningfully applied to them. In each sample, we tested two nested GRM models. In the first model (Model 1), all the item parameters were freely estimated without imposing any equality constraint. In the second model (Model 2), the discrimination parameters of the two items composing each pair were constrained to be equal. Table 2 shows the goodness-of-fit values obtained for these models. The chi-square statistic for Model 1 in Sample 1 (the smallest sample) was statistically nonsignificant for both the “relaxed”–“tense” item pair, ␹2(23, N ⫽ 138) ⫽ 20.5, p ⬎ .05, and for the “optimistic”– “pessimistic” item pair, ␹2(23, N ⫽ 138) ⫽ 18.3, p ⬎ .05. However, the analysis of the parameter estimates yielded by Model 1 in Sample 1 revealed some problems. Some of the parameter estimates showed extreme values with such large standard errors that MULTILOG omitted them. Model 2 presented the same problem of extreme parameter values in this small sample. Therefore, in this sample we decided to consider the results obtained with the Mokken model instead of those provided by the GRM. In the remaining samples, Models 1 and 2 showed an acceptable or very good fit to data, as indicated by the chi-square/degrees of freedom values observed, which ranged from 2.6 to 1.22 (Bock, 1996; Chernyshenko et al., 2001; Drasgow, Levine, Tsien, Williams, & Mead, 1995). The comparison between Models 1 and 2 revealed that the constriction of equal discrimination parameters did not increase model misfit, except for the “optimistic”–“pessimistic” item pair in Sample 2. Therefore, Model 2, the model assuming the invariant ordering of the response options, was selected as the best-fitting model. In Figures 5 and 6, we present the location parameter estimates for the response options (bjk) and the discrimination parameter estimates (aj) yielded by Model 2 in each sample for the pairs of items “tense”–“relaxed” and “optimistic”–“pessimistic.” (Recall that for Sample 1, only the ordering of the response options under the Mokken model was available, and notice that the scalable response options are those above 1 for each item.) The discrimination parameter estimates (aj) revealed that the items were highly discriminating across samples and for both the frequency and intensity response formats. However, the most noticeable result, depicted in Figures 5 and 6, stems from the



Figure 3. The most L-shaped bivariate distributions. Percentages do not include decimals. A: Items “pessimistic”–“optimistic,” Sample 1. B: Items “tense”–“relaxed,” Sample 1.

961


962 ´ LEZ-ROMA ´ LLORET SEGURA AND GONZA

Figure 4. The most linear-shaped bivariate distributions. Percentages do not include decimals. A: Items “pessimistic”–“optimistic,” Sample 4. B: Items “tense”–“relaxed,” Sample 3.


963

Table 1 Means, Standard Deviations, and Correlations Between Opposite Pairs of Items Optimistic–pessimistic

Sample 1 2 3 4

(n (n (n (n

⫽ ⫽ ⫽ ⫽


Note.

138) 290) 403) 1,150)

Optimistic

Pessimistic

M

SD

M

SD

r

M

SD

M

SD

r

3.73 3.55 3.72 3.97

1.21 1.24 1.19 1.34

2.13 2.23 2.38 3.22

0.99 1.01 1.04 1.60

⫺.392 ⫺.438 ⫺.497 ⫺.700

3.50 3.67 3.64 2.98

1.22 1.12 1.23 1.21

2.65 2.61 2.91 4.36

1.00 0.95 1.03 1.36

⫺.392 ⫺.494 ⫺.614 ⫺.593

Table 2 Goodness-of-Fit Statistics Obtained for the Two Nested Graded Response Models ␹2

df

␹2/df a

⌬␹2

“Optimistic”–“pessimistic” item pair Sample 1 (N Model 1 Model 2 Sample 2 (N Model 1 Model 2 Sample 3 (N Model 1 Model 2 Sample 4 (N Model 1 Model 2

⫽ 138) ⫽ 290) ⫽ 403) ⫽ 1,150)

18.3 18.3

23 24

0.79 0.76

0

52.2* 57.4*

23 24

2.26 2.39

5.2*

42.6* 42.6*

23 24

1.85 1.77

0

71.3* 71.3*

35 36

2.03 1.98

0

“Relaxed”–“tense” item pair Sample 1 (N Model 1 Model 2 Sample 2 (N Model 1 Model 2 Sample 3 (N Model 1 Model 2 Sample 4 (N Model 1 Model 2

⫽ 138) ⫽ 290) ⫽ 403) ⫽ 1,150)

Relaxed

Tense

All the correlations are statistically significant, p ⬍ .05.

location parameter estimates for the response options (bjk) of each pair of items (and the ordering of the response options in Sample 1): As we suspected, they clearly support a bipolar construction of the response formats. These results indicate that the response format construction does not depend on the sample or the frequency–intensity response format. It depends only on the affects involved. Now that we know the way respondents have constructed the formats used in this study, we are in a position to examine the validity of the Russell and Carroll (1999a) principle. Recall that

Sample and model

Relaxed–tense

20.5 21.2

23 24

0.89 0.88

0.7

29 29.4

23 24

1.26 1.22

0.4

47.7* 47.7*

23 24

2.07 1.98

0

91.3* 91.3*

35 36

2.60 2.53

0

a 2 ␹ /df ⬎ 3.0 indicates model– data misfit; ␹2/df ⬍ 3.0 indicates an acceptable fit for the model; ␹2/df ⬍ 2.0 indicates an excellent fit (Bock, 1996; Drasgow et al., 1995). * p ⬍ .05.

our formulation of this principle involves two assertions related to items with ambiguous response formats. The first is that correlations between such opposite items will vary between ⫺.467 and ⫺1, depending on the response format construction, and the second is that the more bipolar the format construction, the closer the correlation is to ⫺1. Our results provide empirical support for only this last assertion. Overall, visual inspection of the IRT analysis results shows a general trend: The greater the overlapping among the response options of the items under analysis (and thus, the more bipolar the response format construction), the closer the correlation is to ⫺1. However, the first assertion is not supported. Correlations between opposite items do not vary between⫺.467 and ⫺1, depending on the response format construction. According to our results, under a bipolar format construction, correlations between opposite items may vary between ⫺.392 and ⫺.70. So what is causing the variability among these correlation coefficients? There is another aspect apart from the response format construction that can shift correlations toward unexpected values. This aspect is sample composition5 (Lloret & Gonza´lezRoma´, 2000). Recall that polarized samples will shift toward ⫺1 the theoretical correlation of ⫺.467 expected for unipolar constructions of opposite items. Is the opposite situation possible? That is, can unpolarized samples shift the theoretical correlation of ⫺1 expected for bipolar constructions of opposite items toward zero? That is what we analyze below, but first we need to define sample polarization. We define sample polarization according to the percentage of respondents located in each of the extremes of the affective continuum, that is, highly optimistic and highly pessimistic (or highly relaxed and highly tense). The percentage of subjects on the “positive” extreme is calculated considering subjects who endorse the two most positive response options of the positive item (6. All of the time optimistic/relaxed or 5. Most of the time optimistic/ relaxed) and also the two lowest response options of the negative item (1. Never pessimistic/tense or 2. Occasionally pessimistic/ tense). Analogously, the percentage of subjects on the “negative” extreme is calculated considering subjects who endorse 6. All of the time pessimistic/tense or 5. Most of the time pessimistic/tense and also 1. Never optimistic/relaxed or 2. Occasionally optimistic/

5

We thank one of the anonymous reviewers of this article for suggesting the inclusion of this aspect in our analysis.


964


Figure 5. Results of item response theory analyses: Mokken ordering (Sample 1) and graded response model parameter estimates (bjk and aj; Samples 2– 4) obtained for the item pair “relaxed”–“tense.” In Samples 1–3, T2–T6 are the response options for “tense” from T2 (most of the time) to T6 (never), whereas R2–R6 are the response options for “relaxed” from R2 (occasionally) to R6 (all of the time). In Sample 4, the response options for “tense” vary from T2 (a lot) to T7 (not at all), and the response options for “relaxed” vary from R2 (just a little) to R7 (entirely). Dotted lines show the overlap among the response options of the items under study.

relaxed.6 The higher the percentages of respondents located in each of the extremes of the affective continuum, the greater the sample polarization. The two samples showing the most linear-shaped distributions and the highest negative correlations (Sample 4, item pair “optimistic”–“pessimistic,” and Sample 3, item pair “relaxed”– “tense”) correspond to the most polarized samples, whereas the samples showing the most L-shaped distributions and the lowest correlations correspond to the least polarized samples (item pair “optimistic”–“pessimistic” and item pair “relaxed”–“tense,” both in Sample 1). Specifically, the percentages of subjects on the positive and negative extremes for the most polarized samples are 10.26% and 4.69% (Figure 4A: Sample 4, item pair “optimistic”–

“pessimistic”) and 19.10% and 4.21% (Figure 4B: Sample 3, item pair “relaxed”–“tense”), whereas the percentage of subjects on the 6

Notice that the degree of sample polarization is independent of the response format construction. The degree of polarization depends on the number of respondents feeling highly pessimistic combined with the number of respondents feeling highly optimistic (or tense and relaxed). These subjects will endorse all of the time/most of the time optimistic and never/occasionally pessimistic or the reverse, whatever the bipolar or unipolar construction of the response format. Bipolar and unipolar response formats lead to different response patterns mainly with respect to the respondents located in the middle of the dimension—under unipolar formats they will reject both items—not with respect to the respondents located in the extremes.



965

Figure 6. Results of item response theory analyses: Mokken ordering (Sample 1) and graded response model parameter estimates (bjk and aj; Samples 2– 4) obtained for the item pair “optimistic”–“pessimistic.” In Samples 1–3, P2–P6 are the response options for “pessimistic” from P2 (most of the time) to P6 (never), whereas O2 to O6 are the response options for “optimistic” from O2 (occasionally) to O6 (all of the time). In Sample 4, the response options for “pessimistic” vary from P2 (a lot) to P7 (not at all), and the response options for “optimistic” vary from O2 (just a little) to O7 (entirely). Dotted lines show the overlap among the response options of the items under study.

positive and negative extremes for the least polarized samples approaches 25% and does not reach 2%, respectively (Figure 3A: Sample 1, item pair “optimistic”–“pessimistic”; Figure 3B: Sample 1, item pair “relaxed”–“tense”). These samples are monopolarized in the sense that a noticeable percentage of respondents are located in the positive extreme of the affective continuum, whereas a negligible percentage are located in the negative one. If in these samples the observed distributions do not show the linear-shaped distribution or the high correlations observed in other samples, it is probably due to a lack of respondents feeling extremely pessimistic or tense, not due to the response format construction. The GRM estimates are also affected by the degree of sample polarization. As the sample considered becomes less homogeneous and some polarization appears, the discrimination parameter esti-

mates (aj) become higher (which means that the corresponding items are more informative in that sample), and the location parameter estimates for the extreme response options become more reasonable. Both item pairs illustrate this. In the “pessimistic”–“optimistic” pair, the percentages of respondents on the negative and positive extreme of the affective continuum were 1.70% and 24.80%; 1.90% and 25.30%; and 4.69% and 10.26%, respectively, for Samples 2, 3, and 4. The discrimination parameter estimates (aj) were 1.74, 2.07, and 2.98, respectively. Finally, the location parameter estimates for the extreme response option 5. Most of the time pessimistic (P2) were ⫺4.15 (Sample 2), ⫺3.52 (Sample 3), and ⫺2.39 (Sample 4). As Hambleton and Swaminathan (1985) have pointed out, some heterogeneity in the sample is required to attain proper estimations of the parameters. In light



966

of this aspect, even the improper estimates that led us to reject this model in Sample 1 may be partially explained. This sample was the smallest one and, in addition, showed the most monopolarized response distribution (1.44% vs. 24.00%). Probably these were not the only factors involved in the unreasonable parameter estimates obtained, but certainly they contributed. The results obtained show that testing bipolarity is not as straightforward as commonly expected and that it cannot rely on correlation coefficients. In the present study, we have provided empirical evidence supporting Russell and Carroll’s (1999a) suggestion that “respondents might readily reinterpret ostensibly unipolar formats as if they were bipolar” (p. 8). However, our results support another neglected aspect in the context of the bipolarity tests: that even in this case, the correlation between two bipolar opposite affects may still be low unless the sample is polarized to some degree.

Discussion The aim of this study was to show how IRT models can be used to explore how respondents construe ambiguous response formats of affect items and to ascertain whether the impact of this construction on correlations between opposite items follows the principle stated by Russell and Carroll (1999a): low negative correlations for unipolar constructions and high negative correlations for bipolar constructions. We believe that the results given above clearly show the usefulness of this procedure in resolving both questions. IRT analysis has revealed that the response format construction of the items analyzed here is bipolar and that bipolar constructions do not always produce high negative correlation coefficients between semantic opposite affects (180° apart). The procedure proposed here contributes to clarifying the response format construction by translating the problem of respondents’ construction of ambiguous response formats into the problem of response options’ location along the affective continuum. Bipolar and unipolar response format constructions lead to different response options’ locations along the affective continuum. The response options’ locations can be estimated using the adequate IRT model. Here we have used a nonparametric IRT (NIRT) model, the Mokken model for polytomous items; and then a parametric IRT (PIRT) model, Samejima’s (1969, 1996) GRM. We used the Mokken model with a two-fold purpose: (a) to check the validity with which the GRM could be applied to our data and (b) to obtain useful information about the degree of overlapping between items in case of lack of fit of the GRM (which happened in Sample 1). Both models pointed to the same conclusion: the bipolar construction of the response format of the two pairs of semantic opposite items considered (“relaxed”–“tense” and “optimistic”–“pessimistic”). This result was consistent across different samples of respondents and across frequency and intensity response scales. Overall, these results illustrate that Russell and Carroll’s (1999a) ideas relating correlations and response formats do not always apply. On the one hand, we have found support for their assertion that the more bipolar the format construction, the closer the correlation is to ⫺1. On the other hand, our results show that low correlations between opposite items do not necessarily imply a unipolar format construction. In fact, there is a noticeable factor that moderates this apparently straightforward relation. This factor is the sample composition. Correlations and sample composition

are related regardless of the item format construction. On the grounds of a logical analysis, we noticed in a previous study (Lloret & Gonza´lez-Roma´, 2000) that polarized samples will lead to high negative correlations between opposite affects even for items with unipolar response formats. Now, on the grounds of the empirical evidence found, we have shown that monopolarized samples will lead to low correlations between such affects, even in the case of items with a bipolar response format construction. The results obtained show that restriction of range operates in the expected way. It is well known that correlations are strongly affected by restriction of range, shifting correlations toward zero. The items that we have analyzed vary in their degree of range restriction across samples. Accordingly, the correlations computed vary in magnitude across samples, too. Obviously, bipolarity tests should not rely on such correlations. There are other results to comment on. First, we have not found differences between the way respondents construe the response options of items with a frequency response scale and with an intensity response scale. Contrary to what Diener and Iran-Nejad (1986) suggested, subjects tend to construct both ambiguous response formats as bipolar. Second, we have not found response formats constructed as unipolar. This aspect limits the scope of our study. One possible explanation may be that, as Russell and Carroll (1999a) noticed, “respondents might readily reinterpret ostensibly unipolar formats as if they were bipolar” (p. 8). Further research should provide information about the validity of this principle. However, if this explanation were true, then another explanation going beyond response format construction would arise: that ambiguous response formats are interpreted as bipolar because respondents use a mental representation of affect that is bipolar. As one reviewer suggested, the category locations may reflect the true state of affairs instead of the bipolar–monopolar interpretation given by respondents to the response format. Faced with ambiguous affect items, respondents may project their own representations of affect structure. If this were the case, our results would support the idea that this representation is bipolar. Future research should analyze other ambiguous response formats in order to explore this alternative explanation. Finally, what are the implications that arise in light of these findings? First, we have shown that IRT models can be used to ascertain the response format constructed by respondents when faced with ambiguous items; that is, we have shown how IRT methods can be used to answer substantive questions. In this respect, IRT models can avoid “the widespread use of measuring instruments with properties that are not understood” (Russell & Carroll, 1999a, p. 26) by increasing understanding of their properties and how people respond to them. Further research should explore this new approach in the field of affect. Second, this study illustrates how NIRT and PIRT models can be used in a sequential way, from the less restrictive to the more restrictive models (see Chernyshenko et al., 2001). This sequential strategy may restore “the enthusiasm for IRT [that] has been tempered by the realization that the validity with which these methods can be applied to realistic data sets (e.g., small numbers of items and examinees, multidimensional data) is often poorly documented” (Ip, 2001, p. 109). NIRT models can be useful for developing a deeper analysis of the validity of PIRT models and for providing useful information when a PIRT model does not fit the data (as in our Sample 1). Third, this new approach has shed some light on some important aspects of self-reports of affect: (a) Respondents tend to construe



ambiguous response formats as bipolar; (b) bipolar constructions of ambiguous response formats do not guarantee high correlation coefficients between items that are semantic opposites, unless some polarization is present in the sample; and (c) low to medium correlation coefficients between items that are semantic opposites do not invalidate the bipolar structure of affect in favor of a monopolar one. This study presents some limitations. First, it was based on extended affect (the past few weeks). Future research should address the questions investigated here using momentary affect data. Second, we have only investigated a small number of opposite affects corresponding to two affective dimensions and two different ambiguous unipolar response formats. Future research should ascertain whether the same findings are obtained when different pairs of opposite affects are studied and different ambiguous response formats are used. Other limitations of this study come from the unusual design used. Analyzing scales composed of only two items is not very frequent. In some sense, the strength of this design, which stems from the fact that it focuses only on pairs of opposite items, is just what makes it weak, too. Neither PIRT models nor NIRT models were conceived for analyzing such extremely short scales. Consequently, some of the existing tests for the assumptions of our models were not applicable, and some of these assumptions (especially local independence, which can affect the location of categories) remain formally untested. However, we have tested as many assumptions as possible, and all of them were reasonably satisfied. Therefore, we believe that the IRT models tested were reasonably valid, and the conclusions derived from their estimates are reasonably tenable.

Conclusion Russell and Carroll (1999a) called for progress “on how respondents interpret ambiguous formats” and for “better methods to test the models developed” (p. 26). Here we propose a new approach, an approach based on IRT models, to uncover respondents’ construction of ambiguous response formats. This new approach contributes by testing bipolar relationships between items that are answered on ambiguous response formats, the formats usually used in self-reports on affect. In summary, our results question the principle formulated by Russell and Carroll (1999a) for explaining the variability observed in the correlations between opposite affect items. In this study, the key factor does not seem to be the response format construction. Instead, our data suggest a corollary of our ideas (Lloret & Gonza´lez-Roma´, 2000) on sample polarization as the factor acting here. We noticed in the earlier study that even with response formats constructed as unipolar, the greater the sample polarization, the higher the negative correlation between items referring to affects that are bipolar opposite. The aforementioned corollary could be formulated as follows: Even for response formats constructed as bipolar, the less the sample polarization, the lower the negative correlation between items referring to affects that are bipolar opposites. Consequently, we cannot rely only on correlation coefficients. When carefully appraised, the bipolarity of items correlating as low as ⫺.39 cannot be rejected.

References Adorno, T. W., Frenkel-Brunswik, E., Levinston, D. J., & Sanford, R. N. (1950). The authoritarian personality. New York: Harper.

967

Bock, R. D. (1996). The nominal categories model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 33– 49). New York: Springer Verlag. Chernyshenko, O. S., Stark, S., Chan, K. Y., Drasgow, F., & Williams, B. (2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36, 523– 562. Christie, R., Havel, J., & Seidenberg, B. (1958). Is the F scale irreversible? Journal of Abnormal and Social Psychology, 56, 143–159. Cronbach, L. J. (1942). Studies of acquiescence as a factor in true–false tests. Journal of Educational Psychology, 33, 401– 415. Cronbach, L. J. (1946). Response sets and test validity. Educational and Psychological Measurement, 6, 475– 494. Cronbach, L. J. (1950). Further evidence on response sets and test design. Educational and Psychological Measurement, 10, 3–31. Diener, E. (1999). Introduction to the special section on the structure of emotion. Journal of Personality and Social Psychology, 76, 803– 804. Diener, E., & Iran-Nejad, A. (1986). The relationship in experience between various types of affect. Journal of Personality and Social Psychology, 50, 1031–1038. Drasgow, F., Levine, M. V., Tsien, S., Williams, B., & Mead, A. D. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19, 143–165. Feldman Barrett, L., & Russell, J. A. (1998). Independence and bipolarity in the structure of affect. Journal of Personality and Social Psychology, 74, 967–984. Fraley, C. R., Waller, N. G., & Brennan, K. A. (2000). An item response theory analysis of self-report measures of adult attachment. Journal of Personality and Social Psychology, 78, 350 –365. Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff. Ip, E. H. (2001). Testing for local dependency in dichotomous and polytomous item response models. Psychometrika, 66, 109 –132. Lloret, S., & Gonza´lez-Roma´, V. (2000). Medicioń de constructos bipolares mediante escalas tipo Likert: ¿por que´ aparecen factores monopolares? [Measurement of bipolar constructs with Likert-type scales: Why do monopolar factors appear?]. Psicothema, 12, 471– 479. Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149 –174. Mokken, R. J. (1971). A theory and procedure of scale analysis. The Hague, the Netherlands: Mouton. Molenaar, I. W. (1982). Mokken scaling revisited. Kwantitatieve Methoden, 8, 145–164. Molenaar, I. W. (1986). Een vingeroefening in item response theorie voor drie geordende antwoordcategorieen [An exercise in item response theory for three ordered response categories]. In G. F. Pikkemaat & J. J. A. Moors (Eds.), Liber amicorum Jaap Muilwijk (pp. 39 –57). Groningen, the Netherlands: Econometrisch Instituut. Molenaar, I. W. (1991). A weighted Loevinger H-coefficient extending Mokken scaling to multicategory items. Kwantitatieve Methoden, 37, 97–117. Molenaar, I. W. (1996). Nonparametric models for polytomous responses. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 368 –380). New York: Springer Verlag. Molenaar, I. W., Debets, P., Sijtsma, K., & Hemker, B. T. (1994). User’s manual MSP, a program for Mokken Scale analysis for polytomous items (Version 3.0). Groningen, the Netherlands: ProGAMMA. Molenaar, I. W., & Sijtsma, K. (2000). User⬘s manual MSP5 for Windows: A program for Mokken Scale analysis for polytomous items (Version 5.0). Groningen, the Netherlands: ProGAMMA. Peabody, D. (1961). Attitude content and agreement set in scales of authoritarianism, dogmatism, antisemitism, and economic conservatism. Multivariate Behavioral Research, 17, 221–251. Remington, N. A., Fabrigar, L. R., & Visser, P. S. (2000). Reexamining the


968


circumplex of affect. Journal of Personality and Social Psychology, 79, 286 –300. Rorer, L. G. (1965). The great response-style myth. Psychological Bulletin, 63, 129 –156. Russell, J. A., & Carroll, J. M. (1999a). On the bipolarity of positive and negative affect. Psychological Bulletin, 125, 3–30. Russell, J. A., & Carroll, J. M. (1999b). The phoenix of bipolarity: Reply to Watson and Tellegen (1999). Psychological Bulletin, 125, 611– 617. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores [Psychometric Monograph No. 18]. Iowa City, IA: Psychometric Society. Samejima, F. (1996). Graded response model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 85–100). New York: Springer Verlag. Stout, W. (2001). Nonparametric item response theory: A maturing and applicable measurement modeling approach. Applied Psychological Measurement, 25, 300 –306.

Thissen, D. (1991). Multilog user’s guide: Multiple, categorical item and test scoring using item response theory (Version 6. 0) [Computer software]. Chicago: Scientific Software. Warr, P. (1990). The measurement of well-being and other aspects of mental health. Journal of Occupational Psychology, 63, 193–210. Watson, D., & Tellegen, A. (1999). Issues in the dimensional structure of affect-effects of descriptors, measurement error, and response format: Comment on Russell and Carroll (1999). Psychological Bulletin, 125, 601– 610. Yik, M. S., Russell, J. A., & Feldman Barrett, L. (1999). Structure of self-reported current affect: Integration and beyond. Journal of Personality and Social Psychology, 77, 600 – 619.

Received March 12, 2002 Revision received February 10, 2003 Accepted February 14, 2003 䡲