Structural Equation Modeling

This article was downloaded by: [Chinese University of Hong Kong] On: 08 April 2014, At: 19:35 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Structural Equation Modeling: A Multidisciplinary Journal Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hsem20

Comparing Squared Multiple Correlation Coefficients Using Structural Equation Modeling a

b

Joyce L. Y. Kwan & Wai Chan a

The Hong Kong Institute of Education

b

The Chinese University of Hong Kong Published online: 04 Apr 2014.

To cite this article: Joyce L. Y. Kwan & Wai Chan (2014) Comparing Squared Multiple Correlation Coefficients Using Structural Equation Modeling, Structural Equation Modeling: A Multidisciplinary Journal, 21:2, 225-238, DOI: 10.1080/10705511.2014.882673 To link to this article: http://dx.doi.org/10.1080/10705511.2014.882673

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http:// www.tandfonline.com/page/terms-and-conditions

Structural Equation Modeling: A Multidisciplinary Journal, 21: 225–238, 2014 Copyright © Taylor & Francis Group, LLC ISSN: 1070-5511 print / 1532-8007 online DOI: 10.1080/10705511.2014.882673

Comparing Squared Multiple Correlation Coefficients Using Structural Equation Modeling Joyce L. Y. Kwan1 and Wai Chan2 1

Downloaded by [Chinese University of Hong Kong] at 19:35 08 April 2014

2

The Hong Kong Institute of Education The Chinese University of Hong Kong

In social science research, a common topic in multiple regression analysis is to compare the squared multiple correlation coefficients in different populations. Existing methods based on asymptotic theories (Olkin & Finn, 1995) and bootstrapping (Chan, 2009) are available but these can only handle a 2-group comparison. Another method based on structural equation modeling (SEM) has been proposed recently. However, this method has three disadvantages. First, it requires the user to explicitly specify the sample R2 as a function in terms of the basic SEM model parameters, which is sometimes troublesome and error prone. Second, it requires the specification of nonlinear constraints, which is not available in some popular SEM software programs. Third, it is for a 2-group comparison primarily. In this article, a 2-stage SEM method is proposed as an alternative. Unlike all other existing methods, the proposed method is simple to use, and it does not require any specific programming features such as the specification of nonlinear constraints. More important, the method allows a simultaneous comparison of 3 or more groups. A real example is given to illustrate the proposed method using EQS, a popular SEM software program. Keywords: squared multiple correlation coefficients, structural equation modeling, model reparameterization, multi-sample analysis

In multiple regression analysis, the sample squared multiple correlation coefficient, R2 , measures the proportion of total variance on the dependent variables, Y, that is accounted for by a set of predictors, X1 , X2 , . . . , Xk . It provides an estimate for the overall predictive power of a set of predictors. Social sciences researchers are often interested in comparing R2 in different model conditions because this helps them to better understand the relative importance of a given set of predictors. One typical comparison is to compare R2 of a given regression model in different independent samples. This comparison allows researchers to decide whether a given set of predictors performs equally well in different populations. For example, an educational psychologist might ask if parental involvement, teacher’s influence, and personality traits together provide the same degree of predictive power on academic performance of different groups of students (e.g., boys and girls). Correspondence should be addressed to Wai Chan, Department of Psychology, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong. E-mail: [email protected]

Different methods have been proposed for comparing the squared multiple correlation coefficients in two populations. One common approach is Olkin and Finn’s (1995) asymptotic confidence interval (ACI) for δ = ρ12 − ρ22 , the difference between two population squared multiple correlation coefficients. Given R21 and R22 are the sample estimates of ρ12 and ρ22 obtained from large independent samples of sizes N1 and N2 , respectively, d = R21 − R22 is asymptotically distributed as a normal variate and the 100(1 – α)% ACI for δ is

d − z(1− α2 ) σˆ , d + z(1− α2 ) σˆ ,

(1)

where z(1− α2 ) is the 100 1 − α2 th percentile point of the standard normal distribution, and σˆ =

4 2 4 2 2 2 R1 (1 − R21 ) + R (1 − R22 ) N1 N2 2

(2)

is the asymptotic standard error estimate (ASE) of d. Alf and Graf (1999) also developed formulas for the ACI of

226

KWAN AND CHAN

δ in different cases. Although they approach the problem somewhat differently, their derived formula for the ASE of d in independent samples is the same as Equation 2. In addition to the construction of a confidence interval, the ASE can be used to obtain a standard hypothesis test of H0 : ρ12 = ρ22 (Algina & Keselman, 1999). Olkin and Finn (1995) proposed a second method for testing H0 : ρ12 = ρ22 by using a transformation FZ for R21 and R22 based on Fisher’s Z variance-stabilizing transforma tion for correlation coefficients; that is, FZ R2 = zero-order loge

√ 1+√R2 1− R2


by using FZ

. A test statistic for H0 : ρ12 = ρ22 can be obtained R2 : Z=

FZ (R21 ) − FZ (R22 ) . 4 4 + N1 N2

(3)

Given large samples, this statistic is asymptotically distributed as a standard normal variate under H 0 (Olkin & Finn, 1995). Nevertheless, it has been argued that using the standard normal as the asymptotic sampling distribution of the test statistic in Equation 3 is not appropriate. Gajjar (1967) shows that the limiting distribution of FZ (R2 ) fails to approach normality as sample size increases. As the squared multiple correlations only cover the range from 0 to 1, the transformed values, FZ (R2 ) are severely truncated in the lower tail. Hence, the resulting distribution is positively skewed (Alf & Graf, 1999). Based on Monte Carlo results, Algina and Keselman (1999) showed that Olkin and Finn’s ACI method fails to perform accurately in certain model conditions. This is because, first, the ASE in Equation 2 might not be accurate enough when sample sizes are small. Second, the method relies on the normal approximation to provide the percentile point, z(1− α2 ) , which might not be accurate enough either (Chan, 2009). Instead, the method might perform better if the percentile point is empirically estimated from the observed data rather than using the normal approximation. As a result, Chan (2009) proposed a unified bootstrap procedure for estimating the standard error of d and constructing the bootstrap percentile interval (BP) and bootstrap standard interval (BCI) for δ (Efron & Tibshirani, 1993). Simulation results indicate that the performance of the asymptotic method and bootstrap procedure, in general, are comparable to each other with normal data. However, when data are nonnormal, the bootstrap procedure is more robust against nonnormality and it outperforms its asymptotic counterpart (Chan, 2009). It is obvious that both Olkin and Finn’s (1995) asymptotic methods and Chan’s (2009) bootstrap method provide standard ways for comparing ρ 2 in two populations. However, these methods fail to generalize multiple-group comparisons because both the hypothesis test and the confidence interval are based on d, or the function of it, and hence they are limited to testing the hypothesis, H0 : ρ12 = ρ22 . As pointed

out by Chan (2009), a challenging research question is “to explore the possibility of generalizing the existing method to the case of multiple-group comparison” (p. 583). In fact, there are many situations in which researchers might need to test the equality of ρ 2 in three or more groups. For example, in the meta-analytic study using regression, one has to first decide whether the effect sizes (R2 ) from different primary studies are homogeneous when pooling the data. In other words, one has to test the equality of ρ 2 across the studies before doing any subsequent analysis. Similarly, in a clinical setting, a clinical psychologist might be interested in knowing if the effectiveness of cognitive behavior therapy can be explained by factors such as treatment type, duration and intensity of treatment, and mode of treatment equally well across patients with different types of anxiety disorders. The psychologist thus has to test the equality of ρ 2 in groups of patients with panic disorder, generalized anxiety disorder, obsessive–compulsive disorder, and posttraumatic stress disorder. To conclude, although there is a practical need for researchers to make a multiple-group comparison, such a method is still not readily available. The aim of this study is, therefore, to provide a general solution for testing H0 : ρ12 = ρ22 = . . . = ρm2 (with m ≥ 2). Specifically, we are to propose a method for comparing squared multiple correlations in different groups by using the structural equation modeling (SEM) technique. Before going into the details of the proposed method, we first review the existing methods of using SEM as a tool for comparing ρ 2 in two groups. COMPARING ρ 2 USING SEM SEM is a flexible and powerful statistical technique for studying a variety of models and it is continually gaining popularity among applied researchers (Hershberger, 2003; Tremblay & Gardner, 1996). Methodologists have studied how different models can be formulated in SEM; for example, regression analysis (Bentler, 1995; Jöreskog & Sörbom, 1996), canonical correlation (Fan, 1997), multivariate analysis of variance (MANOVA; Cole, Maxwell, Arvey, & Salas, 1993), and meta-analytic SEM (Cheung & Chan, 2005), to name a few. In addition, the development of user-friendly and powerful software programs allows researchers to perform different SEM analyses with ease. The flexibility of SEM and advancement of software programs enable SEM to be a useful tool for comparing squared multiple correlations in different groups. Phantom Variable Approach Cheung (2009) proposed a general method for constructing the confidence intervals for a number of statistics and psychometric indexes including R2 with the use of phantom variables under the SEM framework. A phantom variable is a latent variable without observed indicators and has no


SQUARED MULTIPLE CORRELATION COEFFICIENTS

residual. It usually has no substantive meaning but is created to parameterize constraints on the model (Rindskopf, 1984). Many SEM programs nowadays have functions for creating new parameters, which help to simplify the model specification involving phantom variables (e.g., AP Keyword in LISREL [Jöreskog & Sörbom, 1996] and NEW option in Mplus [Muthén & Muthén, 2007]). By defining a new parameter as the difference of R2 between two groups, users can readily obtain the parameter estimate, its standard error (SE), and the Wald statistic (cf. Cheung, 2009). Hence, statistical inferences about the equality of ρ 2 between two groups can be made. Although the phantom variable approach provides a method for comparing ρ 2 in two groups in SEM, this method has several limitations. First, the success of the implementation of the method depends entirely on one’s ability to write out the functional expression of R2 . The method requires the user to explicitly specify the new parameter; that is, the difference of R2 between two groups, as a function in terms of the basic SEM model parameters. This procedure can be troublesome and error prone. Applied researchers might have difficulties implementing the method because they have to fully understand the functional relationship among the model parameters before they are able to specify the new parameter correctly. Second, the method requires the specification of nonlinear constraints because the new parameter is defined as a nonlinear function of the basic model parameters. Consequently, it cannot be used with some popular SEM software programs such as EQS (Bentler, 1995) and AMOS (Arbuckle, 2007), which do not support the specification of nonlinear constraints. Third, the method is primarily for a two-group comparison. In fact, this approach is basically equivalent to the statistical test using the ASE of d in Olkin and Finn’s (1995) ACI method. In addition, the phantom variable approach is the same as Chan’s (2009) bootstrap method if one constructs the BCI or BP for the new parameter, d. In other words, like other existing methods, the phantom variable approach also fails to deal with the comparison of ρ 2 in three or more groups. It might be possible to extend the phantom variable method to the comparison of three or more groups simultaneously. Nevertheless, this depends critically on the SEM program’s capacity to provide an overall test for the new parameter concerned. As far as we know, most SEM programs fail to perform such an overall test (Kwan & Chan, 2011). Therefore, applied researchers still lack a method of making a multiple-group comparison of ρ 2 . In this article, we use the flexibility of SEM on model specification and propose a general method for comparing the squared multiple correlations across different groups. Unlike the existing SEM methods, which depend critically on the special features (e.g., specification of nonlinear constraint) supported by a specific class of SEM software programs, the implementation of the proposed method does not require any advanced programming features but the

227

basic functions available in almost all SEM programs on the market. More important, the proposed method is readily able to conduct multiple-group comparisons, whereas the currently existing methods fail to do so. The proposed method is introduced in the next section. A real example that illustrates the implementation of the proposed method using EQS (Bentler, 1995) is given. In addition, a simulation study is conducted to compare the performance of different methods for comparing ρ 2 in different groups.

THE PROPOSED METHOD The proposed method compares the squared multiple correlation coefficients by using model reparameterization, which is the process in which the hypothesized model is transformed into a set of successive covariance-equivalent models that share the same implied covariance matrix as the original model so that a coefficient that does not exist as a model parameter in the original model becomes a model parameter in the final transformed model (Chan, 2007). Chan’s (2007) sequential model-fitting method for comparing the indirect effects and Kwan and Chan’s (2011) two-stage approach for comparing standardized coefficients in SEM demonstrate some of the applications of the model reparameterization technique. In this article, we further extend the usage of the model reparameterization technique on the study of squared multiple correlation coefficients. Similar to the method proposed by Kwan and Chan (2011), the proposed method is a two-stage approach for comparing ρ 2 across groups. At Stage 1, the original model (M1) of each group is transformed into the R2 model (M2) so that the squared multiple correlation coefficient, which does not appear as a free model parameter in the original model, becomes one of the model parameters in the transformed model. Then at Stage 2, the ρ 2 in different groups is compared by imposing linear between-group constraint(s) on the parameters of interest in M2 and statistical inference is performed by using the likelihood ratio (LR) test. The following section gives a detailed description of the model transformation at Stage 1. General Framework of Model Transformation at Stage 1 We first define the model of theoretical interest. Suppose we want to fit a regression model with k predictors, X1 , X2 , . . . , Xk in m groups as in the model shown in Figure 1a. We label this model as the original model, M1. Let γi (i = 1, 2, . . ., k) be the unstandardized regression coefficient of the predictor, Xi ; ij be the covariance between X i and X j ; and E be the error term. Without loss of generality, all variables are assumed to have zero means. Let us consider the model equation of the original model for group p (p = 1, 2, . . . , m) in a standardized form; that is,

228

KWAN AND CHAN

(a)

(b)


(c)

FIGURE 1 A general regression model for group m with k predictors on Y. (a) Original model (M1). (b) Half-transformed model. (c) Final R2 model (M2). k

g(θ ) = φii + 2 (ai )(aj )φij in panel c. Parameters in parentheses are fixed parameters. i=j

i=1

Y=

k

γi Xi + E

i=1

Y = SD(Y)

k i=1

Xi SD(Xi ) γi × SD(Y) SD(Xi )

(4)

+

k

γ ∗i 2 + 2

i=1

where γi∗ = D=

SD(Xi ) γ; SD(Y) i

E SD(Y)

γi∗ γj∗ ∗ij + var(D)

(5)

i =j

∗ij = corr(Xi , Xj ) =

ij ; SD(Xi )SD(Xj )

The squared multiple correlation coefficient of group p is,

By putting Equation 5 into Equation 6 we have

γ ∗i 2 + 2

γi∗ γj∗ ∗ij

(7)

i=j

Figure 1b shows the half-transformed model of M1. Each observed variable is regressed on a dummy latent variable1 (DLV) such that (k + 1) DLVs, F1, F2, . . . , Fk, and FY are added. The function of these DLVs is to manipulate the variance of the variables so as to reparameterize the model. Factor loadings, λi from Fi to Xi are free to be estimated. In addition, the path coefficient from Fi to FY is fixed at ai depending on the sign of γ i ∗ . If γ i ∗ > 0, then ai = 1; otherwise, ai = –1. The model equation for FY is as follows:

and

E . SD(Y)

ρp2 = 1 − var(D)

k i=1

Xi Y Because var SD(Y) = var SD(X = 1, taking variance i) on both sides of Equation 4 gives us 1=

ρp2 =

(6)

1 The function of DLV is similar to Rindskopf’s concept of a phantom variable. However, unlike Rindskopf’s (1984) original definition of a phantom variable, which is a latent variable with no observed indicators, a DLV has other variables loaded on it. Chan (2007) used the term DLV to denote the variable that is used to factorize the original mediator in the sequential model fitting method. We use the term DLV here because the latent variables in the transformed model have the original observed variables loaded on them.


FY =

k

ai Fi + D

i=1

and the variance of FY is, var(FY) = var

k

ai Fi + D

i=1

=

k

φii + 2

(ai )(aj )φij + var(D)

(9)

i =j

i=1

= g(θ ) + var(D)


var(FY) = g(θ) + (1 − g(θ)) = 1

(8)

i =j

i=1

D = F999 + FR − 0.5FR

(11)

Hence, g(θ ) is equivalent to the ρp2 of the original model, M1, and more important, it appears as a single model parameter; that is, the covariance between FR and FR’ in M2. To summarize, three important criteria need to be observed when one performs model transformation. First, the variance of FY in M2 must be fixed at 1.0 nonstochastically. Second, the ρp2 , which is defined by g(θ), must appear as a single model parameter in M2. Third, M2 must have the same implied covariance structure as the original model. Once the original model is successfully transformed into the R2 model at Stage 1, it is straightforward for us to compare the ρ 2 in different groups by using the LR test at Stage 2.

where øii is the variance of Fi; øij is the covariance between Fi and Fj; θ is a vector of unknown model parameters; k

φii + 2 (ai )(aj )φij is defined as the total and g(θ) = variance and covariance due to the antecedent variables. If var(FY) is fixed at 1, Equation 9 will be equivalent to γ ∗ γ ∗ ∗ given that and φ = Equation 5 with φii = γ ∗2 ij i i j ij ai = 1 or –1 and it always shares the same sign as γ i ∗ . Therefore, ρp2 of the original model will be the same as g(θ) in this transformed model. Hence, the rest of the task is to further reparameterize the model such that g(θ) appears as a single model parameter in the final transformed model and at the same time fix the variance of FY at 1.0. To do this, we further transform the model into the final R2 model, M2 (as shown in Figure 1c) by regressing the disturbance term, D on a phantom variable, F999 with unit variance and a pair of phantom variables, FR and FR.’ We label the structure formed by FR and FR’ as the R2 structure. The R2 structure has the following four properties: (a) both the variances of FR and FR’ are fixed at 0; (b) the path coefficient from FR to D is fixed at 1; (c) the path coefficient from FR’ to D is fixed at –0.5; and (d) the covariance between FR and FR’ is constrained to be equal to g(θ ). Notice that g(θ ) is simply a linear function of the model parameters and hence the proposed method does not require the specification of any nonlinear constraints. In addition, the variance of F999 and the path leading from F999 to D are fixed at 1.0. Because var(F999) = 1; var(FR) = var(FR’) = 0; and the covariance between FR and FR’ is constrained to be equal to g(θ ), the variance of D becomes

229

A REAL EXAMPLE A real example based on the data from the Organization for Economic Cooperation and Development (OECD) Programme for International Student Assessment (PISA) 2006 (OECD, 2009)2 is given to illustrate the proposed method using EQS (Bentler, 1995). Readers who are interested in working on this example can refer to Appendix A for the EQS program codes. Cases from three countries— Hong Kong, Japan, and Korea—are selected from the PISA 2006 data set on four variables. They are, namely, general interest in science (V1), enjoyment of science (V2), science self-efficacy (V3), and science ability (V4). The sample sizes are 4,625 for Hong Kong, 5,943 for Japan, and 5,151 for Korea. Table 1 summarizes the sample covariance matrices of the variables. Stage 1: Model Specification of M1 and Model Transformation Into M2 A regression model with three predictors is considered. Specifically, we compared if general interest in science (V1), enjoyment of science (V2), and science self-efficacy (V3) as a set of predictors about students’ views on science provided the same degree of predictive power on students’ science ability (V4) in Hong Kong (HKG), Japan (JPN), and Korea (KOR). Therefore, we tested the hypoth2 2 2 = ρJAP = ρKOR . Figure 2a shows the original esis, H0 : ρHKG model, M1. A multisample analysis was performed using EQS6.1 for Windows. Because M1 is a saturated model with 0 degrees of freedom, it has a perfect fit with model

var(D) = var(F999) + var(FR)

+ (−0.5) var(FR ) + 2(1)(−0.5)g(θ) 2

(10)

var(D) = 1 − g(θ) By substituting Equation 10 into Equation 9, we have the variance of FY fixed at 1.0 nonstochastically:

2 The

PISA is a triennial international assessment of 15-year-old school children’s capabilities of reading literacy, mathematics literacy, and science literacy. The database can be freely accessed through the PISA Web page (www.pisa.oecd.org). In this article, data from PISA collected in 2006 is used. PISA 2006 has been administered in 57 countries or economies (OECD, 2009). We have selected cases from three countries—namely Hong Kong, Japan, and Korea—from the PISA 2006 data, and four variables have been used in this article.

230

KWAN AND CHAN


TABLE 1 Sample Covariance Matrices for Hong Kong, Japan and Korea from Programme for International Student Assessment 2006 Data (Organization for Economic Cooperation and Development, 2009)

Hong Kong (N = 4,625) V1 V2 V3 V4 Japan (N = 5,943) V1 V2 V3 V4 Korea (N = 5,151) V1 V2 V3 V4

General Interest in Science (V1)

Enjoyment of Science (V2)

Science Self-Efficacy (V3)

Science Ability (V4)

0.9451 0.6006 0.4326 27.3990

0.7977 0.3779 28.2320

0.8956 31.2722

8176.0021

1.0453 0.6926 0.5027 34.2501

1.0777 0.4524 35.2189

0.9583 30.6472

9666.8658

0.9271 0.6338 0.4088 31.6266

1.0007 0.3902 37.3062

0.8031 30.9021

8187.6921

(a)

(b)

FIGURE 2 Sequence of models in Programme for International Student Assessment (PISA) example. (a) Original model (M1). (b) R2 Model (M2). V1 = general interest in science, V2 = enjoyment of science, V3 = science self-efficacy, V4 = science ability. ø67 = ø11 + ø22 + ø33 + 2ø12 + 2ø13 + 2ø23 .

chi-square, χ 2 = 0. The estimated R2 coefficients for Hong Kong (R2HKG ), Japan (R2JPN ), and Korea (R2KOR ) are .178, .163, and .224, respectively. We followed the general framework to transform M1 into the R2 model, M2 (Figure 2b). Each observed variable was regressed on a DLV (F1 to F4). As the standardized regression coefficients from V1, V2, and V3 to V4 (i.e., γ1∗ , γ2∗ and γ3∗ ) were all positive in M1, the paths from F1,

F2, and F3 to F4 were all fixed at ai = 1.0 (i = 1, 2, and 3). F5 was the disturbance term of F4.3 F999 was the phantom variable with variance of 1.0, and the path from F999 to

3 In EQS, the disturbance term can appear only as an exogenous variable,

which cannot be regressed on other variables. To solve this problem, we renamed the disturbance term as F5, an ordinary latent factor.


231

TABLE 2 Summary of the R 2 Estimates and Their Estimated Standard Errors of the Original Model (M1) in Programme for International Student Assessment (PISA) Example by Analysis Using Different Structural Equation Modeling Programs and Approaches Program

EQS 6.1

Mplus 5.2

LISREL8.8

Method

Reparameterization

Built-In

Phantom Variable

R2 / φ 67 Hong Kong Japan Korea

Est

SE

Est

SE

Est

SE

0.178 0.163 0.224

0.010 0.009 0.010

0.178 0.163 0.224

0.010 0.009 0.010

0.178 0.163 0.224

0.010 0.009 0.010


Note. Est. = parameter estimate.

TABLE 3 Models Summary in the Programme for International Student Assessment (PISA) Example Models M1 M2 Unconstrained Constrained under 2 2 = ρ2 H0 : ρHKG = ρJAP KOR 2 2 H0 : ρHKG = ρJAP 2 2 H0 : ρJAP = ρKOR

φ 67 (HKG)

φ 67 (JAP)

φ 67 (KOR)

χ2

Df

p

.178 (—)

.163 (—)

.224 (—)

0

0

—

.178 (.010)

.163 (.009)

.224 (.010)

0

0

—

.187 (.006) .170 (.007) .178 (.010)

.187 (.006) .170 (.007) .191 (.007)

.187 (.006) .224 (.010) .191 (.007)

2 1 1