Structural equation model with PLS path modeling for an integrated ...

Jointly published by Akadémiai Kiadó, Budapest and Springer, Dordrecht

Scientometrics Scientometrics, Vol. 81, No. 3 (2009) 683–698 DOI: 10.1007/s11192-009-2058-7

Structural equation model with PLS path modeling for an integrated system of publicly funded basic research JIANCHENG GUAN,a NAN MAb* a

b

School of Management, Fudan University, 200433 Shanghai, P.R. China School of Management, Beijing University of Aeronautics and Astronautics, 100083 Beijing, P.R. China This study develops and tests an integrated conceptual model of basic research evaluation from a varying perspective. The main objective is to obtain a more complete understanding of the external factors affecting the publicly fund basic research in a country. Structural Equation Modeling (SEM) with Partial Least Squares (PLS) is used to test the conceptual model with empirical data collected from WCY (World Competitiveness Yearbook) and ESI (Essential Science Indicators) database. Interrelationships among the research output and outcome, together with three external factors (resource, impetus, accumulative advantage) have been successfully explored and the conceptual model of journal evaluation has been examined.

Introduction Science permeates all aspect of daily life. It is nowadays apparent science and technology has become one of the most critical parts for the development of a country. Increasingly, however, the role of science in the policy making has been problematic as the assumed status of basic research as a public good in governmental decision making has been challenged by scientists, policy makers, and the public. Thus it is necessary to shed light on the whole picture of publicly funded basic research; thereby setting up the right direction of the national policies and corresponding financial support in a country. A common analytic theme runs through several researches [ADAMS, 1990, 1996; EVENSON & KISLEV, 1975; GRILICHES, 1964; HUFFMAN & EVENSON, 1994] on the evaluation of basic research in the form of linear production. Scholars concerned with the input and output aspects of basic research rely (implicitly or explicitly) on a sense that the academic research system is kind of isolated from the economic, industrial and social systems. At the macroeconomic arena, however, there have been several attempts to measure the social rates of return on investment in basic research by linking basic research with industrial change [MANSFIELD, 1991, 1992; GRILICHES, 1995]. Later on, DAVID & AL. [1992[ proposed an alternative approach based on information theory, as * The authors’ names are alphabetically ordered and they contributed equally to this paper. Received December 22, 2008 ; Published online March 18, 2009 Address for correspondence: JIANCHENG GUAN E-mail: [email protected], [email protected], [email protected] 0138–9130/US $ 20.00 Copyright © 2009 Akadémiai Kiadó, Budapest All rights reserved

GUAN & MA: Structural equation model of publicly funded basic research

they found that the “cost-benefit” mode may tend to focus mainly on “success” and has limitations on measuring all the essential complementary factors implied in basic research. MARTIN [1996] raised the multiplicity of basic research with respect to its nature and outputs, and further suggested that necessity of using multiple indicators in the measurement of basic research performance. The existing approaches, however, have a simple structure and do not consider the hierarchical relationship among various factors affecting the performance of basic research. Moreover, insufficient attentions have been given to the importance of the externalities and multiple aspects of spill-over effect for publicly funded basic research. What is novel in this study about the basic research’s indirect engagement with economy and society is the combination of these distinct functional systems into a single operational regime. Importing external factors and interrelationships into the evaluation of basic research is precisely transformative because this movement alters the simple production logics and allocation regimes that define traditional basic research. The main objective of this study is to develop a structural equation model for the evaluation of publicly funded basic research. We use the IMD survey statistics and ISI publications by 46 countries and areas as empirical lens to examine the influential factors involved in the public science system. We draw on a period of 2000–2004 of IMD surveys and other statistical data to examine the direct and indirect linkages between factors implied in the system. These factors include governmental inputs, environmental driving forces, accumulative advantage, scientific publications and several aspects of achievements which represent the spillovers of research efforts. With the proposed SEM, one can find various indices which can be used as barometers of the output and outcome of public science due to national funding and some driving forces. Therefore our results are expected to give valuable and controllable feedback information to improve the effectiveness of government funding and public policy making. Research model The basic research activity can be seen as one of major stages in innovation process from resource inputs to research and development outputs [OECD, 2005]. However, it’s inevitable that the performance of basic research is influenced by environment factors, (refereed here to as impetus factors) and existing resources stocks (referred to as accumulative advantages). For example, WANG & HUANG [2007] investigated the production efficiency from R&D inputs to outputs accounting environmental factors, such as the proportion of highly educated population, the density of personal computers and the academic English proficiency. Basic research is defined by the National Science Foundation (NSF) as “original investigation for the advancement of scientific

684

Scientometrics 81 (2009)


knowledge... which do[es] not have immediate commercial objectives” [MANSFIELD, 1980]. However, the research outputs do have an effect on cultural, educational and economic environments directly or indirectly. It can directly improve the level of theories and methodologies in culture and education fields. Besides, by the scientific innovation based on the basic researches, many valuable innovative ideas which help design patents, production system or service process can be devoted to the development of economy indirectly. CHAVES & MORO [2007] showed the significant interaction and mutual dependence between science (papers) and technology (patents). YANG & CHUNG [1996] indicated that basic scientific researches can yield a synergy effect with domestic innovation in the dynamic process of assimilation, absorption, improvement, and indigenization of the technologies imported from technologically advanced countries. SEM has been increasingly used as a powerful quantitative technique for specifying, estimating, and testing hypothesized models describing relationships among a set of meaningful variables [SOHN & AL., 2007; YOON & AL., 2001]. The structural equation model for the evaluation of publicly funded basic research is shown in Figure 1. The factors used and their relationship in the proposed SEM are explained in the detail in the next subsection.

Figure 1. The conceptual framework of the evaluation system for basic research

Factors We begin with descriptive evidence of a pattern of publicly funded basic research, which is characterized by the resources devoted to research activity, environmental


685


driving forces, quantitative and qualitative measurement of basic research, and multiaspects of spillover outcomes in the economy, society and industry, together with suggestive of accumulative advantage across prior achievements and current scheme of resource allocation. The resource factor is composed of two aspects, viz. governmental budget and research personnel. Implicitly, GPRA (Governmental Performance and Results Act) treats government spending as inputs to the basic research that eventually produce benefits for the public. Several previous studies have also attempted to relate the statistical information about the number of scientific publications and expenditure on R&D with GDP growth. The research on 15 countries in Latin American conducted by the International Bank for Reconstruction and Development (IBRD) reveals that there is a direct relationship among these factors. It concluded that the high level of GDP growth is due to the importance of the production and diffusion of knowledge. Moreover, the amount of R&D expenditures and the interaction between institutions can promote innovative activities [INTERNATIONAL BANK FOR RECONSTRUCTION AND DEVELOPMENT, 1999]. With respect to the research personnel, economists always treated human capital as an important source of knowledge production process. ROMER [1990] use human capital as an input for research activity to produce new knowledge. LUCAS [1988] places human capital as a direct input into the production process. The output, therefore, is directly dependent on human capital [AGHION & HOWITT, 1992]. Government inputs are intended to produce both direct output and related outcome. Direct measures of scientific output such as publication counts and the associated citation measures have generated a whole research field of bibliometrics [VAN RAAN, 1988; ELKANA & AL., 1978]. On the one hand, publication count is by far the most widely used metric of basic research output, finding applications from individual scientists’ evaluation to national science indicators [COZZENS, 1989; NARIN & AL., 1994; VAN RAAN, 1993]. In both UK and USA, publication counts are required to be included in the evaluation program conducted by public agencies [MARTIN & SKEA, 1992; PHILLIMORE, 1989]. CHI (Computer Horizons Inc.) has pioneered the use of bibliometrics to examine the impact of public funding on the production and quality of papers and patents. FRAME & NARIN [1976] provided one of the first comprehensive studies on the evaluation of 229 major NIH (National Institutes of Health) -supported institutions and universities using the Science Citation Index and NIH data on funding for a period of 1965–1982. On the other hand, researchers are usually more concerned with the scientific quality of the knowledge than with its sheer quantity [COZZENS, 1997]. Two major approaches of measuring the quality appear in previous literatures: peer review and citation counts. In a large number of instances, citation counts were shown having correlations with experts’ peer review [NARIN, 1976; HAGSTORM, 1971]. VIRGO’s [1977] study at the level of individual articles found that citation counts could predict the research quality of two experts better than the expert judgment to each other.

686



Actually, because of the spillover consequences associated with the “externality” of basic research, it is also important to measure the possible outcomes related to the above mentioned output factor. Mansfield has conducted the important survey on 75 major American firms in seven manufacturing industries, estimating the economic and social rates of return to basic research in this sample [MANSFIELD, 1991, 1992]. He did most of his estimation mainly under the proposition that “what would happened (the volume of sales of new products that could not have been commercialized) if the resources devoted to the academic research were with draw – and not allowed to do the same or similar work elsewhere”. Later, MARTIN & SKEA [1992] raised the interconnected and mutually supporting form of economic benefits from government-funded basic research, including: increasing the stock of useful information, new instrumentation and methodologies, skilled graduates, professional networks, technological problem solving, and creation of new firms. It is also suggested by MARTIN [1996] that the multidimensional outputs of basic research were classified into the following four dimensions: scientific, educational, technological and cultural aspects. In practice, few would dispute that the environmental impetus is also one of the important sources for basic research except for funding. The 2005 European Innovation Scoreboard employed 5 indicators – New S&E graduates; Population with tertiary education per 1000 population aged 25~64; Broadband penetration rate; Participation in life-long learning; and Youth education attainment level – to constitute the input factor of “Innovation drivers” and further develop a set of composite indices of innovation performance for EU member states, the US, and Japan [COMMISSION OF THE EUROPEAN COMMUNITIES, 2005]. Similarly, the IMD’s World Competitiveness Yearbook 20002004 also compiled the “scientific infrastructure” as an important part of the overall competitiveness of nations. Lastly, MERTON [1968, 1988] has described the phenomenon of accumulative advantage in the words of St. Matthew, “For whosoever has, to him shall be given, and he shall have more abundance”. The role that positive feedbacks play in the formation and maintenance of academic stratification orders has been established at multiple levels of analysis in the social and natural science [ALLISON & STEWART, 1974; ALLISON & AL., 1982; BONITZ & AL., 1997; COLE & COLE, 1973; KEITH & BABCHUK, 1998]. The Matthew Effect reflects a particular type of accumulative advantage, where increasing returns to scientific efforts are driven by peer evaluations based on reputation. Considering the Matthew effect in the evaluation system of basic research, it is supposed to show the accumulation of scientific efforts on a macro-level of countries, leading thereby to clear stratification patterns. Countries that publish many papers and later on earn much more citations will have an advantage relative to those that do not. Early development of such competencies will convey lasting advantage to the extent that prior research reputation stratifies public science budget and current research


687


efforts in a fashion analogous to the Matthew effect. Apparently, this effect is not the result of a simple addition of events at the micro-level but a cooperative effect in the sense of self-organization. Hypotheses Based on the above explanations of factors, our research hypotheses are set as follows. Hypothesis 1. The research impetus of nations such as education, culture, information environment and youth interest in S&T, would have positive effect on the research input factors. This is based on the idea that a nation in a superior status of civilization would be able to devote more money and personnel into some long-term standing basic researches. Hypothesis 2. Accumulative advantage would positively influence a nation’s devotion to the resource factors. In fact, the single blind peer-review process that dominates publications and reputation in the public science explicitly links resource allocation and research efforts [CHUBIN & HACKETT, 1990]. Thus, this hypothesis offers a measurement of academic efforts that focuses more directly on resources made available through reputational channels. Thus, if the Matthew effect exists, the influence would be positive. Hypothesis 3. The research input factor would have positive effect on the output of basic research. Hypothesis 4. Research impetus would positively affect research output factors. This is based on the truth that the external driving forces, including multi-aspects of economic, social, industrial, educational and civil, are also important sources for basic research except for funding. Hypothesis 5. Prior scientific output would directly determine the later research efforts if accumulative advantage drives accomplishment in public science. Hypothesis 6. Research impetus would have positive effect on research outcomes. This holds true when the multi-dimensional national status (the outcome factor) is greatly determined by the level of infrastructure construction (the impetus factor). Hypothesis 7. Research output would have positive spillover effect on the outcome factor [COZZENS, 1997; MARTIN, 1996]. SEM can be sorted into SEM-ML [JÖRESKOG, 1970] “hard modeling” (heavy distribution assumptions, several hundreds of cases necessary) and to PLS [WOLD, 1982, 1985] “soft modeling” (very few distribution assumptions, few cases can suffice) [TENENHAUS & AL., 2005]. Because the performance of basic research in this study is implemented based on the country-level statistic or survey data, however the number of countries with data available is only 46, this study employs the “soft modeling” SEM,

688



i.e., PLS path modeling, to investigate the interaction effects of innovation system with the center of basic research. This paper employs SmartPLS [RINGLE & AL., 2005] to estimate SEM-PLS. Data collection and operationalization The structural equation model with multiple pooled cross-sections data of 46 countries’ (or areas’) scientific profiles was applied to test our model. Five construct variables which operate the conceptual model were presented in Figure 1. Variables that capture some aspects of resources, impetus and spillover outcomes of public science were drawn from the World Competitiveness Yearbook (WCY) published by the Institute for Management Development (IMD, 2000–2004). WCY integrates the most extensive and widely publicized comparisons of nations’ competitiveness via the yearly surveys of a selected group of Organization for Economic Co-operation and Development [OECD, http://www.oecd.org/home] and newly industrialized countries based on more than 200 socio-economic and political indicators. In the present analysis, eleven WCY variables serve as indicators, while 8 of which are survey data. According to the methodological introduction in WCY, the respondents are required to assess the competitiveness issues by answering the questions on a scale of 1-6, with the response 1 generally indicating a negative perception and 6 indicating the most positive perception. The WCY calculates the average value for each economy and then the data will be converted from a 1-6 scale to a 0-10 scale. R&D expenditure and R&D personnel are the primary measures of resources devoted to the basic research. The R&D expenditure reflects the total amount of publicly budgeted R&D expenditures in each nation (or area) excluding funds from business sector. In essence, the basic research programs are mostly funded by government, while the business sector always devote much of their resources to applied research programs that are beyond our scope of discussion. In order to compress the distribution and scale interpretations in terms of percentage rather than unit change, we normalize the R&D expenditures by GDP in the whole model. The R&D personnel also exclude the industrial part and are normalized by the country’s population. The Essential Scientific Indicators (ESI) database provides the key indicators of both prior and current publications and citations. ESI maintains data on publication volume, citation counts, and publication impact for ranking scientists, institutions, countries, and journals. Table 1 summarized the variables, providing definitions, data sources and relating specific measures to the five constructs.


689


Table 1. Factors and indicators with further explanations Factors Accumulative advantage Impetus

Outcome

Output Resource

Indicator Prior scientific articles

Note Logarithm

Definition Collect from ESI

Citations to prior articles Interest in S&T Science in Schools Internet Users National Culture Basic Research

Logarithm Survey Survey

University Education

Survey

Knowledge Transfer

Survey

Venture Capital

Survey

Quality of Life

Survey

Collect from ESI Is (not) strong among the youth Is (not) adequately taught Number of internet users per 1000 people Closed (or open) to foreign ideas Does (not) enhance long-term economic development Does (not) meet the needs of a competitive economy Lacking (or Highly developed) between companies and universities) Is (not) easily available for business development Quality of life in your economy/society is low (high) Collect from ESI Collect from ESI R&D expenditures (excluding the business part) as a percentage of Gross Domestics Product R&D personnel (excluding the business part) per 1000 people, full time equivalent

Survey Survey

Current scientific articles Logarithm value Citations to current articles Logarithm value R&D expenditure % of GDP R&D personnel

FTE (per 1,000 people)

There is an important note on the estimation of the measurement model for factors with more than one variable. Because of the estimation procedure, the factor must be made “scale invariant”, suggesting that the indicators of a construct must be “standardized to make constructs comparable” [LONG, 1983]. One of factor loadings in each construct is set to be fixed at 1.0 in order to make the constructs comparable. And the choice of which factor loading should be fixed is based on the fact that the selected observed indicator should have the highest reliability in the construction of its corresponding factor. Empirical study Confirmatory factor analysis and overall fitness Before the analysis of structural model, confirmatory factor analysis (CFA) was first performed to validate the relationships among indicators and factors (see Table 2). CFA involves the specification and estimation of the hypothesized models of factors’ structure, which proposes a set of factors to account for covariances among a set of indicators [KOUFTEROS, 1999]. While the traditional exploratory factor analysis (EFA) can only offer preliminary analyses with the absence of an adequate theoretical base, the

690



CFA can be employed to determine whether the data confirm the substantively generated model [LONG, 1983; GARVER & MENTZER, 1999]. The larger the factor loadings are, as compared with their standard errors, the stronger is the evidence that there is a relationship between the indicators and their corresponding factors [BOLLEN, 1989; KOUFTEROS, 1999]. Table 2 shows that each indicator exceeds the t-value at the 0.01 level of significance. Thus, all indicators were significantly related to their specific factors, further suggesting the positive relationships implied in the proposed model. Table 2. The confirmatory factor analysis and indicator reliability Factor Accumulative advantage Impetus

Outcome

Output Resource

Indicator AC1 AC2 IM1 IM2 IM3 IM4 OUC1 OUC2 OUC3 OUC4 OUC5 OUP1 OUP2 RE1 RE2

Standardized loading 0.9972*** 0.9971*** 0.7241*** 0.8217*** 0.6723*** 0.4446*** 0.8773*** 0.8151*** 0.8881*** 0.9305*** 0.8092*** 0.9969*** 0.9970*** 0.9727*** 0.5558***

S. E.a 0.0025 0.0029 0.0783 0.0505 0.0768 0.1370 0.0288 0.0355 0.0219 0.0125 0.0326 0.0032 0.0029 0.0183 0.1612

t-value 406.26 341.73 9.25 16.26 8.75 3.25 30.45 22.97 40.59 74.51 24.79 316.42 338.63 53.14 3.45

a

S.E. Standard error of the covariance. Note: ** indicates p < 0.01; *** indicates p < 0.001.

Construct reliability and discriminant validity Estimates of the reliability for each construct are needed to assess whether the specified indicators sufficiently represent the factors. The construct reliability means that a set of latent indicators of factors are consistent in their measurement. In other words, this reliability is the degree to which a set of indicators share the measurement of the same latent factor. Cronbach’s alpha [HAIR & AL., 1998] was basically the most widely used metric finding applications in evaluating the construct reliability. The Cronbach’s alpha is calculated for our model and the results are given in Table 3. Our results show that the reliabilities of three out of five factors are well above 0.75, except for the impetus and the resource factors. However, there are some disadvantages in Cronbach’s alpha that it assumes that all of the measured items have equal reliability and it cannot be used to infer unidimensionality [GERBING & ANDERSON, 1988].


691


With respect to the limitations of Cronbach’s alpha, we complement the analysis with another alternative measures, namely the “Construct Composite reliability”. The calculation is explained at the footnote of Table 3 in details. And the results in Table 3 suggests that the reliability of all five factors have exceeded the recommended level of 0.70 [HAIR & AL., 1998]. This is considered to be satisfactory as the relationships among indicators and factors are reliable. Table 3. The test of composite reliability Measures Accumulative advantage (AC1, AC2) Impetus (IM1, IM2, IM3, IM4) Outcome (OUC1, OUC2, OUC3, OUC4, OUC5) Output (OUP1, OUP2) Resource (RE1, RE2)

Cronbach’s alpha 0.9943 0.6102 0.9153 0.9939 0.5161

Composite reliability 0.9972 0.7673 0.9369 0.9970 0.7582

Besides the test of construct composite reliability, it is necessary for us to perform the test of validity as well. As previously shown in Table 2, the convergent validity has been already proved to be exists by examining each indicator’s factor loading and significance through t-values [DUNN & AL., 1994]. In this context, the discriminant validity is also an important measurement to test the divergent aspect of individual factors, which measures the low correlations between two sets of indicators that accounted for by different factors [KOUFTEROS, 1999]. The average variance extracted (AVE) can be used as a reference value testing the discriminant validity through comparing with the squared correlation between every pair of factors. Since, AVE measures the amount of variances in the specified indicators accounted for by the latent factor, the discriminant validity exists if the variables share more common variance with their respective factor than any variance that factor shares with other factors [FORNELL & LARCKER, 1981]. It is indicated in the above Table 4 that, the AVE for each factor is substantially higher than the squared correlation between one factor and all the other factors. The highest squared correlation was observed between “Accumulative advantage” and “Output” as 0.9874. This was even higher than their individual AVEs. If there is a lack of evidence supporting discriminant validity, a revision of the scales along with a collection of new data may be warranted before further analysis can be undertaken with confidence [KOUFTEROS, 1999].

692



Table 4. The test of discriminant validity Measures Accumulative advantage (AC1, AC2) Impetus (IM1, IM2, IM3, IM4) Outcome (OUC1, OUC2, OUC3, OUC4, OUC5) Output (OUP1, OUP2) Resource (RE1, RE2)

AVE 0.9943

AC 1.0000

IM

0.4623

0.1419 (0.0201) 0.3224 (0.1039) 0.9863 (0.9728) 0.7670 (0.5883)

1.0000

0.7487 0.9940 0.6275

OUC

0.6576 (0.4324) 0.1385 (0.0192) 0.0397 (0.0016)

OUP

RE

1.0000 0.3186 (0.1015) 0.1603 (0.0257)

1.0000 0.7712 (0.5947)

1.0000

Model quality Finally, the quality of a path model can be validated at three levels: (1) the quality of the measurement model, (2) the quality of the structural model, and (3) each structural regression equation, which can be measured by the communality, redundancy and R2 which was proposed by TENENHAUS & AL. [2005]. The last two statistics are for endogenous blocks (factors), taking into account the measurement model. The results are summarized in Table 5. Table 5. The test of model quality and discrimination Factor Accumulative Advantage Impetus Outcome Output Resource Average

Communality 0.9943 0.4623 0.7487 0.9940 0.6275 0.7654

Redundancy

R2

0.4995 0.9926 0.4106 0.6342

0.7115 0.9990 0.7588 0.8231

The results in Table 5 show that communality, redundancy and R2 are all satisfying better levels than prior literatures, such as SOHN & AL. [2007] and TENENHAUS & AL. [2005], which demonstrates the quality of our path model is higher. A global criterion of goodness-of-fit (GoF) can be proposed [AMATO & AL., 2004] as the geometric mean of the average communality and the average R2: The GoF represents an operational solution to this problem as it may be meant as an index for validating the PLS model globally. According to the results in Table 5, the GoF index turns out to be GoF

community u R 2 =0.887.

This result is also satisfactory when we take the complexity of the model into account.


693


Hypothesis testing and model explanation The set of hypothesizes in Hypotheses section were tested to approve the proposed conceptual model, and the results are given in the following Table 6. Table 6. Results of the structural equation modeling Path Accumulative advantage o Output Accumulative advantage o Resources Impetus o Outcome Impetus o Output Impetus o Resources Output o Outcome Resources o Output

Standardized path loading 0.9840*** 0.8791*** 0.7886*** –0.0019 –0.0851** 0.2094*** 0.0181

S. E.a 0.0210 0.0757 0.0326 0.0027 0.0424 0.0476 0.0219

t-value 46.905 11.614 24.200 0.686 2.007 4.403 0.828

Note: ** indicates p