Short Notes. Causal Inferences in Motivation Research: A Reinterpretation of Results from Panel Studies. John E. Sheridan. School of Business Administration.
Journal of Applied Psychology 1977, Vol. 62, No. 4, 510-513
Short Notes Causal Inferences in Motivation Research: A Reinterpretation of Results from Panel Studies John E. Sheridan
John W. Slocum, Jr.
School of Business Administration Wayne State University
School of Business Administration Pennsylvania State University
Study provides a secondary analysis of the data reported in three previous studies that examined the causal relationships between an expectancy construct of motivation and job performance. Corrected cross-lagged correlations indicate that the previous statistical inferences of causality may have been an artifact of varying measurement reliability and that the observed relationships were spurious. magnitudes of the cross-lagged correlations. If one cross-lagged correlation is significantly higher than the synchronous correlations and the opposite cross-lagged correlation, it suggests directional causation, rather than a spurious or reciprocal relationship, among the variables (Campbell, 1963; Pelz & Andrews, 1964). Table 1 summarizes the findings of the three previous studies that examined the direction of causality between motivation and performance. The reported cross-lagged correlations provide inconsistent inferences regarding the direction of causality. 1 Kenny (1975) argued that the interpretation of these crosslagged correlations may be greatly confounded by the lack of stationarity in the causal process. Stationarity presumes that the causal processes between motivation and performance are in equilibrium and do not change during the measurement interval. The fact that the synchronous correlations were not constant over time in any of the three previous studies indicated that the causal processes were not perfectly stationary in any of these studies (Kenny, 1975). Given the lack of perfect stationarity, one might assume quasi-stationarity in the causal process. Quasi-stationarity means that the coefficients in the causal equations for each variable changed by a proportional constant and that each measured variable had its own unique con-
Recent years have seen an increase in the use of panel correlational designs to draw inferences regarding the direction of causality in observed relationships (Feldman, 1975; Kenny, 1975). One application of this design has been in motivation research to test rival hypotheses of whether motivation causes performance (M —> P), performance causes motivation ( P — » M ) , or performance and motivation have mutual causation (P M ; Schwab & Cummings, 1970; Sheridan & Slocum, 1975). Three studies have examined these hypotheses (Lawler, 1968; Lawler & Suttle, 1973; Sheridan, Downey, & Slocum, 1975). Each study used a similar design in which longitudinal measures of an expectancy construct of motivation (M) and job performance (P) were obtained at a 1-year interval. The two measures of each variable generate six correlations. There are two autocorrelations (fM^Mj and fpjF,,) that represent the testretest stability of the measure over t i m e ; two synchronous correlations (fMjFj and fM 2 p 2 ) that represent concurrent relationships between motivation and performance; and two cross-lagged correlations (/M^., and !"P,M,,) that represent time-lagged relationships between motivation and performance. Inferences regarding the direction of causality are made by examining the differences between the The authors wish to acknowledge E. E. Lawler and J. L. Suttle for making their data available to the researchers for reanalysis and D. A. Kenny for his comments on an earlier draft of this article. Requests for reprints should be addressed to John E. Sheridan, who is now at the School of Business Administration, Pennsylvania State University, University Park, Pennsylvania 16802.
J The statistical significance of the difference between the cross-lagged correlations was not reported in the original studies. In this analysis, the Pearson and Filon test was used to test the significance of the cross-lagged differential. This test is discussed in Kenny (1975, p. 896).
510
SHORT NOTES
511
Table 1 Cross-Lagged Panel Correlations Between Motivation and Performance Autocorrelations Study
^PiPa
^MI P I
^M^Pa
.48*** .59***
.44**
.52***
.52*** .70***
.27**
.31***
.36*** .54***
.23** -.12
fM[M 2
Lawler (1968) n = 55 Lawler &Suttle (1973) n = 34 Sheridan et al. (1975) n = 72
Synchronous correlations
Cross-lagged correlations ^MiPa .55*** -.20
Cross-lagged differential
?"PiM 2
0"MiF 2 ~ ^PiM 2 ) inference
.39***
.16*
M -> P
.44**
-.64***
P->M
.36***
M->P
.30*** -.06
* p = .11. ** p < .05.
'P < .01. stant.2 Quasi-stationarity implies that the synchronous correlations would be equal if they were corrected for attenuation due to measurement error. Given the assumption of quasi-stationarity, Kenny provided a procedure for estimating reliability ratios that can be used to correct the cross-lagged correlations for changes in measurement reliability over time. The present article estimates the reliability ratios for variables measured in each of the three previous studies and performs a secondary analysis of the direction of causality between motivation and performance using the corrected cross-lagged correlations.
estimated as follows: v
X
X
M
X
(1)
X
(2)
When there are more than three measured variables, a pooled estimate of the reliability coefficient is obtained. The reliability ratios can be used to correct the observed cross-lagged correlations in a way similar to correcting a correlation for attenuation. The corrected cross-lagged correlations are as follows : (3)
Method Kenny (1975) defined the reliability ratio as a variable's Time 2 reliability divided by its Time 1 reliability. A reliability ratio greater than 1 indicates an increase in reliability over time; a value less than 1 indicates a decrease in reliability. The estimates of the reliability for a specific variable represent a measure of its communality with respect to each other variable measured in the study. Thus, there must be at least three variables measured at both time periods in order to obtain reliability estimates with respect to a common third variable. In general, there will be (n — 2) (n — l)/2 estimates of the reliability coefficient for each variable, where n equals the number of measured variables. In the three variable cases, the reliability ratios (K) for the measures of motivation and performance with respect to a third variable (X) are
(4)
Longitudinal measures were obtained for at least three variables in the three previous studies. In the Lawler (1968) study, the reliability ratios for measures of motivation (SEa X V) and the superior's evaluation of job performance were estimated with respect to a measure of peer ratings of job performance. In the Lawler and Suttle (1973) study, the reliability ratios for the measures of motivation (E\ X S£2 X V) and the superior's evaluation of performance were estimated with respect to peer and self-reported ratings of job performance. In the Sheridan et al. 2 A condition of proportional stationarity would exist if the constant was the same for each variable.
SHORT NOTES
512
Table 2 Corrected Cross-Lagged Correlations Between Motivation and Performance Corrected cross-lagged correlations
Reliability ratios
Corrected cross-lagged differential
Study Lawler (1968) Lawler &Suttle (1973) Sheridan et al. (1975)
.71 1.81 .13
1.97 .52 1.69
.42 -.27 .16
.50 .32 -.11
.08 -.59* .27
* p < .01.
(1975) study, the measure of job performance was based on the percentage of incentive wages earned. The reliability ratios for the measures of motivation (Ei X ££2 X V) and performance were estimated with respect to measures of the respondent's job satisfaction on five dimensions of the Job Descriptive Index instrument (Smith, Kendall, & Hulin, 1969). Results Table 2 reports the estimated reliability ratios for the measures of motivation and performance in each previous study. 3 The results indicate that the causal inferences made from the reported cross-lagged correlations were confounded by varying measurement reliability. In each study the variable that decreased in reliability appeared to be the cause, whereas the variable that increased in reliability appeared to be the effect. Both Campbell (1963) and Kenny (1975) indicate that this statistical artifact would occur when the two endogenous variables have different measurement reliabilities. The corrected crosslagged correlations reported in Table 2 suggest that the observed cross-lagged relationships between motivation and performance were most likely spurious. The corrected crosslagged differential was significant in only one study (Lawler & Suttle, 1973). Moreover, the significant cross-lagged differential could be attributed to reversed signs of the cross-lagged correlations and not to a difference in the magnitude of the correlations. The reversed signs would make it extremely difficult to interpret the underlying causal process (Rozelle & Campbell, 1969). Kenny further noted that correcting the observed correlations by the estimated reliability ratios tends to increase the sampling error resulting in an approximate test of significance. Therefore, the significance
test for the corrected cross-lagged differential in the Lawler and Suttle study is confounded by the small sample size (n = 34). Discussion When conducting longitudinal field studies over an extended period of time, one might expect a number of extraneous variables to influence the respondent's attitudes and behavior. Consequently, it would be unusual for any causal process between motivation and performance to be perfectly stationary during the study period. The presence of a quasistationary causal process could seriously confound the interpretation of the observed cross-lagged correlations, and the researcher should be cautious to correct the observed correlations for varying measurement reliability. The lack of stationarity was particularly evident in the Sheridan et al. (1975) study where the two synchronous correlations were low and had opposite signs. The lack of even a quasi-stationary causal process in this study could contribute to the extremely low reliability ratio for the measure of motivation and would most likely invalidate the interpretation of causality in the observed relationships. A rule of t h u m b might be to correct only if all or most of the synchronous correlations are significant. Correcting the observed cross-lagged correlations under a quasi-stationary condition would tend to increase the sampling error. The Pearson and Filon test thus becomes an approximate test of significance. Kenny, therefore, suggested that the corrections should be made only when there is a large sample 3 The intercorrelation matrices needed to compute the reliability coefficients were not reported in the original studies and were provided through personal correspondence with the authors.
SHORT NOTES (n > 100). The sample size in all three previous studies failed to meet this condition. Kenny also indicated that both the synchronous and cross-lagged correlations should be computed based on the sample for which there is complete data at both measurements. Sampling attrition over time would confound the interpretation of the panel correlations when the correlations are based on different samples. References Campbell, D. T. From description to experimentation: Interpreting trends as quasi-experiments. In C. W. Harris (Ed.), Problems in measuring change. Madison: University of Wisconsin Press, 1963. Feldman, J. Considerations in the use of causal-correlational techniques in applied psychology. Journal of Applied Psychology, 197S, 60, 663-670. Kenny, D. A. Cross-lagged panel correlation: A test for spuriousness. Psychological Bulletin, 1975, 82, 887-903. Lawler, E. E. A correlational causal analysis of the relationship between expectancy attitudes and job
513
performance. Journal of Applied Psychology, 1968, 52, 462-468. Lawler, E. E,, & Suttle, J. L. Expectancy theory and job behavior. Organizational Behavior and Human Performance, 1973, 9, 482-503. Pelz, D. C., & Andrews, F. Detecting causal priorities in panel study data, American Sociological Review, 1964, 29, 836-848. Rozelle, R. M. & Campbell, D, T. More plausible rival hypotheses in the cross-lagged panel correlation technique. Psychological Bulletin, 1969, 71, 74-80. Schwab, D. P., & Cummings, L. L. Theories of performance and satisfaction. Industrial Relations, 1970, 10, 408-430. Sheridan, J. E., Downey, H. K., & Slocum, J. W. Dynamic correlational analysis of the expectancy model of motivation. Proceedings of the 18th Annual Conference, Midwest Division, Academy of Management, April 1975, 43-52. Sheridan, J., & Slocum, J. W. The direction of the causal relationship between job satisfaction and work performance. Organizational Behavior and Human Performance, 1975, 14, 159-172. Smith, P. C., Kendall, L. M., & Hulin, C. L. The measurement of satisfaction in work and retirement. Chicago: Rand McNally, 1969. Received August 26, 1976 •
