Jan 16, 2019 - ARTICLE AUTHOR: Glenn M McEvoy and Paul Buller ... User reactions were negatively associated with per ceived friendship .... The supervisor reviewed the peer evaluation forms, removed the high and low ratings ..... Longer-term employees are more negative toward peer appraisals - newer employees.
Rapid #: -14179516 CROSS REF ID:
294405
LENDER:
CS1 :: Main Library
BORROWER:
WUG :: Foley Center Library
TYPE:
Article CC:CCL
JOURNAL TITLE:
Personnel psychology
USER JOURNAL TITLE:
Personnel Psychology
ARTICLE TITLE:
User Acceptance of Peer Appraisals in an Industrial Setting
ARTICLE AUTHOR:
Glenn M McEvoy and Paul Buller
VOLUME:
40
ISSUE:
4
MONTH: YEAR:
1987
PAGES:
785-797
ISSN:
0031-5826
OCLC #:
Processed by RapidX:
1/16/2019 5:29:13 PM This material may be protected by copyright law (Title 17 U.S. Code)
PERSONNEL PSYCHOLOGY 1987, 40
USER ACCEPTANCE OF PEER APPRAISALS IN AN INDUSTRIAL SETTING GLENN M. McEVOY, PAUL F. BULLER Department of Business Administration Utah State University A survey of 218 industrial employees who had used a peer evaluation system for over a year found a higher degree of user acceptance than that reported in previous research. Users were significantly more favorable in their attitudes toward peer appraisals when the appraisals were used for developmental rather than evaluative purposes. Favorable user reactions were positively associated with the satisfaction that employees expressed with prior peer ratings and with the amount of leniency perceived in the peer rating process. User reactions were negatively associated with per ceived friendship bias and years of company experience. No relationship was found between peer or self-ratings of performance and reactions to the peer appraisal process.
Extant research on peer performance evaluation is generally favorable. Peer assessments exhibit acceptable reliabilities and above average predic tive validities (DeNisi & Mitchell, 1978; Imada, 1982; Kane & Lawler, 1978; Lewin & Zwany, 1976; Love, 1981; Reilly & Chao, 1982; Schmitt, Gooding, Noe, & Kirsch, 1984; Siegel, 1982). Compared with supervi sory ratings, peer evaluations (1) tap different dimensions of performance (Borman, 1974; Zammuto, London, & Rowland, 1982), (2) are more sta ble over time (Smith, 1976), and (3) are more likely to differentiate effort from performance and to focus on task-relevant abilities and competen cies (Klimoski & London, 1974; Tucker, Cline, & Schmitt, 1967). Upon reviewing this history of ratings research, Wexley and Klimoski (1984) concluded that peer ratings were “potentially the most accurate judgments of employee behavior” (p. 60). Despite this favorable evidence, there has been a reluctance on the part of practitioners to adopt peer ratings. This is evidenced by the fact that the vast majority of peer ratings studies cited in Kane and Lawler’s (1978) comprehensive review were conducted for research purposes only (25 of the 33 studies cited involved the use of peer appraisals in a research project, not the investigation of existing peer appraisal systems). Kane and Lawler (1978) advanced two reasons for this reluctance: (1) the common belief that peer ratings are simply a “popularity contest” and (2) the failure on Correspondence and requests for reprints should be addressed to Glenn M. McEvoy, Department of Business Administration, Utah State University, Logan, UT 84322. COPYRIGHT © 1987 PERSONNEL PSYCHOLOGY, INC.
785
786
PERSONNEL PSYCHOLOGY
the part of most managers to recognize a need to include other raters in the performance appraisal system. Smith (1976) suggested a third possi bility: relinquishing control of the performance appraisal process may be threatening to managers. DeNisi and Mitchell (1978) voiced other con cerns with peer appraisals: (1) friendship bias, (2) subgroup effects (the tendency for members of any identifiable subgroup to give inflated ratings to fellow group members), (3) reliance of peers on stereotypes in rating, and (4) the possibility of retaliation in subsequent ratings when receiving low ratings from peers at one point in time. The high reliabilities and validities of peer ratings coupled with practi tioner reluctance to use such ratings suggest that user acceptance in those organizations that have incorporated peer ratings may be low. Indeed, the limited research available indicates exactly that. Three of the four studies that have investigated user acceptance of peer evaluations have indicated problems (Cederblom & Lounsbury, 1980; Downey, Medland, & Yates, 1976; Love 1981). Only one earlier study was positive (Roadman, 1964). This study involved the developmental (as opposed to the administrative) use of peer evaluations made during a management training course. Ap proximately 98% of the participants evaluated the peer feedback process as a “very valuable educational experience” (Roadman, 1964, p. 214). Downey, Medland, and Yates (1976) sampled 1,168 Army colonels and found that over half felt peer evaluations were unsuitable for use in mak ing either promotion decisions or job assignments. As with the Roadman (1964) study above, peer evaluations had not previously been used either developmentally or administratively in the subjects’ organization. In a study of 145 police officers rated by their peers for research pur poses only. Love (1981) found that all three methods of peer evaluation— nomination, rating, and ranking—generated negative reactions. Overall, the officers did not like the peer evaluation process and believed that it would be unfair, inaccurate, and subject to friendship bias. Love (1981) noted, however, that the paramilitary nature of his sample may have lim ited the generalizability of these findings to other settings: “future research must collect similar reaction data in nonmilitary-oriented organizations” (p. 457). The only study of user acceptance of an existing peer evaluation system was reported by Cederblom and Lounsbury (1980). In a survey of 174 faculty members (representing a 59% response rate), they found relatively low acceptance of peer evaluations as practiced for six years. They also found that faculty who perceived a high level of friendship bias in peer ratings were more negative toward the process. Conversely, more positive reactions were expressed by those who perceived that peer ratings had high feedback value, high validity, and positive effects on morale and who were satisfied with prior peer rating outcomes. Contrary to their hypothesis.
McEVOY AND BULLER
787
no relationship was found between user acceptance and self-reported prior peer ratings. It is clear that the data are quite limited on user acceptance of peer appraisals. Research designs have been weak (e.g., peer evaluations intro duced for research purposes only, self-reports of performance levels), and samples have been limited in both size and type of jobs represented. The paucity of data pertaining to user acceptance has resulted in numerous calls for more research in this area (e.g., Cederblom & Lounsbury, 1980; Kane & Lawler, 1978; Love, 1981). Unfortunately, little has been done along these lines. This is regrettable because of the high reliabilities and validi ties that peer ratings generally possess and, thus, the significant potential that such ratings have for improving performance appraisal in practice if the problems of negative user reactions can be overcome (Lawler, 1967). Therefore, the objectives of the present study were (1) to explore the reactions of an industrial sample to an existing peer evaluation system and (2) to examine the correlates of these reactions. On the basis of the research reviewed above, and particularly the Cederblom and Lounsbury (1980) study, the following six hypotheses were tested: Hypothesis I: Reactions to peer evaluations are generally negative. Hypothesis 2: Reactions to peer evaluations performed for developmental purposes are more positive than reactions to those performed for evaluative (e.g., wage determination) purposes. Hypothesis 3: Users’ prior performance ratings (from peers) are unrelated to user acceptance. Hypothesis 4: Users’ satisfaction with prior performance ratings relates pos itively to user acceptance. Hypothesis 5: Employees who cite friendship bias as a problem with peer evaluations respond more negatively to such evaluations. Hypothesis 6: User acceptance is positively correlated with the degree to which the respondent expects peers to provide lenient ratings.
Additionally, this study investigated four demographic variables (age, sex, company experience, and education) to see if they were related in any systematic way to user acceptance. Methods Setting and Subjects The research was conducted with employees of a nonunion food-process ing plant in the western United States. The plant employed 303 hourly
788
PERSONNEL PSYCHOLOGY
employees, 20 of whom were classified as “leadmen” because of their sen iority and job experience. The mean age of the employees was 33, and they had worked at this organization for an average of seven years. Fifty-seven percent were males; fewer than 10% had college degrees. Three years before this research was undertaken, the personnel direc tor at the plant implemented a peer evaluation system for use in making decisions to promote hourly employees from temporary to full-time status. This system applied to only a small portion of the work force. About one year before this study, the plant introduced a more comprehensive peer appraisal system for evaluating all hourly employees on a regular basis. The Peer Evaluation System The revised peer evaluation system was developed by a team of man agers and hourly employees. The purpose of the system was to provide input for both wage decisions and developmental progress reports for em ployees. Under the new system, each hourly employee was evaluated at least once a year for wage decision purposes and twice a year for devel opmental purposes. Employees were rated on seven factors: attendance, attitude, safety, adaptability, cooperation and teamwork, sanitation, and work performance. The first six factors were defined identically for all employees. The last factor was defined differently for each major job clas sification in the plant. Four to seven subdimensions were rated under work performance, and detailed standards were established on each subdimen sion. Written justification was provided by peers for each rating, and those written comments provided the basis for employee feedback. Wage and developmental evaluations were conducted independently, but the same forms and performance dimensions were used for both types of evaluation. Further, the procedures for conducting wage and developmental evaluations were the same. First, supervisors were notified by the personnel office about which employees were due to be evaluated. Supervisors then distributed appraisal materials to five evaluators (one “leadman” and four co-workers). There was no systematic procedure established for selecting peer raters other than the requirement that they be in a position to observe the employee’s work performance. These individuals then rated the target employee and returned the forms to the supervisor. The supervisor reviewed the peer evaluation forms, removed the high and low ratings on each dimension, and averaged the remaining three rat ings. The supervisor then arranged for a feedback session with the em ployee. Employees were informed only of averaged performance scores and could not identify how any single individual had rated them. After discussing the ratings, both supervisor and employee signed the evaluation form and forwarded it to the personnel office.
McEVOY AND BULLER
789
At the time this study was conducted, the new peer appraisal system had been in place about one year. Thus, each hourly employee had been evaluated at least once for wage purposes and at least twice for developmen tal purposes. Peer evaluations were one of three factors considered when making wage adjustment decisions. Attendance and tenure were the other two factors. Objective criteria were used in making these decisions (e.g., minimum tenure, attendance, and peer rating scores). Thus, peer ratings at least partially determined the distribution of important organizational rewards. Measures To determine employee reaction to the new peer appraisal system, the authors developed a survey in cooperation with the plant’s personnel di rector. The survey was pilot tested with a sample of employees, revised, and then administered in conjunction with a general climate survey of all employees. Specifically, supervisors met with groups of their employees to conduct an in-person climate survey. At the end of this group survey, which took about half an hour, supervisors gave each employee the peer ratings reaction survey. Employees were told that their participation in the survey was voluntary and that their responses would only be seen by the researchers. They were then asked to take 5-10 minutes to complete the survey independently, seal it in an envelope addressed to the researchers, and drop it in a box as they left the room. A total of 218 surveys were re turned, representing a response rate of 72%. Not all employees completed the survey because some chose not to, some were absent from work the day the survey was given, and some were not present at the group meetings due to minimum staffing requirements in the plant. User acceptance. Following Cederblom and Lounsbury (1980), it was believed the best single measure of user acceptance of the peer appraisal system would be the response to the question “Would you recommend that the present peer evaluation process be (a) discontinued, (b) continued with changes, or (c) continued as is?” Responses to this question were coded 1, 2, and 3, respectively. Using a five-point scale, subjects were also asked: “Overall, how satisfied are you with the use of peers as evaluators of your performance?” The correlation of this latter measure with the first was r = 0.48 (p < .001). While this correlation suggested that some overlap existed between the two measures of user acceptance, it was not high enough to warrant exclusion of one or the other variable from the analysis. Therefore, analyses were performed for both dependent variables, and they are reported separately below. Subjects were also asked seven questions about their reactions to the use of peer evaluations for developmental purposes and the same seven
790
PERSONNEL PSYCHOLOGY
questions in regard to wage evaluations by peers. Specifically, they were asked about the perceived fairness, accuracy, usefulness, ease of use, clar^ ity, confidentiality, and satisfactoriness for each of these uses on five-point scales. Responses were summed to form two overall assessments: one of user reaction to peer evaluations for developmental purposes and the other of reaction to peer appraisals for wage determination purposes. The coeffi.. cient alpha reliabilities of these two scales were 0.90 and 0.88, respectively. Friendship bias. Subjects were not asked directly about friendship bias on the survey. However, they were asked the open-ended question: “What don’t you like about the current peer evaluation system?” The authors performed a content analysis on the responses to this question and coded friendship bias as present (value =1) whenever mention was made of friends rating each other higher, or of the ratings being a “popularity contest,” or some similar remark. When no mention was made of friendship bias in response to this question, this dichotomous variable was coded 0. The authors independently coded all responses, and the degree of agreement between the two sets of coded responses was 97%. Disagreements were resolved through discussion. Perceived leniency. To determine perceived leniency of peer evalua tions, the survey contained the question: “Peers are more likely to rate each others’ job performance (lower than, about the same as, higher than) supervisors.” Responses were coded 1, 2, and 3, respectively, so higher numbers were indicative of more perceived leniency for peer ratings. Performance ratings. Supervisors at the plant were responsible for identifying major performance dimensions for each job category. When ever peers were asked to evaluate each other’s performance, they provided overall (global) ratings on those performance dimensions identified by the supervisors as important to each job category in the plant. A ratee’s fi nal rating was determined as described above (five peer raters, with high and low scores removed and the remaining three scores averaged), using a six-point scale with the following anchors: 0 = unacceptable, 1 = unsatis factory, 2 = less than adequate, 3 = adequate, 4 = good, and 5 = excellent. In practice, no average scores lower than 3 were present. The firm in our sample did not retain individual peer ratings (as opposed to averages) in personnel files, so no measure of interrater reliability for this performance measure was available. Peer ratings were gathered from personnel files for all hourly employ ees; however, fewer than half of these could be used for the present study. The reason for this was that the provision of the employee’s name on the peer ratings reaction survey was made optional, and only 80 subjects chose to fill in their names. Therefore, actual performance ratings could be matched with survey responses for only 80 out of the sample of 218. Sur vey responses and mean performance levels for those providing names were
McEVOY AND DULLER
791
compared with those not providing names, and no significant differences were present. In addition to actual performance ratings data from personnel files, self-reported performance ratings and satisfaction with these ratings were measured as part of the survey. Each of these two items was measured using a five-point scale (from 1 to 5), with larger numbers indicating higher per ceived performance and satisfaction with ratings. The correlation between actual overall performance ratings from personnel files and self-reported performance ratings was r = 0.36 (p < .001), suggesting only modest overlap in these two performance measures. Analyses Descriptive statistics and comparison with prior research were used to determine the degree of user acceptance among subjects. A t test was used to determine whether there were differences between acceptance levels for developmental and wage peer evaluations. Simple correlation analy ses were used to determine the degree of individual relationship between appraisal system characteristics, demographic variables, and user accep tance. Regression analyses were used to determine the extent to which any independent variable provided a unique explanation of variance in the de pendent variable of user acceptance when the effects of other independent variables were controlled for. Results The first research question dealt with the overall favorability of em ployee response to peer appraisals. This could only be assessed by com paring the results of the present study to those of Cederblom and Lounsbury (1980). In their sample of university professors, over half of the respon dents (59%) favored outright elimination of peer evaluations, while 30% favored continuing them if they were revised, and 11% favored continu ation as they were. In our sample, the percentages favoring those same three options were 16%, 67%, and 17%, respectively. Obviously, many fewer employees in the present sample were dissatisfied enough with peer evaluations to recommend their outright elimination. On the other hand, fewer than one in five wanted to see them continued as they had operated in the past. On the whole then, employees seemed guarded in their endorse ment of peer evaluations but substantially more favorable than suggested by previous research. The second research hypothesis was examined using a dependent-sam ples t test of the difference between the means of the responses mea suring reactions to peer ratings for developmental versus evaluative (wage)
PERSONNEL PSYCHOLOGY
792
^ VO *
^ ON o C7V ^ OO
w
-S
oP
S§ So ^
00 i VO
oci o ^
^
to
Vm i-^
c/s
(iii3 00 CM
c/3 (/i
o •g o s 2 Bg jc/3 fc ‘=^ 01 u 't 0.0 01D. o B* 3 -a
•Bo
't01
o.
01
00 01
0^ 'O 01
2 g S' L, 01 01 >
DdO
IT)
!oI
01
8.
o.
€ 13
SO Q.
£ T3
00
_> tsO C c 2 eS /2 00 2 CLi
vd
00
2 01
1
oc '£2 > *01
2 /2n Q d
■o fa 3: 01
0*0 o,-^
cg.ia £2
McEVOY AND BULLER
793
TABLE 2 Correlations Between User Acceptance of Peer Appraisals and Four Demographic Variables
Demographic variable Age Sex Years with company Education
User acceptance Favor continuation, modification, or Satisfaction with discontinuation of peers as appraisers peer appraisals -.02 (199) .12* (200) -.13* (203) -.11 (201)
- 18** (200) -.02 (201) -.17** (204) -.15* (202)
Note: Numbers in parentheses are sample sizes. *powney RG, Medland FF, Yates LG. (1976). Evaluation of a peer ” ® psychology, 61, subsequent promotion of senior military officers. Journal ofApP 206-209. f „prformance evaluations, yourna/ Greenberg J. (1986). Determinants of perceived fairness o p of Applied Psychology, 71, 340-342. stereotypes as determinants of difImada AS. (1982). Social interaction, observation ^ performance, 29. ferentiation in peer ratings. Organizationa e 397.^15. _ Psychological Bulletin, 85,555JS, Lawler EE. (1978). Methods of peer assessm 586. . „4^nrmance appraisal. Journal of •^'imoski RJ, London M. (1974). Role of the rater tn p Applied Psychology, 59, 445-451. „rnarh to measuring managerial job per Lawler EE 111. (1967). The multitrait-multirater app ,gj foimance. Journal of Applied Psychology, merature critique, and a paradigm Lewin AY, Zwany A. (1976). Peer nominations: A rno , for research, personnel psychology, 29, (U-^s- Reliability, validity, friendship
„
KG. (1981). Comparison of peer 66, 451-457. bias, and user reaction. Journal of Applied PsyoKomy’^^^^^^^^ employee selection Reilly RR, Chao GT. (1982). Validity and fairne^ o s
Sehmitt'N.lioding RZ, Noe RA. Kirsch M. fished between 1964 and 1982 and the mvestig PSYCHOLOGY, 37, 407-422. managerial effectiveness by peers an B'egel L. (1982). Paired comparison 352. hlem of supervisors, personnel psychology, ’ . effectiveness: . i Smith PC. (1976). Behaviors, results, ^^°^\f,^^strial and organizational psy eriteria. In Dunnette MD (Ed.), (2nd ed. pp. 745-775). Chicago: Randr creativity and other perfo Tucker MF, Cline VB, Schmitt JR. pharmaceutical scientists. Journa o measures from biographical information among p Applied Psychology, 51, 131-138. ^^raisal' An update. In Row an ' 2 vv ^’Vexley KN, Klimoski R. (1984). resources management (Vo . ,PGR lEds.), Research in personnel and human ^
35-79). Greenwich, CT: JAI
organization and rater differences
^muto RF, London M, Rowland KM. ^ formance appraisals, personnel psychology,
643-658. 4 .
