Research Report
“You Talkin’ to Me?”: Self-Relevant Auditory Signals Influence Perception of Gaze Direction
Psychological Science 21(12) 1765–1769 © The Author(s) 2010 Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/0956797610388812 http://pss.sagepub.com
Raliza S. Stoyanova, Michael P. Ewbank, and Andrew J. Calder MRC Cognition and Brain Sciences Unit, Cambridge, England
Abstract In humans, direct gaze typically signals a deliberate attempt to communicate with an observer. An auditory signal with similar signal value is calling someone’s name. We investigated whether the presence of this personally relevant signal in the auditory modality would influence perception of another individual’s gaze. Participants viewed neutral faces displaying different gaze deviations while hearing someone call their own name or the name of another person. Results were consistent with our predictions, as participants judged faces with a wider range of gaze deviations as looking directly at them when they simultaneously heard their own name. The influence of this personally relevant signal was present only at ambiguous gaze deviations; thus, an overall response bias to categorize gaze as direct when hearing one’s own name cannot account for the results. This study provides the first evidence that communicative intent signaled via the auditory modality influences the perception of another individual’s gaze. Keywords face perception, facial features, auditory perception, social cognition Received 1/26/10; Revision accepted 6/18/10
Eye gaze not only carries information regarding a person’s direction of attention, but also provides clues to his or her future intentions and actions (Baron-Cohen, 1995). The ability to discriminate direct from averted gaze is present from birth (Farroni, Csibra, Simion, & Johnson, 2002). In adults, direct gaze potentiates detection of faces (Senju & Hasegawa, 2005), as well as discrimination of their gender (Macrae, Hood, Milne, Rowe, & Mason, 2002) and identity (Hood, Macrae, Cole-Davies, & Dias, 2003). Direct gaze may facilitate perception because it is a deliberate, or ostensive, signal (Sperber & Wilson, 1995) that indicates to the observer that something of importance is about to be communicated (Frith, 2008). However, gaze direction is only one component of a much richer set of social signals from visual and auditory modalities, including facial expression, body posture, and verbal and nonverbal vocal information. In the auditory modality, a signal with value similar to that of direct gaze is hearing one’s own name (Moray, 1959). Although sensitivity to one’s own name emerges over the first year of life (Mandel, Jusczyk, & Pisoni, 1995), sensitivity to another auditory ostensive cue―being spoken to in a slow, deliberate manner (motherese)―is present in the first month after birth (Cooper & Aslin, 1990). In a study indicative of the
common ostensive value of these auditory and gaze cues, infants followed an adult’s gaze to objects in their environment only if the gaze shift was preceded by an ostensive signal, such as a period of direct eye contact or infant-directed speech (Senju & Csibra, 2008). Note that these two signals not only have a similar function but also often co-occur. For example, a mother would typically speak in motherese as she engages her infant in mutual gaze. Similarly, in order to attract the attention of an adult, people often call the person’s name as they make direct eye contact with him or her. An important, but as yet unaddressed, question arising from the natural cooccurrence and shared value of these two signals is whether hearing one’s own name would enhance one’s perception of direct gaze in the visual modality. Previous work has shown that visual context may affect where one thinks others are looking. For example, Lobmaier, Fischer, and Schwaninger (2006) showed that an object near the line of sight causes the perceived gaze direction to Corresponding Author: Raliza S. Stoyanova, MRC Cognition and Brain Sciences Unit, 15 Chaucer Rd., Cambridge CB2 7EF, United Kingdom E-mail:
[email protected]
1766
Stoyanova et al.
gravitate toward that object, presumably because one expects people to be attending to objects rather than to an empty space. Of more relevance to the current study is research showing that the range of gaze deviations that participants perceive to be directed at themselves is affected by facial expression, with angry expressions increasing this range relative to fearful or neutral expressions (Ewbank, Jennings, & Calder, 2009; Lobmaier, Tiddeman, & Perrett, 2008). To the best of our knowledge, no prior research has addressed whether any form of auditory cue exerts an influence on gaze discrimination. However, there is good reason to think that this may be the case, in view of the principle that social stimuli with the same signal value (Adams & Kleck, 2003, 2005) influence the perception of one another. Moreover, recent work suggests that the prefrontal region involved in inferring the mental states of other people is engaged both in response to hearing one’s own name (vs. another name) and in response to viewing direct gaze (vs. averted gaze; Kampe, Frith, & Frith, 2003). Kampe et al. (2003) suggested that visual and auditory ostensive cues activate theory-of-mind computations in order to facilitate the interpretation of subsequent communication. However, Conty, N’Diaye, Tijus, and George (2007) have reported that the prefrontal response to direct gaze temporally precedes the response in perceptual regions, such as the superior temporal sulcus, that are thought to be involved in coding the direction (Calder et al., 2007; Nummenmaa & Calder, 2009) or intentionality (e.g., Pelphrey, Viola, & McCarthy, 2004) conveyed by gaze. This suggests that the analysis of gaze direction may be amenable to top-down influences from prefrontal regions that are sensitive to direct gaze and the relevance of one’s name. On the basis of these findings and the observation that social stimuli with shared signal value can influence the perception of one another (Adams & Kleck, 2003, 2005), we hypothesized that hearing one’s own name, relative to another name, would increase the range over which one feels that another individual’s gaze is directed at oneself. To test this hypothesis, we had participants categorize the gaze direction of neutral faces whose gaze varied from left to right in small, incremental steps. Faces were presented individually and were accompanied by auditory presentation of the participant’s own name or another person’s name. Psychometric functions, fitted to the proportion of “left,” “right,” and “direct” responses, provided an objective dependent measure of the range of gaze deviations perceived as direct when Left 10
Left 7
Left 4
participants heard their own name and when they heard another person’s name.
Method Participants Eighteen volunteers with normal or corrected-to-normal vision (6 male, 12 female; mean age = 24.7 years, SD = 4.92 years) were recruited from the MRC Cognition and Brain Sciences Unit Volunteer Panel. They provided written and informed consent and were paid for participating. One participant did not complete the entire experiment and was excluded from the analysis, leaving a final sample of 17 (6 male, 11 female; mean age = 24.8 years, SD = 5.03 years).
Stimuli The face stimuli consisted of gray-scale photographs of four males posing neutral expressions. The photographs were selected from the NimStim Face Stimulus Set (Tottenham et al., 2009) and the Karolinska Directed Emotional Faces database (Lundqvist, Flykt, & Öhman, 1998). Nonfacial areas and hair were masked, leaving the central face area visible. The facial images subtended a visual angle of 12° by 8°. Following previous research (Adams & Kleck, 2005; Ewbank et al., 2009), we manipulated gaze by altering the position of the iris of each eye in incremental steps of 1 pixel per image using Adobe Photoshop. This alteration is equivalent to a shift of 0.03 cm, or a visual angle of approximately 1/12°. We used a total of seven gaze deviations: true direct gaze plus shifts of 4, 7, and 10 pixels for left and right gaze (see Fig. 1). These deviations were chosen on the basis of a previous study showing that true direct gaze is largely perceived as direct, gaze that is 10 or more pixels from direct is mostly perceived as averted, and gaze that is 4 pixels from direct is perceived as direct and averted equally often (Ewbank et al., 2009). Auditory stimuli consisted of recordings of four male native English speakers calling participants’ first names (e.g., “Sarah!”), as well as a set of control names. Recordings were made in a soundproof booth. For each participant, three genderand syllable-matched names served as controls. There was no significant difference (p > .3) in the duration of participants’ own names (M = 571.65 ms, SE = 8.9 ms) and the duration of
Direct
Right 4
Right 7
Right 10
Fig. 1. Examples of the stimuli. Each facial identity displayed a neutral expression at seven gaze deviations: 10 pixels to the left, 7 pixels to the left, 4 pixels to the left, direct gaze, 4 pixels to the right, 7 pixels to the right, and 10 pixels to the right.
1767
Self-Relevant Signals Influence Gaze Perception control names (M = 562.88 ms, SE = 4.75 ms). All sound files were normalized in amplitude, using Adobe Audition 2 (http:// www.adobe.com).
own name and the three control names each appeared on one quarter of the trials.
Results
Procedure Participants were seated 50 cm in front of a computer monitor. A chin rest was used to maintain head position and distance from the screen. Each trial began with a central fixation cross, which remained on-screen for between 500 and 1,750 ms (M = 1,125 ms). A variable duration was used in order to reduce expectancy effects. The fixation cross was followed by a centrally presented face for 600 ms. The face was displayed on a gray background and was presented together with an auditory name delivered through Sennheiser (Weddemark, Germany) HD465 stereo headphones. Following the offset of the face and name, there was a 2,000-ms interval during which participants were required to press one of three buttons according to whether they perceived the face to be looking to their left, to their right, or directly at them. Participants were instructed that auditory stimuli were irrelevant to the task and that they should concentrate on categorizing the direction of the gaze as accurately as possible; speed of response was not emphasized. There were a total of 256 randomly presented trials: Sixty-four trials presented direct gaze, and 64 presented each of the deviated gaze directions (4, 7, and 10 pixels). The trials presenting deviated gaze were equally divided between left-oriented and right-oriented gaze. At each gaze deviation, the participant’s
“Left” Responses
For each participant, separate logistic functions were fitted to the proportion of “left” and “right” responses as a function of gaze deviation. A function for “direct” responses was calculated by subtracting the sum of “left” and “right” responses from 1 at each gaze deviation. From these data, we calculated the crossover point between the fitted “direct” and “left” functions and between the fitted “direct” and “right” functions (see Fig. 2). The sum of these two absolute values provided an objective measure of the range of gaze deviations that each participant perceived to be direct for each of two conditions: the own-name condition and the other-name condition. Following Gamer and Hecht (2007), who likened this range to a cone, we refer to these as cone-of-gaze values. A repeated measures analysis of variance (ANOVA) revealed a significant main effect of condition, F(1, 16) = 4.61, p < .05, ηp2 = .22, reflecting a wider cone of gaze for the own-name condition (M = 7.5 pixels, SE = 0.47) than for the other-name condition (M = 6.8 pixels, SE = 0.41). Although the effect of condition appeared to be slightly larger for rightward than for leftward gaze deviations, the difference between the crossover points for the own- and other-name conditions did not differ significantly between rightward and leftward deviations, t(16) = −1.03, p = .32, two-tailed. Hence,
“Direct” Responses
“Right” Responses
1.0
Proportion of Responses
.9 .8 .7 .6
Own Name
.5
Other Name
.4 .3 .2 .1 .0 –10
–8
–6
–4
–2
0
2
4
6
8
10
Gaze Deviation Fig. 2. Plot showing the mean fitted logistic functions for “left,” “right,” and “direct” responses in the own-name condition and the other-name condition. The dashed lines indicate the crossover points, and the arrows indicate the width of the cone of gaze.
1768 the following analyses collapsed across leftward and rightward gaze deviations. To exclude the possibility that the effects reflected an overall bias for participants to label gaze as direct when they heard their own name, we entered the proportion of “direct” responses into a 2 × 4 ANOVA with condition (own-name or other-name) and gaze deviation (0, 4, 7, or 10 pixels) as repeated measures factors. This analysis revealed the expected main effect of gaze deviation, F(3, 48) = 408.86, p < .001, ηp2 = .96, and a trend for an effect of condition, F(1, 16) = 4.21, p = .06, ηp2 = .21. Critically, there was also a significant interaction between the two factors, F(3, 48) = 7.94, p < .001, ηp2 = .33. Simple-effects analyses, at each gaze deviation, showed an effect of condition only at the 4-pixel deviation, F(1, 16) = 9.61, p < .01, ηp2 = .38, for which the perceived direction of gaze was somewhat ambiguous. There was no effect of condition for faces that showed true direct gaze (0 pixels, F < 1), nor for faces with gaze deviations of 7 or 10 pixels (Fs < 2.13, ps > .16). Thus, there was no evidence that hearing one’s own name resulted in an overall bias to respond “direct.”
Discussion This study provides the first evidence that an ostensive signal in the auditory modality (calling a person’s name) has an influence on the perception of an ostensive signal with which it frequently co-occurs in the visual modality (direct gaze). As we predicted, participants reported direct gaze over a wider range of gaze deviations when they heard their own name than when they heard the name of another person. Further analyses showed that this auditory signal did not affect responses when gaze was clearly direct or clearly averted, providing no evidence of an overall bias to respond “direct” in response to hearing one’s own name. Instead, the effect was maximal when gaze direction was intermediate, or ambiguous (i.e., the 4-pixel gaze deviation). This pattern accords with previous research showing that, within the visual modality, facial expression has an influence on the perception of gaze when gaze direction is relatively difficult to discriminate (Ewbank et al., 2009), and, similarly, that gaze has a greater reciprocal influence on the processing of expression when the expression is more difficult to discriminate (Graham & LaBar, 2007). Similar effects have also been found in the multisensory perception of emotional expression: Signals in the nontarget modality exert a larger influence on perception of signals in the target modality when the latter are degraded or ambiguous (e.g., Collignon et al., 2008; de Gelder & Vroomen, 2000). Future research should examine whether the bidirectional influence between visual and auditory emotional cues extends to ostensive cues. For example, would direct gaze also increase the probability of detecting ambiguous or degraded presentations of one’s own name? Although gaze direction is an important cue to the current and future intentions of individuals in one’s environment (Baron-Cohen, 1995), it occurs in a rich visual and auditory context. It is important to note that some of the signals that
Stoyanova et al. co-occur with gaze convey the same behavioral intent. Within the visual modality, for example, perception of an angry or joyful facial expression is facilitated by direct gaze, as both signals are associated with approach-related behavior (Adams & Kleck, 2003; Graham & LaBar, 2007). By contrast, perception of fearful facial expressions may be facilitated when they display averted gaze, because both fearful expressions and averted gaze signal avoidance (Adams & Kleck, 2005; but see Bindemann, Burton, & Langton, 2008). Similarly, as already discussed, a reciprocal influence of angry expressions on gaze perception is also found; observers are more likely to perceive direct gaze in an angry facial expression than in a fearful or neutral facial expression (Ewbank et al., 2009; Lobmaier et al., 2008). The present study builds on this work by showing that without any change in the visual characteristics of a face, a concurrent auditory signal that conveys the intent to communicate can influence the perception of direct gaze, a visual signal conveying the same behavioral intent. An interesting empirical question is whether the effect of auditorily presented names on gaze discrimination is limited to participants’ own names or whether it extends to other people’s names. For example, might participants be more likely to perceive ambiguous gaze as averted toward a friend sitting beside them if the friend’s name was called out? This would provide evidence that the range of gaze deviations perceived as direct can be both widened and narrowed and would also discount the idea that familiarity of the name alone can explain the current results. It would also suggest that the reported effect is not limited to self-relevant cues, but is a more general property of cues that share intent and often co-occur. Another direction for future research concerns the neural mechanisms through which visual and auditory ostensive signals may interact. As suggested by research showing that the prefrontal response to direct gaze is earlier than the superior temporal response (Conty et al., 2007), it is possible that the increased perception of direct gaze when one hears one’s own name is a result of a top-down modulation of superior temporal regions involved in gaze perception. Although a number of researchers have suggested that prefrontal mentalizing regions might be a source of top-down signals in tasks that involve making inferences about other individuals’ intentions (Frith & Frith, 2006; Nummenmaa & Calder, 2009; Teufel et al., 2009), this remains to be established empirically. Finally, given infants’ early sensitivity to both visual and auditory ostensive signals (e.g., Senju & Csibra, 2008), further work, using behavioral and neuroimaging methods, could also examine at what point in development ostensive cues in one modality begin exerting their influence on perception of ostensive cues in another modality. In summary, we have shown that gaze is more likely to be perceived as direct when accompanied by auditory presentation of one’s own name, a signal that shares similar behavioral intent and frequently co-occurs with direct gaze. This suggests that perception of another person’s gaze direction is affected not only by salient visual cues, such as facial expression (e.g.,
Self-Relevant Signals Influence Gaze Perception Ewbank et al., 2009; Lobmaier et al., 2008), but also by auditory cues with similar signal value. Acknowledgments R.S.S. holds a Gates Cambridge Scholarship and an Overseas Research Studentship. We thank Simon Strangeways for preparing the visual stimuli and Colin W. Clifford for helping with the logistic function analysis.
Declaration of Conflicting Interests The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.
Funding This research was funded by the United Kingdom Medical Research Council (U.1055.02.001.0001.01 to A.J.C.).
References Adams, R.B., & Kleck, R.E. (2003). Perceived gaze direction and the processing of facial displays of emotion. Psychological Science, 14, 644–647. Adams, R.B., & Kleck, R.E. (2005). Effects of direct and averted gaze on the perception of facially communicated emotion. Emotion, 5, 3–11. Baron-Cohen, S. (1995). Mindblindness: An essay on autism and theory of mind. Boston, MA: MIT Press. Bindemann, M., Burton, A., & Langton, S.R.H. (2008). How do eye gaze and facial expression interact? Visual Cognition, 16, 708–733. Calder, A.J., Beaver, J.D., Winston, J.S., Dolan, R.J., Jenkins, R., Eger, E., et al. (2007). Separate coding of different gaze directions in the superior temporal sulcus and inferior parietal lobule. Current Biology, 17, 20–25. Collignon, O., Girard, S., Gosselin, F., Roy, S., Saint-Amour, D., Lassonde, M., et al. (2008). Audio-visual integration of emotion expression. Brain Research, 1242, 126–135. Conty, L., N’Diaye, K., Tijus, C., & George, N. (2007). When eye creates the contact! ERP evidence for early dissociation between direct and averted gaze motion processing. Neuropsychologia, 45, 3024–3037. Cooper, R.P., & Aslin, R.N. (1990). Preference for infant-directed speech in the first month after birth. Child Development, 61, 1584–1595. de Gelder, B., & Vroomen, J. (2000). The perception of emotions by ear and by eye. Cognition & Emotion, 14, 289–311. Ewbank, M.P., Jennings, C., & Calder, A.J. (2009). Why are you angry with me? Facial expressions of threat influence perception of gaze direction. Journal of Vision, 9(12), Article 16. Retrieved November 29, 2009, from http://www.journalofvision.org/content/9/12/16.full Farroni, T., Csibra, G., Simion, F., & Johnson, M.H. (2002). Eye contact detection in humans from birth. Proceedings of the National Academy of Sciences, USA, 99, 9602–9605. Frith, C.D. (2008). Social cognition. Philosophical Transactions of the Royal Society B: Biological Sciences, 363, 2033–2039.
1769 Frith, C.D., & Frith, U. (2006). How we predict what other people are going to do. Brain Research, 1079, 36–46. Gamer, M., & Hecht, H. (2007). Are you looking at me? Measuring the cone of gaze. Journal of Experimental Psychology: Human Perception and Performance, 33, 705–715. Graham, R., & LaBar, K.S. (2007). Garner interference reveals dependencies between emotional expression and gaze in face perception. Emotion, 7, 296–313. Hood, B.M., Macrae, C.N., Cole-Davies, V., & Dias, M. (2003). Eye remember you: The effects of gaze direction on face recognition in children and adults. Developmental Science, 6, 67–71. Kampe, K.K.W., Frith, C.D., & Frith, U. (2003). “Hey John”: Signals conveying communicative intention toward the self activate brain regions associated with “mentalizing,” regardless of modality. The Journal of Neuroscience, 23, 5258–5263. Lobmaier, J.S., Fischer, M.H., & Schwaninger, A. (2006). Objects capture perceived gaze direction. Experimental Psychology, 53, 117–122. Lobmaier, J.S., Tiddeman, B.P., & Perrett, D.I. (2008). Emotional expression modulates perceived gaze direction. Emotion, 8, 573–577. Lundqvist, D., Flykt, A., & Öhman, A. (1998). The Karolinska Directed Emotional Faces—KDEF [CD-ROM]. Stockholm, Sweden: Karolinska Institute. Macrae, C.N., Hood, B.M., Milne, A.B., Rowe, A.C., & Mason, M.F. (2002). Are you looking at me? Eye gaze and person perception. Psychological Science, 13, 460–464. Mandel, D.R., Jusczyk, P.W., & Pisoni, D.B. (1995). Infants’ recognition of the sound patterns of their own names. Psychological Science, 6, 314–317. Moray, N. (1959). Attention in dichotic listening: Affective cues and the influence of instructions. Quarterly Journal of Experimental Psychology, 11, 56–60. Nummenmaa, L., & Calder, A.J. (2009). Neural mechanisms of social attention. Trends in Cognitive Sciences, 13, 135–143. Pelphrey, K.A., Viola, R.J., & McCarthy, G. (2004). When strangers pass: Processing of mutual and averted social gaze in the superior temporal sulcus. Psychological Science, 15, 598–603. Senju, A., & Csibra, G. (2008). Gaze following in human infants depends on communicative signals. Current Biology, 18, 668– 671. Senju, A., & Hasegawa, T. (2005). Direct gaze captures visuospatial attention. Visual Cognition, 12, 127–144. Sperber, D., & Wilson, D. (1995). Relevance: Communication and cognition (2nd ed.). Oxford, England: Blackwell. Teufel, C., Alexis, D.M., Todd, H., Lawrance-Owen, A.J., Clayton, N.S., & Davis, G. (2009). Social cognition modulates the sensory coding of observed gaze direction. Current Biology, 19, 1274–1277. Tottenham, N., Tanaka, J.W., Leon, A.C., McCarry, T., Nurse, M., Hare, T.A., et al. (2009). The NimStim set of facial expressions: Judgments from untrained research participants. Psychiatry Research, 168, 242–249.
