Persuasion and the expressivity of gestures in

0 downloads 5 Views 787KB Size Report
Indian. Index and middle finger in V shape on top of the head. Animal. Horse ..... expressions were coded using FACS (Facial Action Coding System, Ekman and ...

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 391

17

Persuasion and the expressivity of gestures in humans and machines Isabella Poggi and Catherine Pelachaud

17.1 Introduction Gestures are manual behaviors that accompany speech by repeating, contradicting, or adding further information to that which speech itself communicates. They are a rich resource for human communication because they convey various kinds of information, have different motor structures, and adapt to various kinds of contexts. After rare, though brilliant, studies in antiquity and later, since the middle of the last century human gesture has been a subject of famous studies by Efron (1941), Ekman and Friesen (1969), Argyle (1988), and Morris (1977). These were followed by research on sign language (Stokoe 1978; Klima and Bellugi 1979; Volterra 1987), and led to a great opportunity for growth in the study of how hands—and the whole body—communicate. Since the 1990s, gesture studies exploded, mainly with the work of McNeill (1992, 2000, 2005), Kendon (1988, 1995, 2004), Santi et al. (1998), Calbris (1990, 2003), Camurri and Volpe (2004), Rector et al. (2003), Müller and Posner (2004), Payratò (1993, 2003), and with the foundation of the journal Gesture and of the International Society for Gesture Studies. But at the same time, another concern solicited interest in gestures—the construction of Embodied Conversational Agents (Johnson et al. 2000; Gratch and Marsella 2001; Cassell et al. 2000; Pelachaud and Poggi 2001; Wachsmuth and Sowa 2002; Kopp, et al. 2004; Bonaiuto and Thórisson, this volume; Sowa et al., this volume). Within this body of research, we focus here on the relationship between gestures and persuasive discourse. The very start of gesture studies is due to their importance in persuasion and dates back to Cicero’s De Oratore and Quintilian’s Istitutiones Oratoriae (100), two outstanding books about persuasive discourse. In the rhetorical tradition, gestures are studied as an indispensable part of “Actio” (discourse delivery), due to their capacity for summoning, promising, exhorting, inciting, prohibiting, and approving, and to their ease in expressing, showing attitudes, and indicating objects of the orator’s thought. Quintilian’s work on gestures, primarily contained in his Book XI, is guided in part by normative intents: He often stresses what gestures should not be used by an orator, while they are—and just because they are—typical of comic actors; but he does so on the basis of a deep and detailed knowledge of the gestures’ forms and meanings. For every gesture he tells us in which segment of the rhetorical structure of discourse it can be used: “placing the middle finger against the thumb and extending the remaining

17-Wachsmuth-Chap17

392

6/3/08

10:16 AM

Page 392

POGGI AND PELACHAUD

three (…) is suitable to the exordium (…). It is also useful in the statement of facts, but in that case the hand must be moved with firmness and a little further forward” (Quintilianus 100, XI, 4, 92). However, from his description one can see that particular movements, sometimes even combined with the same handshape, quite precisely convey the meaning of specific, different speech acts: for example, we “lower [our hands] in apology or supplication (…) or raise them in adoration, or stretch them out in demonstration or invocation” (p. 115). Gestures can also express emotions: “we sometimes clench the hand and press it to our heart when we are expressing regret or anger” (p. 104); but sometimes through them we may induce persuasive effects: “Slapping the thigh (…) is becoming as a mark of indignation, while it also excites the audience” (p. 123). In present times, while a huge number of studies address the use of gesture in everyday conversation, only some are devoted to analyzing gestures in persuasive discourse. Some overview aspects of the body’s relevance in political communication (Atkinson 1984), or focus on the synchronization of gestures with pauses and intonation and other rhetorical devices, frequently used to quell applause (Bull 1986). Some investigate the audience’s physiological, cognitive, and emotional reactions to the politicians’ facial expression and other vocal and bodily behaviors (Frey 2000; Bucy and Bradley 2004). Here we focus on recent work that provides detailed morphological and semantic descriptions of gestures which may give us pertinent insights into the relation between gesture and persuasive discourse. Here we report the views of three authors, Calbris, Kendon, and Streeck, while in subsequent sections we present our own definitions and typologies, in terms of which we will outline our idea of persuasive gestures. In her book The Gestural Expression of a Politician’s Thought, Calbris (2003) analyses the political discourses by Lionel Jospin at the time he was the French Prime Minister. Starting from a view of language as an expression of embodied cognition, she views gesture as “a stage of the mental process of access to abstraction starting from our different perceptive experiences”, the “intermediate stage that, detached from concreteness and already relatively abstract, allows us to access abstraction” (p. 20, our translation). So she studies gesture as a route to understand the “intimate expression of the Speaker’s thought”, and more specifically Jospin’s coverbal gestures as a way to capture his political thought. According to Calbris, a coverbal gesture is a contextual, conventional and motivated sign. Moreover, it is both “polysémique” and “polysigne” (that is, multimeaning and multisign). In fact, a gesture is composed of various physical elements (segment, configuration, orientation, movement, laterality), all of which can bear meaning, so if one or the other component carries a notion, the gesture can provide more than one meaning; it is “polysémique” if it bears different meanings in different contexts, and “polysigne” if it conveys more than one meaning at the same time. An example of polysemic gesture by Jospin is the flat hand palm down over the table, which in one context refers to the reality, concreteness of a phenomenon, while in another expresses a notion of control over something. A “polysigne” gesture is his lowering down the hand in the shape of a pyramid pointed at himself: the pyramid configuration, with converging fingers, expresses the idea of a conclusion; the orientation to himself tells that the enterprise he is talking of is due to himself; the downward movement expresses insistence.

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 393

PERSUASION AND THE EXPRESSIVITY OF GESTURES

Through a careful and insightful analysis of the metaphors exploited by Jospin’s manual behavior, Calbris demonstrates how his gestures, or aspects of their execution—for example the shape of the hand, or even which hand is used, right or left—can express abstract notions like effort, objective, decision, balance, priority, and private or public stance. At the same time, though, they fulfill discourse functions: they can delimit or stress, enumerate or explicate the topics of discourse. A distinction between different functions of gestures is drawn also by Kendon (2004). He analyzes gestures in different cultures and different types of interaction, by people talking about their past life or comment on everyday life events, sometimes also with an argumentative intent, and distinguishes three main functions of coverbal gestures: a referential function, when they convey parts of the propositional content of an utterance; an interactive function, as they help the turn-taking management; and a pragmatic function, as they relate to the pragmatic aspects of an utterance. Within this, he attributes a gesture a performative function if it clarifies the type of speech act that is being performed; a modal function if it alters the interpretation of the utterance, for example through negation or intensification; and a parsing function if it marks the syntactic or textual structure of a sentence or discourse. Then Kendon analyzes some gesture families: for each specific handshape and orientation he singles out typical contexts of use and finds unifying semantic themes. Some of the gestures he analyzes can well have a persuasive use: for example, the “ring” gestures, that bear a meaning of “making precise” or “clarifying”, are used every time this clarification is important “in gaining the agreement, the conviction or the understanding of the interlocutor” (p. 241). Another work directly relevant to our topic is Streeck’s (2007) analysis of the gestural behavior of the Democratic candidates during the political campaign of 2004 in the USA. He found they did not use many different gestures, as to handshape and movement pattern, and that their gestures were very rarely iconic, partly because, just as advocated by Quintilian, iconics are highly characterized as a popular style of gesturing, and partly because their function in political discourse is, in Kendon’s (2004) terms, not so much referential as mainly pragmatic. Yet, among the candidates’ gestures with pragmatic functions, he maintains they mainly fulfilled a parsing function, rather than a performative one, and they did not unequivocally indicate which speech act was being performed, since they did not imply a fixed form–function relationship. For example, Streeck doubts that the “ring” always has a meaning of precision, or that the “power grip” of moving the fist always conveys an assertion of power. However, it is concerning the almost unique gesture used by Howard Dean, the “index up” or “finger wag”, that Streeck seems to attribute a subtly self-defeating effect. This gesture “displays the Speaker’s claim that what is saying is both important and instructive”; but since Dean is enacting this “hierarchical act” in permanence, he might give the impression he is presenting himself as one of “superior knowledge”, thus spoiling, with a body behavior somehow contemptuous toward the audience, the ascendancy that he had previously gained, as reported by Streeck, through his early textual presence. While Streeck is doubtful about the performative functions, he maintains that the candidates’ gestures mainly fulfill the parsing function. Primarily, the gesture tempo is a clear

393

17-Wachsmuth-Chap17

394

6/3/08

10:16 AM

Page 394

POGGI AND PELACHAUD

cue to the discourse structure: the alternation of rapid and slow beats, or whether the stroke combines with the peak syllable or with all stressable syllables, distinguishes between background and foreground information. In this chapter we present a study on the gestures used in persuasive discourse. Our goal is to: (1) distinguish a class of gestures that we could intuitively define as “persuasive” because they contribute a special meaning to the persuasive strength of ongoing discourse; (2) consider the specific meanings of these gestures and possibly to distinguish them into different classes based on the specific role they have within the structure of a persuasive discourse; (3) characterize the properties of a persuasive gesture per se, in the context of a discourse, and in relation to surrounding gestures. In the following sections, we first present our definition of gesture, we overview some studies about gesture expressivity, and propose our own typology of gestures, based on their meanings (Sections 17.2–17.5 and 17.7). Then we illustrate a model of persuasive discourse in terms of the notions of goal and belief (Section 17.8), and we analyze some fragments of persuasive discourse to find out what gestures or aspects of gestures in them specifically have a persuasive function (Section 17.9). Finally we present a computational model of gestures implemented in Greta, an Embodied Conversational Agent (Section 17.6).

17.2 A definition of gesture People use hands, arms, and shoulders to do things, to touch objects, other people, or themselves, and finally to communicate. But what is a communicative gesture? Our definition of communicative gesture stems from a notion of communication based on a model in terms of goals and beliefs (Conte and Castelfranchi 1995). After Poggi (2007), we define as communication the case in which an Agent (Sender) produces a perceivable stimulus (a signal) by performing an action (a word, a gesture, a glance) or exhibiting a morphological trait (a blush, a pale face) in order to achieve the goal (a conscious deliberate intention, an unconscious need, or even a biological instinct) of providing another Agent (Addressee) with some belief or set of beliefs (meaning) mentally represented either as a mental image or in a propositional format (for example, the meaning of “chair”, which includes both some visual image of the shape of a chair and conceptual information such as its being a piece of furniture). The signal and the meaning are linked to each other in both the Sender’s and the Addressee’s mind by a system of rules (communication system). Starting from this definition of communication, we define as communicative gesture a particular movement of hands, arms, or shoulders that is used by a Sender for the goal of communicating some meaning to some Addressee. A communicative gesture is thus a signal–meaning pair: the signal is a particular movement of the upper limbs that produces a particular shape of the hands and a particular position of hands, arms, and shoulders; the meaning is a belief or a set of beliefs. So a gesture can be described and classified as to its signal and as to its meaning. Moreover, signal and meaning are linked to each other in the mind of some people. This signal–meaning link may be either codified, that is, represented in long-term memory in a stable way—as it is for symbolic gestures that can be

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 395

PERSUASION AND THE EXPRESSIVITY OF GESTURES

viewed as forming a lexicon in a gesturer’s mind—or creative, that is invented on the spot— as when one performs a pantomime or an iconic gesture to represent some meaning to which no specific signal corresponds: for example, miming someone playing a cello to mean “cello”, or depicting a cylinder to mean “drainpipe” (Poggi 2007). Let us now focus on the signal and the meaning sides of gestures, respectively.

17.3 The signal of gesture: describing gestures and

their expressivity Various scholars have proposed ways to describe gestures from the point of view of the signal, that is of the motor actions performed by the Sender to produce them, and of their perceptual appearance in the eyes of the Addressee (among others, see Kendon 1988, 2004; Calbris 1990; McNeill 1992). One of the most influential systems for gesture description and transcription came from sign language studies. In 1960, William Stokoe first analyzed the signs of American Sign Language in terms of a small set of parameters: the shape assumed by the hand in making the gesture, the movement performed, the location, over or around the body, where it occurs (Stokoe 1960; 1978). Later other parameters, like hand orientation, wrist orientation, and arm position were added (Klima and Bellugi 1979; Volterra 1987; Prillwitz et al. 1989; Radutzky 1992; HamNoSys 2004). With respect to each parameter, a gesture assumes a specific value out of a number of possible values. This way to analyze signs in sign languages of the deaf was also used to analyze the symbolic gestures and other kinds of gestures of the hearing (Sparhawk 1978; Poggi and Magno Caldognetto 1997; Poggi 2007). The parameters taken into account were: handshape—the shape of the hand in making the gesture (for example, closed fist with extended index finger, or flat hand); location—where the hand is located in making the gesture (e.g. nose, mouth, breast, neutral space); orientation—palm and metacarp direction (downward, leftward, and so on); and movement—how the hand moves in making the gesture. The temporal structure of gestures has been explored by Kendon (1972, 2004), McNeill (1992), and Kita et al. (1998). A gesture has an excursion, from when the hand leaves its resting position up to its coming back to it; a gesture unit is formed by one or more gesture phrases, each including various phases: preparation, stroke, hold, post-hold, and recovery. The stroke is always necessarily present, and it is the phase of the excursion in which the shape of the gesture and the movement dynamics are clearest (Kendon 2004). Since the very beginning of these studies it was clear that the parameter of movement is a very complex one since it includes different subparameters, as described, for instance, by Volterra (1987) and Radutzky (1992) for LIS (Italian Sign Language) and by Romagna (1998) and Poggi (2001, 2007) for Italian Symbolic Gestures. This complex aspect of the gesture execution, often referred to as the “manner of movement”, is also called gesture expressivity, and is somehow linked to an intensity factor (Wallbott and Scherer 1986). In fact, a gesture can be performed more or less intensely, and several elements in the kinematics and dynamic domains enter into its execution: the space that arms and hands cover during the gesture, the speed and acceleration of the limbs are examples of such a factor.

395

17-Wachsmuth-Chap17

396

6/3/08

10:16 AM

Page 396

POGGI AND PELACHAUD

The dimensions of gesture expressivity were disentangled and specified by experimental research. In her studies about style, relying on Allport and Vernon (1933), Gallaher (1992) proposed three main dimensions to represent gesture expressivity: “an Areal factor of broad versus constricted movements, a Centrifugal factor of movements away from the body versus toward the body, and an Emphasis factor of forceful versus weak movements.” (p. 134). Interindividuality differences were found between subjects performing different tasks, but there was also a lot of intraindividual consistency: individuals showed the same behavior quality over the different tasks they performed. “Motor activity”, “kinematics”, and “dynamics” were also found to be consistent factors of individuals (McGowan and Gormy 1976; Rimoldi 1951). Gallaher aimed to define how behaviors differ across individuals, she used the word “style” to refer to gesture expressivity, and looked at the general tendency of behavior characteristics. To avoid using lab studies, she asked subjects to rate their friend’s behavior tendency after a few days of observation. Subjects could use their own terms to characterize their friend’s style. Using factor analysis, four factors were found: 1. expressiveness, corresponding to the energy level with which the behavior is performed; 2. animation, representing the dimension ‘lethargic–animation’; 3. expansiveness, primarily linked to the amount of space taken by the behavior; and 4. coordination, that represents the fluidity of movements (continuous versus hectic). These factors are not independent from each other but do characterize behavior style. Other studies carried in the domain of emotional behavior contributed new findings about gestures expressivity. To study if there is a link between emotions and body movements, Wallbott and Scherer (1986) conducted perceptual studies. They gathered a video corpus of actors portraying emotions (elated joy, happiness, sadness, despair, fear, terror, cold anger, hot anger, disgust, contempt, shame, guilt, pride, and boredom). Twelve actors were used, performing each emotion twice. The performance followed a scenario approach: the actors had to say two non-sense but phonetically-balanced sentences (Banse and Scherer 1996). The annotation scheme to analyze the video encompasses the description of the upper body, arm, and hand configuration as well as movement quality. As to movement quality, three factors were annotated on a three-degree intensity scale (weak, medium, high): movement activity, expansiveness/spatial extension, movement dynamics/energy/ power. The first factor, movement activity, encodes the total number of behaviors. The second one refers to the space covered by the upper limbs and the last one describes the strength with which the gesture is performed. Wallbott and Scherer’s (1986) and Wallbott’s (1998) studies were primarily aimed at finding out if there exist postures and body movements that are characteristic to emotional states, but they also gathered evidence on the relation between emotions and gesture quality. The three movement quality factors were found to differentiate emotions in a statistically significant manner: for example “hot anger” is characterized by high movement activity, expansive movements, and high movement dynamics, while boredom has opposite gesture expressivity, namely, low movement activity, inexpansive movements, and low movement dynamics (Wallbott 1998).

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 397

PERSUASION AND THE EXPRESSIVITY OF GESTURES

On the basis of introspection and observational studies, Romagna (1998) and Poggi (2001, 2007) proposed to analyze the parameter of movement into various subparameters: 1. part of the hand or arm involved. Different parts of the limbs may be involved in the gesture movement: for example saying “no” by shaking only the index finger with still fist is less intense than shaking it along with the whole fist, or even with all the forearm; 2. direction: the point in space toward which the gesture is directed; forward (toward the Hearer), backward (toward the Speaker), outward, inward, upward, downward, and their combinations; 3. path: the route a gesture outlines in space (straight, oblique, circular, half-circular, thrumming, oscillation); 4. size: how long is the run of the movement or how large is its width (long, large, normal, short, narrow); 5. pressure: the strength of the movement, which includes two subparameters: a. tension: the muscular tension of the hand or arm in performing the movement (tense, normal, relaxed, delicate); b. impact: the way in which the gesture stops at the end of the movement (block, normal, skim) 6. tempo, the set of temporal features of the movement, that can be distinguished into: duration: how long the movement lasts (long, normal, short); speed: how speedy the gesture moves (speedy, normal, slow); and finally rhythm: if and how the gesture is repeated, and with which rhythmical structure. The movement is unique when it is very short and not repeated; single if it remains a while on the touched surface and can be repeated, but in a continuous manner; repeated in jerks if the same movement occurs repeatedly and alternatively in two locations; continuous if the movement is repeated with no pause. Finally, partly drawing on the above mentioned literature, Hartmann et al. (2006) proposed the following subparameters of movement to analyze gesture expressivity: 1. spatial extent—how large is the excursion of the gesture from the rest position; 2. temporal extent—the speed of the gesture stroke; 3. fluidity—the presence of sudden changes of direction in the gesture path; 4. power—the rapid change in speed of the gesture stroke (i.e. the acceleration); 5. repetition—whether the same movement is repeated, and how many times.

17.4 The meaning of gesture: a semantic typology Although the link between signal and meaning may be different across cultures (the same signal—the same shape and movement of the hands—may have a different meaning, or no meaning at all, in a different culture), communicative gestures by definition have

397

17-Wachsmuth-Chap17

398

6/3/08

10:16 AM

Page 398

POGGI AND PELACHAUD

meanings, and hence can be distinguished and classified as to the type of meaning they convey. According to Poggi (2006, 2007), the semantic content of our communicative acts, including gestures, concerns different types of Information (see Table 17.1): ◆

Information on the World, concerning the very content the Sender is talking about: concrete and abstract events and entities—objects, persons, animals, times, and places;

Table 17.1 The meanings of gestures Type of Meaning

Meaning

Signal

Person

Indian

Index and middle finger in V shape on top of the head

Animal

Horse

Fists, palms down in neutral space, moving upand down like if holding reins

Object

Scissors

Index and middle finger in V shape opening and closing;

Cigarette

Index and middle finger in V shape upward with palm facing Sender moving back and forth before in front of the mouth

Action

Walk

Index and middle finger in V shape downward, moving alternatively

Property

Thin

Extended little finger up

Relation

Link between two

Fists palms down with index fingers extended parallel approaching to each other

Quantity

Two

Index and middle finger extended upward from fist

Time

Yesterday

Relaxed hand, or index finger extended from fist, moving backward near the shoulder

Place

There

Extended index finger pointing

IDENTITY

Social Id. Image

Nazi salute Noble, fair

Extended arm with flat hand Open hand on heart

MIND

Beliefs

Certainty Metacognitive

Self-evident I am reflecting

Palm up open hand Chin leaning over fist

Goals

Performative

Attention!

Raise flat hand

Topic-comment

This is the topic/ this is he comment

Finger bunch-open hand

Metadiscursive

Quote

Hands raised, palms to Hearer, with index and middle fingers bent

Turn-taking

I want to speak

Raise hand

Backchannel

No, I don’t agree

Shaking index finger left-right

Emotion

Despair

Pulling one’s hair

WORLD

Emotions

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 399

PERSUASION AND THE EXPRESSIVITY OF GESTURES



Information on the Sender’s Identity, concerning stable characteristics of the Sender: sex, age, culture, personality, image, and self-image;



Information on the Sender’s Mind: his/her mental states—beliefs, goals, and emotions— specifically concerning the content and structure of the discourse s/he is delivering.

Thus every gesture can be classified as to the type of information it bears. Among those providing Information on the World, the meanings we convey can fill in various slots within the propositional content of a communicative act. For example, among the symbolic gestures of hearing Italians, some mention persons (“Indian”), animals (“horse”), objects (“scissors”, “cigarette”); other gestures convey actions (“cut”, “smoke”, “walk”), properties (“thin”, “stubborn”), relations (“link between two”), times (“yesterday”), quantifiers (“two”). Among non-codified gestures, typically those of pointing indicate places, while creative iconic gestures generally mention actions and properties. Other gestures inform about the Sender’s Identity. They may show the Sender’s identification with a social, political, or ideological group (see the Nazi salute extended arm with flat hand, or the communist one, raised arm with closed fist). Other gestures, instead, do not inform about a social choice, but rather aim at projecting a particular image of the Sender (e.g. putting the open hand on one’s heart to give the impression of a noble and fair person). Finally, many gestures are “Gestural Mind Markers” (Poggi 2003; Poggi et al. 2004), that is hand movements devoted to convey Information about the Sender’s mind: they inform about the Sender’s beliefs, goals and emotions referred to what s/he is talking about. Within the category of Belief markers, information about the Sender’s beliefs, some gestures inform about the degree of certainty of the beliefs we are mentioning: for example, the palm up open hand (Müller 2004) means that what we are saying is obvious, selfevident, while showing empty hands while lowering forearms means we are quite uncertain about something. Other gestures provide metacognitive information, that is they inform about the source of what we are saying (e.g. snap thumb and middle finger = “I am trying to remember” tells the information we are going to provide is to be retrieved from our long-term memory); or else they inform of the mental state we are in (e.g. leaning chin on fist—Rodin’s “Thinker” posture = I am in concentration). Within the “Gestural Goal Markers”, some express a performative, that is the act a Sender has the goal to perform by his communication: raising the flat hand or the index finger near the shoulder is like saying: “attention please”; the Italian purse hand or tulip hand (Poggi 1983; 2007) means “I ask you a question”. Other gestures distinguish topic and comment in a sentence, thus marking what we want to stress versus what we take for granted—see the up and down movement of beats in general, or specific gestures such as Kendon’s (1995) finger bunch-open hand sequence. Again, metadiscursive gestures inform about our discourse plan and the logical relations among sentences in a discourse: for example bending index and middle fingers of both hands = “quotes” to take distance from what we are saying; in Italy, fist rotating on wrist with curved thumb and index finger to state a link of cause–effect or proximity between two things. But also locating discourse characters or topics in the space and then pointing at them, thus meaning “now I come back to this” is a metadiscursive gesture. Finally, some gestures are used to manage the

399

17-Wachsmuth-Chap17

400

6/3/08

10:16 AM

Page 400

POGGI AND PELACHAUD

turn-taking system in conversation (e.g. raising a hand to ask for speaking turn), or to provide backchannel to the Interlocutor (shaking index finger to say “no, I don’t agree”). Finally, “Gestural Emotion Markers” inform about the Sender’s emotion: for example raising fists to express elation, pulling one’s hair to express despair. Moreover, according to Poggi (2003, 2007) gestures, as all other signals, may have, beside their literal meaning, an indirect meaning, one that the Sender wants the Addressee to understand through automatic or context dependent inferences, and that can be very different from the literal one. So each gesture can be classified in one of the types above both as to its literal meaning and to its indirect meaning, with the two classifications possibly being different. For example, the flat hand outlining a horizontal plane literally means “something on a firm ground”, that is information on a property (Information on the World), but since its indirect (metaphorical) meaning is “I am completely certain”, it comes to be a Belief marker (Information on the Sender’s Mind).

17.5 Meaning of the gesture and meaning of its expressivity In the examples provided so far, the meanings we mention are carried by the gesture as a whole: that is, given a gesture with its particular values of handshape, location, orientation, movement, and its expressivity parameters, the meaning is attached to the globality of the gesture itself; like for words, where the meaning is borne by the word as a whole, not by each of its phonemes. But this might not always or necessarily be the case for gestures, since in a gesture its particular handshape, movement, or location (the particular value the gesture assumes as to these parameters) may well bear a specific meaning by itself (they are more similar to morphemes than to phonemes). For example, in many different signs in LIS the index finger protrusion bears a meaning of something unique (Boyes-Braem 1981; Volterra 1987); the progressive movement from left to right may give the idea of numbering, whatever the handshape (Calbris 2003); and many gestures where hands touch or point at the head refer to mental functions (Kendon 1992).But if this is so, we could suppose that even particular values in the parameters or subparameters of movement bear specific pieces of meaning; that they have a morphemic value. To go deeper in the study of gesture expressivity and of its possible meanings, we now introduce our computational model of gesture expressivity for an Embodied Conversational Agent and present some studies in this field.

17.6 A computational model of gesture expressivity In constructing ECAs—Embodied Conversational Agents, that is virtual agents that exhibit intelligent motion and communicative multimodal behavior driven by internal representations—a relevant goal is to have them communicate by gesture, face, and body behavior, and perform all the communicative acts that humans perform, including persuasive communication. We are still far from reaching this goal, but we want to illustrate some of our work in this direction. We have been developing an Embodied Conversational Agent system, Greta, over several years (Pelachaud and Poggi 2001; Hartmann et al. 2006). At first, we have been concentrating on modeling communicative and affective non-verbal behaviors. Our agent

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 401

PERSUASION AND THE EXPRESSIVITY OF GESTURES

system is based on the taxonomy of meanings presented in Section 17.4. To control the agent’s behavior a representation language, called Affective Presentation Markup Language, APML, has been defined (De Carolis et al. 2004). This language drives the ECA’s behavior from the “communicative functions” (the types of meaning) mentioned in Section 17.4. These functions are instantiated within the behavioral engine that also synchronizes the non-verbal behaviors with the verbal stream (Pelachaud et al. 2002). At first we have been concentrating on building a lexicon of non-verbal behaviors and a language to encode these behaviors, as well as on developing a behavioral engine that computes the multimodal behaviors the ECA should display in a synchronized manner; lately we have turned our attention toward behavior expressivity. Our aim was to embed within the ECA system not only facial behaviors, but also gestures and, within these, not only the contents provided by the gestures or facial behaviors per se, but also those conveyed by their expressivity. Based on the perceptual studies presented below, we have defined a gesture expressivity model encompassing six dimensions that are: ◆

Spatial extent: quantity of space taken by a body part (how extended are the arms, how raised are the brows). This dimension is related to the dimension “expansiveness/ spatial extension” defined by Wallbott (1998) and to the dimension “expansiveness” by Gallaher (1992).



Temporal extent: velocity of execution of a movement (how fast or slow an arm moves or the head turns). It is related to the “animation” factor of Gallaher (1992).



fluidity: level of continuity of successive movements (jerky vs. smooth movements). It is similar to the “coordination” dimension defined by Gallaher.



Power: movement dynamism (weak vs. strong). It is related to the degree of acceleration of body parts. It corresponds to the dimension “movement dynamics/energy/ power” defined by Wallbott (1998).



Overall activation: overall quantity of movement (a lot vs. no movement). This dimension embodies similar information as the “expressiveness” dimension defined by Gallaher.



Repetition: repetition of the stroke of a movement. We have added this dimension from the studies presented above to encompass the notion of stroke expansion. For example the stroke of a gesture can be rhythmically repeated to mark an emphasis.

We have implemented these six dimensions in our system (Hartmann et al. 2006). They act directly on the gesture parameters. A gesture is composed of several keyframes, each defined by a set of formational parameters. The keyframes correspond to the gesture phases (preparation phase, stroke, relaxation (McNeill 1992; Kendon 2004)). Each keyframe is defined by the wrist location, the palm orientation, the finger orientation, the handshape. Handshapes follow the HamNoSys (2004) annotation scheme. The expressivity model modifies the parameters. “Spatial extent” scales the wrist location outward or inward with respect to the body space of the ECA. “Temporal extent” modifies the stroke start of the gesture to vary the duration length of the stroke phase, making it faster or slower. “Fluidity” acts on the wrist trajectory between two successive gestures. This trajectory will be more or less continuous to simulate fluid versus hectic gestures.

401

17-Wachsmuth-Chap17

402

6/3/08

10:16 AM

Page 402

POGGI AND PELACHAUD

The fourth dimension, “power”, adds (or not) overshooting of the target to be reached at the end of the stroke phase. “Overall activation” is modeled by a threshold value that determines whether a behavior will be displayed or not. Finally “repetition” may repeat the stroke of a gesture. 17.6.1

Evaluation

This model of gesture expressivity has been evaluated on two issues. At first we studied whether subjects would recognize each dimension of expressivity separately. Our aim was to evaluate the implementation of each dimension. In a second study we assessed whether these dimensions could be combined to create coherent expressive behaviors. The subjects were 106 students in a French university, aged between 17 and 26. In the first study, subjects had to select which dimension was modified by comparing each video with a reference one. The outcome of this study showed that two dimensions were highly recognized, “spatial extent” and “temporal extent”. “Fluidity” and “power” were recognized above chance level but showed quite a lot of confusion. This was partly due to the incomplete implementation of these parameters. “Fluidity” acts only over consecutive gestures and not within a given gesture. On the other hand, the dimension “power” gathers two notions, “strength” and “tension”. Both are synonymous with power but have opposite effects. While strength is displayed with a strong acceleration, tension is shown by holding the limb in a tense manner. We have partially implemented both these aspects: acceleration of the limb and tenseness of the hand. We believe that by decomposing and implementing “power” into two factors, “strength” and “tension”, we will get a higher recognition rate. For the last two dimensions, “repetition” and “overall activation”, confusion by the subjects on their definition can be seen. Subjects did not distinguish whether a gesture whose stroke was repeated should be considered as a single gesture or a gesture with several stroke. Identically, the presence or absence of behaviors was not perceived as being a sign of behavior activation. In the second study, three gesture manners were considered: vigorous, sluggish, and abrupt. For each gesture manner, four videos were created. One video showed the ECA performing with “neutral” behavior; another one with extreme expressivity (e.g. very vigorous); the third video was done for mild expressivity; and the last one was obtained by setting the expressivity dimensions to values opposite to the expressive ones. Subjects had to rank the videos from the most appropriate to the least. The analysis of the results showed a high recognition rate for the “vigorous” and “abrupt” cases. Subjects distinguished with no problem the videos with the adequate expressivity settings. This result highlights that the expressivity dimensions can be used to create gestures with different manners of movement and that these manners can be differentiated and recognized. On the other hand the third case, the sluggish manner, was recognized poorly by subjects. In all cases, the only parameters that were modified in the videos were the expressivity parameters. In all three cases, the ECA did the same gestures (same gesture morphology) and uttered the same text. But while gesture, text, and expressivity values were all three coherent in the “vigorous” and “abrupt” cases, they were not for the “sluggish” one.

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 403

PERSUASION AND THE EXPRESSIVITY OF GESTURES

Subjects perceived such a discrepancy but did not know which factor should prevail when assigning a meaning to the gesture manner. 17.6.2

Future work

Though we have conducted too few studies to draw real conclusions, we can say that expressivity intervenes in gesture, and at different levels (gesture phrase, whole gesture, whole utterance). At present our expressivity model acts either at the gesture phrase level or at the gesture level. We are currently working on a system that allows the local modulation of gesture expressivity (Mancini and Pelachaud 2007). Moreover, the expressivity model has been extended to facial expressions and head movements as well (Bevacqua et al. 2007).

17.7 The meanings of gesture expressivity Gesture expressivity typically seems to convey meanings concerning emotions, metadiscursive information, intensity and style. Here we present some studies about these semantic areas and how they are conveyed by the expressivity of gestures. 17.7.1

Emotion in gesture expressivity

Perhaps the most typical kind of meaning that is conveyed by gesture expressivity is emotion. On this topic, we have used our computational model of gesture expressivity in two studies. Here we report a study we performed to determine the quantity of meaningful information that is needed to describe behaviors, using two different types of corpus: naturalistic data (EmoTV) and acted data (GEMEP). We also present a study of the role of gesture expressivity as metadiscursive function. 17.7.1.1 Gesture expressivity in naturalistic data

We have conducted a study using EmoTV, a corpus of real data made of video clips from French TV news (Martin et al. 2006). The people interviewed in the video clips showed complex emotions, which might arise from the evaluation of the same event from different perspectives (Scherer 2000; Devillers et al. 2005). Emotion labels, behavior descriptions, and expressivity dimensions were annotated. The methodology of copy-synthesis (Kipp 2004; Martin et al. 2006) was adopted, which consists of driving the ECA’s behaviors from the annotation of the video corpus. In our study the use of the copy-synthesis methodology consists of using annotation extracted from a video corpus to drive the agent’s animation. Perceptual studies are conducted to evaluate the similarity and/or differences between the original videos and the animation of the agent, and the results from this comparison are used in turn to further refine our computational model of expressivity as well as to measure the quality and relevance of the quantity of data being annotated. In the annotation task of EmoTV, gestures were described using an annotation scheme that distinguishes spatial extent, temporal extent, fluidity, power, and repetition. Facial expressions were coded using FACS (Facial Action Coding System, Ekman and Friesen 1978) while gaze was described by taking into account eyes, head direction, and head movements.

403

17-Wachsmuth-Chap17

404

6/3/08

10:16 AM

Page 404

POGGI AND PELACHAUD

All the annotated behaviors were stored in a lexicon containing all the behaviors to be displayed by the agent. The ECA system takes as input the sentence the agent wants to communicate, enhanced with APML (Affective Presentation Markup Language) tags representing the “communicative functions” (the types of meaning) that the agent wants to convey. The expressivity values are also given in input to the system, specifying the expressivity with which the agent will display her behaviors. The ECA system computes the corresponding animation of the agent. Thus the synthesized animation of the agent is a copy of the annotation extracted from the video corpus. The aim of the copy-synthesis methodology was to study if the annotation scheme captured the pertinent information to characterize emotional behaviors. The annotation describes behaviors symbolically, it does not describe all the events and changes happening in the video. Through a first perceptual study, we checked if the annotation scheme encodes enough information for our agent to convey the same emotional state and communicative intents (Martin et al. 2005). Having obtained positive results, we turned our attention to study if complex emotions could be conveyed through particular facial expressions as well as through expressivity values. Using annotations from two video clips, we prepared four animations for the agent, including one that copies the annotation (of the complex facial expressions and of the expressivity values), one that uses a computational model of complex facial expressions (Niewiadomski and Pelachaud 2007), and one that uses data from the literature on facial expression of emotion and on expressivity values for a given emotion (Ekman and Friesen 1975; Wallbott 1998). The task of the perceptual studies was to range the four videos from the most similar to the original video to the least. Another perceptual study showed that subjects did attribute complex emotions to the ECA (Buisine et al. 2006). For both original videos, subjects chose the videos displaying complex expressions. Moreover, in this study, the expressivity values obtained from the video corpus annotation and given to the ECA system affected the ECA’s behavior for a complete speaking turn, that is on a whole sequence of gestures. One of the conclusions of the study is that expressivity dimensions can be used to convey affective states. 17.7.1.2 Gesture expressivity in acted emotions

In another study, we used a corpus of acted data, GEMEP, provided by Tanja Bänziger and Klaus Scherer. GEMEP (GEneva Multimodal Emotion Portrayals) is a corpus for the study of multimodal emotional expression (Bänziger et al. 2006). In this very large corpus, ten actors said two nonsense utterances (Banse and Scherer 1996) for 18 emotions with three different intensity factors (low, medium, high). The actors based their play on a scenario and acted under the indication of a director, but they did not receive any instruction on the facial expression or gesture to use. Perceptual tests were conducted on the videos to validate the emotions displayed by the actors; videos with too high confusion level were disregarded and we worked on a small set of the remaining videos, in which gesture shape and gesture phase were annotated. Gesture expressivity was also annotated at the gesture level (but not at the gesture phrase level). Using these annotations we applied the copy-synthesis methodology to drive an ECA. In contrast to the

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 405

PERSUASION AND THE EXPRESSIVITY OF GESTURES

previous study, here we copied only the expressivity values. Thus the agent does not exhibit the same gesture, but produces her behaviors with the same annotated expressivity. Perceptual tests are being conducted to measure if subjects can perceive the same emotion from the actors and from the agent solely through expressivity values (Castellano and Mancini 2007). Moreover, current attempts are being made for the automatic extraction of gesture expressivity parameters from the analysis of a video using the Eyesweb library (Camurri et al. 2006). 17.7.2

Metadiscursive information

In some cases, expressivity parameters convey information about discourse structure. In a study with two-dimentional cartoons (Tex Avery cartoons), our purpose was to understand how traditional animators use gesture expressivity in characters, but rather than looking solely at the affective function these parameters can play, we wondered if they could have any metadiscursive functions. The sequences of the cartoons analyzed correspond to a conversational setting: a character is talking to other characters, either to persuade them to beware of the “big bad wolf ” in Blitz Wolf or to explain which action to perform at a precise moment in Henpecked Hoboes. The gestures of the character in the video were segmented into their various phases (McNeill 1992; Kendon 2004). Expressivity dimensions were annotated for each gesture phase. By analyzing how the value of the expressivity dimensions varies over time, we derived two rules: ◆

Irregularity: during a short lapse of time, a parameter changes value but goes back to its previous value right after. The parameter changes abruptly in value making a sharp excursion but maintains, overall, a general tendency. Thus the gesture changes expressivity value only for a very short while (a gesture phase).



Discontinuity: there is a sudden change of value but it is maintained from there on: until time t, the gesture exhibits a certain gesture quality, but at time t+1 it shows a very different quality and maintains it.

When irregularity is used on a gesture phase, more visibility is credited to the gesture. As such, the introduced irregularity attracts the attention of the viewer on the gesture by attracting the viewer’s gaze to look at a given location in the image. But irregularity is not a sign of discontinuity in the discourse context: no change of discourse element is introduced. On the other hand, discontinuity is used to contrast discourse contexts: by creating an interruption in gesture manner, it sets the start of a new discourse subject. Thus gesture expressivity may have a pragmatic function at the discourse structure level. Irregularity has an anticipatory function: in attracting the viewer’s gaze, it drives the viewer to pay attention to the important information that will come next. Discontinuity embeds a contrast function: it introduces a new topic in the discourse structure. 17.7.3

Intensity

A typical feature of expressivity is the intensity of gestures: for instance, a gesture produced with high power or low fluidity shows high intensity, one of low amplitude and low velocity does not. But what is there, in semantic terms, behind this sense of intensity?

405

17-Wachsmuth-Chap17

406

6/3/08

10:16 AM

Page 406

POGGI AND PELACHAUD

In what cases, and for which reasons, do we need to communicate some meaning in a more intense way? Gestures that communicate a sense of intensity through their expressivity parameters may be aimed at various communicative goals. In some cases, the goal is to convey Information on the World, for example quantity or measure: we make an intense gesture as we refer to some quantity or measure of objects, or intensity of qualities, that is higher than a standard or default value. In other cases intensity conveys Information on the Sender’s Mind, like certainty (we want to show that we are self-confident and certain of what we are saying) or clarity (we want to be very clear in our communication); often gesture intensity conveys metadiscursive information, namely a meaning of importance (what we are saying is particularly important within our discourse plan); or finally, Information about emotions: we make intense gestures when we feel (or we want to show we feel) some emotion concerning what we are talking about. And very often, providing these types of information through intensity is driven by persuasive goals. 17.7.4

Style

As shown by Gallaher (1992), a higher or lower animation or expansiveness are a cue to a person’s style. In our terms (Ruttkay et al. 2007), style is a part of an individual’s identity. Communicative style is an individual’s stable tendency to prefer some signals or arrangements or aspects of signals instead of others, and is determined in its turn by the individual’s most typical goals and by his or her cognitive, cultural, and personality characteristics. As a consequence, even the tendency to display intensity, emotions, or the need to convey metadiscursive information may be part of a Sender’s style. In every communicative act these aspects of the individual’s identity get instantiated into lowlevel communicative goals that are implemented in particular signals, arrangements of signals, or simply in some values of the signals’ parameters. For example, an enthusiastic person will usually make ample and energetic gestures, while a shy one will express his tendency to avoid social contact through small amplitude and slow movements of gestures.

17.8 Gestures and persuasion After this overview of the signal and meaning aspects of gestures, we finally wonder: are there some gestures, types, parts, or aspects of gestures that we intuitively feel as “persuasive”? Before answering this question, we provide a definition of persuasion and illustrate a model of persuasive discourse. The model of persuasion we adopt is based on a view of mind and social interaction in terms of goals and beliefs (Conte and Castelfranchi 1995). 17.8.1

A model of persuasion in terms of goals and beliefs

According to Poggi (2005), persuasion is a case of social influence, that is a case in which an Agent A causes the likeliness for another Agent B to pursue a goal G to increase or decrease: in other words, if one aims at persuading, one wants another person to have a goal that he did not have before, or to give up a goal he had. But while there are many

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 407

PERSUASION AND THE EXPRESSIVITY OF GESTURES

different ways of influencing people—ranging from violence to education—persuasion is characterized by the following features: 1. It is a communicative way of influencing people, that is one where A not only wants to influence B, but also wants B to know this. 2. It is carried on through a device of “goal hooking”: A, in order to have B decide to pursue a goal GA (the goal “proposed” by A) tries to convince B, that is to make B firmly believe that bringing about GA is a means (a subgoal) to achieve another goal GB, which B already had as a goal of his. (In a sense, then, persuasion does not trigger totally novel goals, but only hooks new subgoals to pre-existing ultimate goals—see discussion in Poggi 2005). 3. It aims at influencing B by relying on B’s free choice (Piattelli Palmarini 1995): B is free to take goal GA as a goal of his own (to do what A wants), not because A is threatening or promising, but as an autonomous choice of his for achieving GB. In other words, if persuasion is successful, in the end B is sincerely convinced that GA is the very best thing to do, and that, had he understood it before, he would have chosen to pursue GA even without external solicitation or coercion. In persuasion, A wants to influence B, that is to activate goals in B, through providing B with beliefs. In particular, to have B activate and pursue goal GA, A must convince B (cause B to firmly believe) that a means–end relation holds between GA, the goal proposed by A, and GB, the goal of B to which A is hooking it. For example, if politician A wants to persuade elector B to vote for his party (GA), he may argue that voting for his party is the only or the best means for a goal, that B in fact has, of enhancing employment (GB). To do so, A must lead B to attribute a high value to goal GA, and possibly a higher value than that attributed to other possible goals like GC—say, voting for other parties. Therefore, a very important part of persuasion is evaluation. In fact, in the model adopted (Miceli and Castelfranchi 1989) evaluating means to have beliefs about whether and how much some object, person, or event is a means for a given goal. Moreover, to convince B of the desirability of GA, A may resort to three different strategies, as first described by Aristotle (1973): logos, ethos, and pathos. Logos are the Persuader A’s rational arguments; pathos is A’s capability to trigger or evoke such emotions that can activate goal GA in the Persuadee B, and it is based on the high motivating power of emotions (Castelfranchi 2000); ethos is the image that the Persuader can project of herself, her credibility and reliability. In fact, the Persuadee is likely to be convinced (that is, to raise the level of certainty with which he believes GA is a right thing to do) inasmuch as he trusts the Persuader. But trust, according to Falcone and Castelfranchi (2003), encompasses two aspects, that must both be present at the same time: benevolence and competence. On the one side, the Persuader must look benevolent to the Persuadee: she must look to be one who wants B’s goals to be fulfilled, who does not want to hurt B, and more specifically does not want to cheat, to deceive B; on the other side, the Persuader must look competent: she must look to be one who is able to fulfill her own goals and the goals of others that she takes care of, one who has the necessary capacity and expertise for doing so. In other words, according to Falcone and Castelfranchi (2003) and Poggi (2005), trusting someone implies assuming s/he is both clever and good (at least to me).

407

17-Wachsmuth-Chap17

408

6/3/08

10:16 AM

Page 408

POGGI AND PELACHAUD

17.8.2

Persuasive gestures?

On the basis of this model, we can define as “persuasive gestures” those gestures in a discourse that have the goal of persuading, and to do so convey some parts or aspects of the cognitive structure of a persuasive communicative act, that is the types of information typical of persuasion. But what are the types of information contained in the cognitive structure of a persuasive discourse? When A, the Persuader, wants to persuade B, the Persuadee, to activate and pursue a goal GA, A should: 1. state the goal GA 2. state that goal GA is one of high value, possibly of higher value than alternative goals GC, GD … 3. show that GA is linked to a goal GB of B by a means–end relation, and state which is the goal GB 4. induce emotions apt to trigger goal GA in B (pathos) 5. convince B that what A says about the desirability of GA and its means–end relation to GB is true, by providing logical arguments (logos) 6. give the impression that A is reliable (ethos), by showing that a. A is certain of what s/he is saying in order to induce confidence in B (certainty) b. A proposes goal GA not out of her own concern but in order to the goals of B (benevolence) c. A has good capacities of action and planning (competence). Thus, to assess how persuasive gestures are in a discourse, we wondered if some gestures or aspects of gestures convey parts of this cognitive structure, of these “persuasive” elements, as part of their meaning. To answer this question, we analyzed two fragments of persuasive discourse.

17.9 Gestures in political persuasion: a research study We conducted an observational study on the gestures performed during political campaigns. The fragments we analyze here are drawn from two political debates held in Italy, a few days before political elections, in March 1994 and March 2006 respectively. In both, the political discourse under analysis is one of the leader of the center-leftist array— Achille Occhetto in 1994 and Romano Prodi in 2006—and in both, their opponent is Silvio Berlusconi, the leader of the center-right array. Fragment 1 lasts 30” and contains 14 gestures, fragment 2 is 1’ 32” long and contains 27 gestures. 17.9.1

An annotation scheme for gestures in persuasive discourse

The gestures in persuasive discourse were analyzed according to the annotation scheme in Table 17.2, which relies on the principles for the analysis of body behavior stated in Poggi (2007).

2. Speech

Si è detto recentemente con ironia: Recently people ironically said

“ma guarda Prodi fa il discorso con la CGIL e con la confindustria” Ok look Prodi is talking to both trade unions and factory owners

Sì, faccio il discorso con la CGIL e la cofindustria Ya I am talking to trade unions and factory owners

Perché se voglio prendere una decisione Because if want to make a decision

1. Time

1 0.00.2

2 0.00.6

3 0.00.8

4 0.00.13

Right hand turns the pen repeatedly

Object manipulation

I defy you

ISM Performative

ISM Metadiscursive

ISM Metadiscursive

5. Meaning type

I am self-confident in doing so

I want you to laugh about them

6. Indirect meaning

ISM Certainty PERS (Ethos Competence)

ISM Performative PERS (Pathos)

7. Meaning type

10:16 AM

Left arm near body, hand on hip, bowing rhythmically Spatial extent: +1 Fluid: – 0.5 Power: + 0.5 Temporal extent: + 1 Rep.: 4

I am miming those who ironically judge by looking down to us

Open, Public I show, I exhibit

4. Literal meaning

6/3/08

Left arm near body, hand on hip + shoulder shaking Spatial extent: 0 Fluid: +1 Power: – 1 Temporal extent: 0 Rep.: 0

Hands palms up oblique open outward Spatial extent: +1 Fluid: – 0.5 Power: – 0.5 Temporal extent: 0 Rep.: 0

3. Gesture description

Table 17.2 Prodi’s discourse; Fragment 1

17-Wachsmuth-Chap17 Page 409

PERSUASION AND THE EXPRESSIVITY OF GESTURES

409

Io dovrò mediare sulla situazione complessa del paese I will have to negotiate about the complex situation of our country

Non posso far finta di agire su un paese I cannot pretend to act in a country

diverso da quello che è different from what it is

5 0.00.15

6 0.00.21

7 0.00.23

ISM Emotion

ISM Certainty PERS (Ethos Competence)

ISM Metadiscursive PERS (Logos)

Important

Left h. ext. index f. palm I am sure down, pointing forward up; Spatial extent: – 1 Fluid: – 1 Power: +1 I feel an emotion Temporal extent: +1 Rep: 3

ISM Certainty PERS (Ethos Competence) IW Action

IW Quantity

5. Meaning type

I am sure, I assert, I am rather decided I take back to the previous place

All (the complex situation)

4. Literal meaning

I incite you to feel your Italian identity

I am stubborn

6. Indirect meaning

ISM Performative PERS (Pathos)

ISI Personality PERS (Ethos Competence)

7. Meaning type

10:16 AM

Left h. ext. index f. palm down; forearm rotates moving left-right four times Spatial extent: +1 Fluid: – 1 Power: – 1 Temporal extent: 0 Rep.: 4

Hands palms down, with ext index fingers describe a circle from in outward, stopping in 7 points Spatial extent: + 1 Fluid: – 0.5 Power: + 0.5 Temporal extent: + 1 Rep.: 0

3. Gesture description

6/3/08

Rep.= repetitions; h.= hand; f.= finger; ext.= extended; bold = pitch accent; IW = Information on the World; ISM = Information on the Sender’s Mind; ISI = Information on the Sender’s Identity; PERS = Persuasive

2. Speech

410

1. Time

Table 17.2 (continued) Prodi’s discourse; Fragment 1

17-Wachsmuth-Chap17 Page 410

POGGI AND PELACHAUD

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 411

PERSUASION AND THE EXPRESSIVITY OF GESTURES



Column 1 contains the time in the video.



Column 2 contains the speech parallel to the gesture under analysis.



Column 3 contains a description of the gesture: specifically, the values on the parameters of handshape, location, and orientation are described in a discursive way; in four subparameters of movement (Spatial extent, Fluidity, Power, Temporal extent) values are quantified in a 5 point scale: –1 (low), –0.5 (medium low), 0 (default), +0,5 (medium high), +1 (high); while for the last subparameter, Repetition, the number of repetitions after the first movement, is written down. The values were rated on a qualitative ground, simply viewing the video, by two independent raters.



In Column 4 we provide a verbal formulation of the gesture’s literal meaning, in some cases also distinguishing if, within the gesture, distinct meanings are borne by values in the various parameters or subparameters.



In Column 5 we classify these meanings according to the typology above, and if this meaning has a persuasive import (PERS), we write this under its semantic classification. In this case we also specify which of the three persuasive strategies it pursues, whether logos, pathos, or ethos; and for ethos we further specify whether the benevolence aspect or the competence aspect of it is salient.



In Column 6, for gestures which have an indirect meaning beside their literal meaning, we write a verbal formulation of the indirect meaning.



In Column 7 we classify the indirect meaning written in Column 6 in terms of the semantic typology and in terms of the persuasive strategy.

17.9.2

An example of analysis

We present the analysis of the first seven gestures in Prodi’s fragment. Romano Prodi, chief of the center-leftist array and candidate to the elections to be held 20 days later, is answering a question during a political debate. He says: Si è detto recentemente con ironia: “Ma guarda Prodi fa il discorso con la CGIL e con la confindustria”. Sì faccio il discorso con la CGIL e la confindustria; perché se voglio prendere una decisione io dovrò mediare sulla situazione complessa del paese; non posso far finta di agire in un paese diverso da quello che è. [Recently people ironically said: “Oh look, Prodi is talking to both trade unions and factory owners”. Yes am talking to trade unions and factory owners; because if I want to make a decision, I will have to negotiate about the complex situation of our country; I cannot pretend to act in a country different from what it is.]

At first Prodi quotes an ironic objection to his political action in order to counterobject to it. While saying “Si è detto recentemente con ironia” (“recently people ironically said”; line 1), his hands with palms up a bit oblique open outward: an iconic gesture referring to something open, public; a way to open a new topic in your discourse, like when the curtain opens on the stage: a metadiscursive gesture. Then (line 2), while saying “ma guarda Prodi fa il discorso con la CGIL e con la confindustria” (“Oh look, Prodi is talking to both trade unions and factory owners”), he puts his left hand on his hip, and at the same time, with his chest erected, he shakes his shoulders (first left shoulder forward and

411

17-Wachsmuth-Chap17

412

6/3/08

10:16 AM

Page 412

POGGI AND PELACHAUD

right backward, then the reverse). His hand on hip bears the meaning of someone taking the stance of a judge, the erected bust shows self-confidence, almost, a self attribution of superiority, and shoulders shaking show that he is gloating for the other being judged and ridiculed. This whole movement is a way to mimic those saying the quoted sentence, while making fun of them. Actually, he is somehow meta-ironizing: he is being ironic about others’ irony, by ridiculing their attitude of superiority through exaggeration. Irony in fact is often brought about through hyperbole (Attardo et al. 2003). This gesture has a persuasive import in that ridiculing brings about the Addressees’ emotion of amusement, thus exploiting a pathos strategy in order to elicit a negative evaluation of the ridiculed people. And by inducing a negative evaluation of the oppositors Prodi intends to lead the audience to prefer him. Then he says (line 3): “sì faccio il discorso con la cigielle e la confindustria” (“Yes I am talking to trade unions and factory owners”), again with left hand on hip, but with bust bowing five times rhythmically, simultaneously with the stressed syllables in the concomitant sentence. The bust bow, like an ample nod, means: “I acknowledge that what you say is true”, while the hand on hip claims self-confidence. But acknowledging that an accusation or a criticism is true while showing confidence means that you accept it as a neutral or even positive statement, devoid of any negative evaluation: thus the combination of the two movements means “I will really do what they accuse me of ”, conveying a meaning of defiance, hence giving the impression of an even higher self-confidence. Then, while saying “se voglio prendere una decisione” (“if I want to make a decision”) (line 4), his right hand manipulates the pen by turning it repeatedly: an object manipulation gesture that is not communicative; and while saying (line 5) “io dovrò mediare sulla situazione complessa del paese” (I will have to negotiate about the complex situation of our country), his hands with extended index fingers, palm down, describe a circle moving symmetrically from in outward, as if pointing and counting many things or persons around him. This gesture thus carries Information on the World, bearing a meaning of quantity through an iconic gesture. Then he says (line 6): “non posso far finta di agire in un paese” (“I cannot pretend to act in a country”), and his left forearm rotates from left to right with extended index finger forward and closed fist palm down. He repeats the gesture four times simultaneously with the four pitch accents of the phrase. Here the shape of the hand, index finger extended, means decision and assertion; the direction of movement of the hand rotating inward in a cyclic way gives an impression of someone taking something back every time to the right place, almost in a stubborn way; and this indirectly conveys (Column 6) a self-presentation of “I am stubborn”: Information on the Sender’s Identity that provides a persuasive element of ethos, concerning in particular the Sender’s competence (Column 7). At the same time, the rhythmical repetition of the gesture conveys the importance of what is being said (Column 6), and thus a persuasive element of logos. Finally, at the end of the previous sentence (line 7), while saying “diverso da quello che è” (“different from what it is”), the gesture is the same as before, but the spatial extent decreases (–1) while the power increases (+1). This expresses an emotion, possibly toward Italy, the “country” mentioned in the sentence, which might indirectly convey the

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 413

PERSUASION AND THE EXPRESSIVITY OF GESTURES

intention to transfer pride of the Italian identity through contagion (Column 6), and thus provide a persuasive element of pathos. 17.9.3

Do persuasive gestures exist?

From the analysis of this and other fragments of persuasive discourse, we can state that, actually, something that we can really call a “persuasive gesture” does not exist. The persuasive import of a gesture seems to be contained more in the “expressivity” parameters than in the global meaning of the gesture, and more in the inferences the gesture encourages than in its literal meaning. Apart from very rare gestures that mean “I incite you to…” or bear other kinds of performatives, it is very rare to find gestures that convey meanings like “I want to convince you that…” or “I want to persuade you to…”. Thus, to find persuasive strength in manual communication you must look for those gestures that convey meanings in some way linked to the communicative strategies of logos, ethos, and pathos. In particular, the gestures that have a persuasive import should be, by definition, those that convey the following types of information: 1. Importance. If I hold that something is important, to obtain it will be a high-value goal, possibly one I want you to activate and pursue. In general, gestures conveying the meaning “important” mention the high value of a proposed goal, thus trying to convince the Perusadee to pursue it. This meaning is typically contained in some performative gestures, like incitations and requests for attention, or other gestures like Kendon’s (2004) “grappolo” (“finger bunch”), that convey a notion of importance as their very meaning; but “important” is also the core meaning of beats, since every beat stresses a part of a sentence or discourse, hence communicating “this is the important part of the discourse I want you to pay attention to and to understand”; or finally this can be the meaning of either irregularity or discontinuity in movement. 2. Certainty. To persuade you I must convince you, that is, cause you to have beliefs with a high degree of certainty: beliefs about what goals to pursue and how to pursue them (means–end relations). And in order to induce certainty in you, I should generally show myself as self-confident and certain about what I am saying. This is why gestures that convey high certainty, like the “ring” mentioned by Kendon (2004) and Streeck (2007) may be persuasive. Yet, since persuading can mean either to convince someone to believe something, or to convince someone to do something, the gestures that convey a high degree of certainty generally persuade to believe, and only indirectly persuade to do. 3. Evaluation. To express a positive evaluation of some object or event implies that it is a useful means to some goal; thus, to bring about that event or to obtain that object becomes desirable, a goal to be pursued. In a marketplace, to convince someone to buy a food, a “cheeck screw” (rotating the tip of the index finger on cheek), that means “good”, “tasty”, made by a grocer, would be a good example of persuasive gesture. Of course, we cannot find an example like this in our fragments, due to obvious reason of social register. However, as we see below, a persuader, to pursue an ethos strategy, can make gestures that induce a positive evaluation of himself.

413

17-Wachsmuth-Chap17

414

6/3/08

10:16 AM

Page 414

POGGI AND PELACHAUD

4. Sender’s benevolence. In persuasion not only the evaluation of the means to achieve goals is important, but also the evaluation of the Persuader: the Sender’s ethos. If I am benevolent to you, you can trust me, so if I tell you that a goal is worthwhile you should pursue it. A gesture driven by the ethos strategy of showing one’s moral reliability is, for example, putting one’s hand on one’s breast, that means “I am noble, I am fair”. This gesture is quite frequent in political communication (Serenari 2003), and it is also represented in our corpus. 5. Sender’s competence. If I am an expert in the field I am talking about, if I am intelligent, efficient, you might join with me and pursue the goals I propose. For example, in one of our fragments the candidate Berlusconi, in talking of quite technical things concerning taxes, uses his right hand curve open, with palm to left, rotates rightward twice. This gesture means that he is passing over these technicalities, possibly difficult for the audience; but at the same time the relaxed appearance of his movement lets them infer that he is smart because he is talking of such difficult things easily, and in an unconstrained manner. This provides an image of competence in what he is talking about. 6. Emotion. If I express an emotion, and this is transferred to you through contagion (Poggi 2004), since emotions trigger goals, a goal will be activated in you, thus implementing a pathos strategy. At line 7 of Table 17.2, Prodi says: “(I cannot pretend to act in a country) different from what it is”, obviously referring to Italy. The movement of his forearm shows low spatial extend and fluidity (it is short and jerky) and high power and velocity, thus conveying an emotional load; by this he could aim to transmit a sense of pride in being Italian, thus eliciting the Italians’ desire to vote for him. Among these types of information, (6) emotion, typically makes part of a pathos strategy; (5, 4, and 2), the Sender’s competence and benevolence and Sender’s certainty, are clearly ethos information; while (1 and 3), importance and evaluation, are generally conveyed through a logos strategy. Nonetheless, these categories can merge with each other: for example, expressing an emotion about some possible action or goal may imply it is an important goal for me (and it should be so for you). In this case, at a first level there is a pathos strategy—the goal of inducing an emotion, but this pathos is aimed at demonstrating the importance of the proposed goal, thus conveying a logos strategy at the indirect level. Now, is there a specific set of gestures that convey these types of meaning? Of course, the category of Gestural Mind Markers, those that bear information about the Sender’s goals, beliefs and emotions, are the best candidates. Yet, not all Gestural Mind Markers are necessarily persuasive gestures. Of course, some performative gestures are persuasive, for example those conveying incitation, request of attention, or encouragement; then, among metadiscursive ones, typical gestures communicating importance are the beats, that mean “this part of the sentence/discourse is the most important”; and, finally, among the gestures concerning degrees of certainty, obviously only those that convey high certainty are persuasive. But besides this, what comes out of our analysis is that the persuasive types of information are not necessarily conveyed by the gesture per se. First, they can be contained not

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 415

PERSUASION AND THE EXPRESSIVITY OF GESTURES

in the global meaning of the gesture but in its “expressivity” parameters, that is, in the particular movement of a gesture which, in itself, does not convey them. For example, gestures providing Information on the World, like a deictic pointing at someone or oneself, or gestures that simply mean “nothing” or “decide” may be seen as persuasive just due to the strength or amplitude with which they are made. In this sense, we agree with Calbris (2003) that gestures can be “polysigne” (multisign), that is a gesture can convey some meaning with its handshape but also another meaning through its movement or its orientation. Thus, in some cases a gesture provides persuasive information only through its manner of movement, not through the meaning conveyed by its shape or by its global meaning. This is clear in our annotation scheme, where the specification of the meaning is written on the same line of the corresponding signal or aspect of signal. For example, in line 7 of Table 17.2 the meaning “I feel an emotion” is conveyed only by the value +1 on the parameter “power”. Second, the persuasive import of a gesture may not appear from its literal meaning, but from its indirect meaning, that is from the inferences the gesture induces in the Addressee. For instance, when the Sender expresses emotions, either through gestures that mention some emotion explicitly or through the gesture expressivity, this is not yet the pathos aspect of persuasion, which is defined as the emotion elicited in the Addressee. Rather, in this case we can say that the Sender expresses an emotion, but his goal is to induce the same emotion in the Audience. So an expression of emotion only indirectly conveys a pathos strategy. 17.9.4

Persuasive gestures and persuasive expressivity

Let us now see what aspects of manual behavior have a persuasive import in Prodi’s fragment. First, we observe that out of seven manual behaviors, four (lines 2, 3, 6, and 7) have a persuasive import, while one (line 1) provides Information on the Sender’s Mind but not of a persuasive kind; another one (line 5) gives Information on the World, and finally one (line 4) is a non-communicative Object manipulation. In the persuasive cases, three (lines 3, 6, and 7) bear meanings of certainty, and therefore an image of self-confidence functional to an ethos strategy, and among these, in line 3 this is conveyed only indirectly through a performative of defiance. The pathos strategy is pursued by two gestures (lines 2 and 7), and only through their indirect meaning. Finally, logos (the value of the proposed goal) is conveyed by gesture 6 at the literal level. As hypothesized, the persuasive import is not always borne by the gesture itself but often only or mainly by its expressivity parameters. This can also be clearly seen by comparing some fragments in the GEMEP corpus with a passage in Occhetto’s fragment. In one of the videos of the GEMEP corpus, an actor points toward the camera with his index finger while performing anger. The stroke of this deictic gesture is repeated several times. The annotation of the expressivity dimensions shows that the spatial extent is medium–high (+0.5), the temporal extent is very high (+1), the fluidity is low (–1), and the power is medium–low (–0.5). In his debate with Berlusconi, Occhetto does several deictic gestures in a row (Poggi 2005). While reminding the audience of a case in which he was fair to Berlusconi’s brother, he now accuses Berlusconi of not being fair to him.

415

17-Wachsmuth-Chap17

416

6/3/08

10:16 AM

Page 416

POGGI AND PELACHAUD

At first Occhetto points toward Berlusconi with his chin up, then he makes three deictic gestures. They are three deictic gestures and not a single deictic gesture with three stroke repetitions. At each new deictic gesture, the power dimension of the gesture expressivity increases. The spatial and temporal extent of the deictic gestures is medium–high (+0.5), while the power increases from medium–high to very high (from +0.5 to +1). The last deictic gesture is done with a much stronger power than the execution of the first gesture. As explained above, Occhetto accuses Berlusconi by indicating very clearly to whom he is referring. The gesture’s expressivity also shows his anger arising from the cause of his accusation. In these two examples, a deictic gesture is used as a sign of anger, and as an accusation. In both the emotion of anger is present. However, in the GEMEP example the actor communicates an affective states, whereas in the TV debate the politician aims to persuade his addressees (the journalist and the audience) that Berlusconi is responsible for some acts. In the former example, the expressivity of the actor is constant over the whole video, while this is not the case with the latter. The power dimension varies greatly while the gesture is being repeated, and it is responsible for its persuasive import. 17.9.5

Persuasive hands in political discourse

We can now overview the quantity and quality of persuasive gestures in the two fragments considered. Based on the analyses with the annotation scheme of Table 17.2, the meanings borne by manual communication can be computed by taking into account the gestures not only in their globality but in a more fine-grained fashion: inside each gesture, more than one communicative unit can be found, and in the same vein also the persuasive import is computed in terms of these communicative units, conveyed also by the values in the different parameters and in the expressivity of movement. Table 17.3 shows that the number of communicative units (line 3) is higher than the number of gestures per se (line 2); then the percent of the total communicative units having a persuasive import (line 4) is computed. And finally, how many of all the persuasive units using a logos, pathos, ethos–competence, and ethos–benevolence strategy (lines 5–8) are computed. Table 17.3 The gestures of Occhetto and Prodi Occhetto Number

Prodi %

Number

1. Length

30”

1’ 32”

2. Gestures

14

27

3. Communicative units

24

49

4. Persuasive units

%

20

83

34

69

5. Logos

1

5

8

24

6. Pathos

6

30

4

12

7. Ethos competence

9

45

17

49

8. Ethos benevolence

4

20

5

15

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 417

PERSUASION AND THE EXPRESSIVITY OF GESTURES

Figure 17.1 Occhetto’s strategies in gestures.

The analysis of the two fragments (Figures 17.1 and 17.2) highlights some interesting differences, some probably depending on the different contexts of discourse, and others presumably due to the different style of the two orators. The percentage of persuasive units within the total of communicative gestures is higher for Occhetto than for Prodi: 83% of Occhetto’s meaningful manual behaviors are persuasive, as opposed to 69% for Prodi. Prodi in fact sometimes uses iconic gestures to illustrate his discourse: gestures conveying Information on the World that often have no persuasive import or only have some in the gesture expressivity. Within persuasive gestures, the persuasive strategies adopted are somewhat different between the two politicians. They differ as to the proportion of pathos and logos: Occhetto relies much more on pathos than on logos gestures (30% vs. 5%), while Prodi uses them in a more balanced way, but with a preference for logos (26% vs. 17%). On the other hand, a similarity between the two orators is that for both the majority of gestures pursue an ethos strategy: 65% for Occhetto and 64% for Prodi. Both tend to project more an image of competence than one of benevolence, but Occhetto displays the ethos–competence strategy less and the ethos–benevolence more, as opposed to Prodi (45% vs. 49%, and 20% vs. 15%, respectively). This may be due to at least two different reasons. First, Occhetto’s

Figure 17.2 Prodi’s strategies in gestures.

417

17-Wachsmuth-Chap17

418

6/3/08

10:16 AM

Page 418

POGGI AND PELACHAUD

Table 17.4 Gesture versus expressivity, literal versus indirect Logos Occhetto

Prodi

Pathos

Gesture Expressivity Literal Indirect

1

1

6

Gesture Expressivity Literal Indirect

2 6 4 4

2 2

6

4

Ethos competence

Ethos benevolence

6 3 4 5

3 1 1 3

13 4 12 5

4 1 5

fragment is taken from a passage of the political debate in which he is attacking Berlusconi’s position from an ethical point of view and therefore he aims to project an ethically valuable image of himself, to highlight the differences between himself and the opponent. In Prodi’s fragment, by way of contrast, he is describing his program and thus he wants to project the image of one who is able to carry it on in an effective way. Second, the two orators have different political origins: Prodi is a center-leftist coming from a former catholic party (the Christian Democrats), while Occhetto is a communist; and Berlusconi still makes appeal to the old prejudice that the Communists “eat the kids”!— hence a higher need for Occhetto to show his image of benevolence. From our annotation scheme, we can also compute how much each strategy is adopted at the literal or the indirect level, and if it dwells more in the expressivity parameters or in the gesture as such—that is in other parameters or in its global meaning (Table 17.4). As hypothesized, Figures 17.3 and 17.4 show that the persuasive import is contained both in the gestures as such and simply in their expressivity parameters; but while pathos is contained more typically (or exclusively, for Occhetto) in the expressivity of gesture, ethos in both orators is more often conveyed by the gesture as a whole. As to the literal/indirect dimension, as already observed, the pathos strategy is conveyed only at the indirect level in both politicians. They differ, though, in their level of

Figure 17.3 Occhetto’s strategies in gesture and expressivity.

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 419

PERSUASION AND THE EXPRESSIVITY OF GESTURES

Figure 17.4 Prodi’s strategies in gesture and expressivity.

indirectness (Figures 17.5 and 17.6): while Prodi conveys his persuasive gestural messages almost half of the time in a direct way (16 out of 34 persuasive items, 47%), Occhetto’s persuasive messages are preferably indirect (the persuasive import is explicit five times out of 20, 25%). This is coherent with previous analyses of Occhetto’s discourse (Poggi 2007), according to which, evaluative information is also preferably conveyed in an indirect way. To sum up, pieces of the persuasive import of a gesture can be borne either by entire gestures or simply by the expressivity of gestures that, per se, do not tell anything persuasive; and, moreover, they may be either explicitly stated or let to be inferred by the Addressee.

17.10 Conclusion In this chapter, through observational, experimental, and computational studies, we have investigated the structure and the meaning of gestures, and their role in persuasion. Starting from a model that highlights the differences among the persuasive strategies of logos, ethos, and pathos, we have analyzed gestures in some fragments of political

Figure 17.5 Occhetto’s direct and indirect strategies.

419

17-Wachsmuth-Chap17

420

6/3/08

10:16 AM

Page 420

POGGI AND PELACHAUD

Figure 17.6 Prodi’s direct and indirect strategies.

discourse; we have seen that they can have a persuasive value, and yet this is often due not to the very meaning of the gesture per se, but to its expressivity parameters: it is how I make a gesture, more than what gesture I make, that is more or less persuasive. And this is so because not only the globality of a hand movement, but even the speed, the amplitude, the jerkiness of my gestures bear meanings: they tell you that I am feeling some emotion, that what I am saying is important, that I am serene or enthusiastic, that I am relaxed, hectic, or strong. This study again shows how rich, mysterious, and hence challenging to investigate, are the potentialities of the human body, and of gestures in particular, and what delicate intertwining of different competences is required to create Embodied Agents that exploit all the subtleties of human communication. A final warning to keep in mind. Since all the nuances of gestures in an Embodied Agent, just as in a human, may have strong persuasive effects on the Users, thanks to their conveying meanings, the researchers in this field must be conscious of their responsibilities and be careful in handling an object, a tool, that can have a deep social influence on other people.

References Allport GW and Vernon PE (1933). Studies in Expressive Movements. New York: McMillan Company. Argyle M (1988). Bodily Communication. London: Methuen. Aristotle (1973). Retorica. Bari: Laterza. Atkinson M (1984). Our Masters’ Voices. The Language and Body Language of Politics. London: Methuen. Attardo S, Eisterhold J, Hay J, and Poggi I (2003). Multimodal markers of irony and sarcasm. Humor. International Journal of Humor Research, 16, 243–60. Banse R and Scherer K (1996). Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70, 614–36. Bänziger T, Pirker H, and Scherer K (2006). GEneva Multimodal Emotion Portrayals: A corpus for the study of multimodal emotional expressions. In L Deviller et al., ed. Proceedings of LREC’06 Workshop on Corpora for Research on Emotion and Affect, pp. 15–19. Genova, Italy: Springer. Bevacqua E, Mancini M, Niewiadomski R, and Pelachaud C (2007). An expressive ECA showing complex emotions. In P Olivier and C Kray, eds. Proceedings of the AISB’07

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 421

PERSUASION AND THE EXPRESSIVITY OF GESTURES

Annual Convention. Workshop Language, Speech and Gesture for Expressive Characters, pp. 208–216. Newcastle UK, April 2007. Boyes-Braem P (1981). Significant Features of the handshape in American Sign Language. Unpublished PhD. Thesis, University of California, Berkeley. Bucy EP and Bradley SD (2004). Presidential expressions and viewer emotion: counterempathic responses to televised leader displays. Social Science Information, 43, 59–94. Buisine S, Abrilian S, Niewiadomski R, Martin J-C, Devillers L, and Pelachaud C (2006). Perception of blended emotions: from video corpus to expressive agent. In J Gratch, M Young, R Aylett, D Ballin, P Olivier, and T Rist, eds. Proceedings of the 6th International Conference on Intelligent Virtual Agents, Marina del Rey, August 2006, pp. 93–106. Berlin: Springer. Bull PE (1986). The use of hand gesture in political speechers: Some case studies. Journal of Language and Social Psychology, 5, 102–18. Calbris G (1990). The Semiotics of French Gestures. Bloomington, Indiana: Indiana University Press. Calbris G (2003). L’Expression Gestuelle de la Pensée d’un Homme Politique. Paris: Éditions du CNRS. Camurri A, Castellano G, Ricchetti M, and Volpe G (2006). Subject interfaces: measuring bodily activation during an emotional experience of music. In S Gibet, N Courty and JF Kamp, eds. Gesture in Human-Computer Interaction and Simulation: 6th International Gesture Workshop, GW 2005, pp. 268–79. Berlin: Springer. Camurri A and Volpe G, eds (2004). Gesture-based communication in human-computer interaction. Proceedings of the 5th International Gesture Workshop, GW 2003, Genova, Italy, April 15–17, 2003. Berlin: Springer. Cassell J, Sullivan J, Prevost S, and Churchill E, eds. (2000). Embodied Conversational Agents, Cambridge, MA: MIT Press. Castelfranchi C (2000). Affective appraisal versus cognitive evaluation in social emotions and interactions. In A Paiva, ed. Affective Interactions, pp. 76–106. Berlin: Springer. Castellano G and Mancini M (2007). Analysis of emotional gestures from videos for the generation of expressive behaviour in an ECA. In M Sales Dias and R Jota, eds. Proceedings of the 7th International Workshop on Gesture in Human–Computer Interaction and Simulation, Lisbon, May 2007. Berlin: Springer. Conte R and Castelfranchi C (1995). Cognitive and Social Action. London: University College. DeCarolis B, Pelachaud C, Poggi I, and Steedman M (2004). APML, a mark-up language for believable behavior generation. In H Prendinger, ed. Life-like Characters. Tools, Affective Functions and Applications. Berlin: Springer. Devillers L, Abrilian S, and Martin JC (2005). Representing real life emotions in audiovisual data with non basic emotional patterns and context features. In J Tao, T Tan and RW Picard, eds. 1st International Conference on Affective Computing and Intelligent Interaction (ACII’2005). Beijing, China, October. Berlin: Springer. Efron D (1941). Gesture and Environment. New York: King’s Crown Press. Ekman P and Friesen W (1969). The repertoire of nonverbal behavior: categories, origins, usage and coding. Semiotica, 1, 49–98. Ekman P and Friesen W (1975). Unmasking the Face. A guide to recognizing emotions from facial clues. Englewood Cliffs, NJ: Prentice-Hall, Inc. Ekman P and Friesen W (1978). Facial Action Coding System. Palo Alto, CA: Consulting Psychologists Press, Inc. Falcone R and Castelfranchi C (2003). A belief-based model of trust. In M-L Huotary and M Iivonen, eds. Trust in Knowledge Management and Systems in Organizations. Hershey/London/ Melbourne: Idea Group Publishing. Frey S (2000). Die Macht des Bildes. Der Einfluss der nonverbalen Kommunikation auf Kultur und Politik. Bern: Huber.

421

17-Wachsmuth-Chap17

422

6/3/08

10:16 AM

Page 422

POGGI AND PELACHAUD

Gallaher PE (1992). Individual differences in nonverbal behavior: dimensions of style. Journal of Personality and Social Psychology, 63, 133–45. Gratch J and Marsella S (2001). Tears and fears: modeling emotions and emotional behaviors in synthetic agents. In JP Muller, E André, S Sen and C Frasson, eds. Proceedings of the Fifth International Conference on Autonomous Agents, Montreal, May 2001, pp. 278–285. New York: ACM Press. HamNoSys (2004). http://www.sign-lang.unihamburg.de/Projekte/HamNoSys/ HamNoSysErklaerungen/englisch/Contents.html Hartmann B, Mancini M, and Pelachaud C (2006). Implementing expressive gesture synthesis for embodied conversational agents. In S Gibet, N Courty and J-F Kamp, eds. Proceedings of the 7th International Gesture Workshop, Lecture Notes in Computer Science 3881, pp. 188–199. Berlin: Springer. Johnson W, Rickel J, and Lester JC (2000). Animated pedagogical agents: face-to-face interaction in interactive learning environments. International Journal of Artificial Intelligence in Education, 11, 47–78. Kendon A (1972). Some relationships between body motion and speech. An analysis of an example. In A Siegman and B Pope, eds. Studies in Dyadic Communication. New York: Pergamon Press. Kendon A (1988). Sign Languages of Aboriginal Australia. Cultural, semiotic, and communicative perspectives. Cambridge: Cambridge University Press. Kendon A (1992). Abstraction in gesture. Semiotica, 90, 225–50. Kendon A (1995). Gestures as illocutionary and discourse structure markers in southern italian conversation. Journal of Pragmatics, 23, 247–79. Kendon A (2004). Gesture. Visible action as utterance. Cambridge: Cambridge University Press. Kipp M (2004). Gesture Generation by Imitation. From Human Behavior to Computer Character Animation. Boca Raton, FL: Dissertation.com. Kita S, van Gijn I, and van der Hulst H (1998). Movement phases in signs and co-speech gestures, and their transcription by human coders. In I Wachsmuth and M Fröhlich, eds. Gesture and Sign Language in Human-Computer Interaction, pp.23–35. Berlin: Springer. Klima ES and Bellugi U (1979). The Signs of Language. Cambridge, MA: Harvard University Press. Kopp S, Tepper P, and Cassell J (2004). Towards integrated microplanning of language and iconic gesture for multimodal output. In R Sharma, T Darrell, M Harper, G Lazzari and M Turk, eds. Proceedings of the International Conference on Multimodal Interfaces (ICMI’04), pp. 97–104. New York: ACM Press. Mancini M and Pelachaud C (2007). Implementing distinctive behavior for conversational agents. In M Sales Dias and R Jota, eds. Proceedings of the 7th International Workshop on Gesture in Human–Computer Interaction and simulation, Lisbon, May 2007. Berlin: Springer. Martin JC, Niewiadomski R, Devillers L, Buisine S, and Pelachaud C (2006). Multimodal complex emotions: Gesture expressivity and blended facial expressions. International Journal of Humanoid Robotics [Special issue on Achieving Human-Like Qualities in Interactive Virtual and Physical Humanoids], 20, 477–98. Martin JC, Pelachaud C, Abrilian S, Devillers L, Lamolle M, and Mancini M (2005). Levels of representation in the annotation of emotion for the specification of expressivity in ECAs. In T Panayiotopoulos, J Gratch, R Aylett, D Ballin, P Olivier and T Rist, eds. Proceedings of the 5th International Conference on Intelligent Virtual Agents, September 2005, Greece, pp 405–417. Berlin: Springer. McGowan J and Gormy J (1976). Validation of personality traits: A multicriteria approach. Journal of Personality and Social Psychology, 34, 791–5. McNeill D (1992). Hand and Mind. Chicago: University of Chicago Press. McNeill D, ed. (2000). Language and Gesture. Cambridge, MA: Cambridge University Press.

17-Wachsmuth-Chap17

6/3/08

10:16 AM

Page 423

PERSUASION AND THE EXPRESSIVITY OF GESTURES

McNeill D (2005). Gesture and Thought. Chicago: University of Chicago Press. Miceli M and Castelfranchi C (1989). A cognitive approach to values. Journal for the Theory of Social Behaviour, 19, 169–93. Morris D (1977). Manwatching. London: Jonathan Cape. Müller C (2004). Forms and uses of the palm up open hand: A case of a gesture family? In C Mueller and R Posner, eds. The Semantics and Pragmatics of everyday Gestures. Berlin: Weidler. Müller C and Posner R, eds (2004). The Semantics and Pragmatics of Everyday Gestures. Berlin: Weidler. Niewiadomski R and Pelachaud C (2007). Model of facial expressions management for an embodied conversational agent. In A Paiva, R Prada and R Picard, eds. 2nd International Conference on Affective Computing and Intelligent Interaction ACII. Lisbon, September. Berlin: Springer. Payratò L (1993). A pragmatic view on autonomous gestures: A first repertoire of Catalan emblems. Journal of Pragmatics, 20, 193–216. Payratò L (2003). What does ‘the same gesture’ mean? Emblematic gestures from some cognitivelinguistic theories. In M Rector, I Poggi, and N Trigo, eds. Gestures. Meaning and use, pp.73–81. Porto: Universidade Fernando Pessoa. Pelachaud C and Poggi I, eds (2001). Multimodal communication and context in embodied agents. Proceedings of the AA∗01 Workshop W7 at the 5th International Conference on Autonomous Agents, Montreal, Canada, May 29, 2001. Aachen: CEUR Workshop Proceedings.Pelachaud C, Carofiglio V, De Carolis B, de Rosis F, and Poggi I (2002). Embodied contextual agent in information delivering application. In C Castelfranchi and W Lewis Johnson, eds. Proceedings of the First International Joint Conference on Autonomous Agents and Multi-Agent Systems, Bologna, July, pp. 758–765. New York, ACM Press. Piattelli Palmarini M (1995). L’Arte di Persuadere. Milano: Mondadori. Poggi I (1983). La mano a borsa: analisi semantica di un gesto emblematico olofrastico. In G Attili and PE Ricci Bitti, eds. Comunicare senza parole. La comunicazione non-verbale nel bambino e nell’interazione sociale tra adulti. Roma: Bulzoni. Poggi I (2001). Towards the lexicon and alphabet of gesture, gaze, and touch. In P Bouissac, ed. Multimodality of Human Communication. Theories, problems and applications. Virtual Symposium. http://www.semioticon.com/virtuals/index.html, 2001–2002. Poggi I (2003). Mind markers. In M Rector, I Poggi, and N Trigo, eds. Gestures. Meaning and Use, pp. 119–32. Porto, Universidade Fernando Pessoa. Poggi I (2004). Emotions from mind to mind. In A Paiva, R Aylett, and S Marsella, eds. Proceedings of the Workshop W13: Empathic Agents. Third International Conference on Autonomous Agent Systems, AAMAS 2004, New York, July 19–23, 2004. Poggi I (2005). The goals of persuasion. Pragmatics and Cognition, 13, 298–335. Poggi I (2006). Le Parole del Corpo. Roma: Carocci. Poggi I (2007). Mind, Hands, Face and Body. A goal and belief view of multimodal communication. Berlin: Weidler. Poggi I and Magno Caldognetto E (1997). Mani che Parlano. Gesti e Psicologia Della Comunicazione. Padova: Unipress. Poggi I, Pelachaud C, and Magno Caldognetto E (2004). Gestural mind markers in ECAs. In A Camurri and G Volpe, eds. Gesture-Based Communication in Human-Computer Interaction. Proceedings of the 5th International Gesture Workshop, GW 2003, Genova, Italy, April 15–17, 2003, pp. 338–49. Berlin: Springer. Prillwitz S, Leven R, Zienert H, Hanke T, Henning J, et al. (1989). Hamburg Notation System for Sign Language. An Introductory Guide. International Studies on Sign Language and Communication of the Deaf, Vol.5, German Institute of German Sign Language and Communication of the Deaf, University of Hamburg.

423

17-Wachsmuth-Chap17

424

6/3/08

10:16 AM

Page 424

POGGI AND PELACHAUD

Quintilianus M F (100). Institutiones Oratoriae. Translation by HE Butler, The Institutio Oratoria of Quintilian (The Loeb Classical Library). New York: Putnam and Sons. Radutzky E (1992). Dizionario Bilingue Elementare della Lingua Italiana dei Segni. Roma: Kappa. Rector M, Poggi I, and Trigo N, eds (2003). Gestures. Meaning and use. Porto: Universidade Fernando Pessoa. Rimoldi H (1951). Personal tempo. Journal of Abnormal and Social Psychology, 46, 283–303. Romagna M (1998). L’alfabeto dei Gesti. I Parametri Formazionali nei Gesti degli Udenti Italiani. Unpublished Thesis. University Roma Tre. Ruttkay Z, Pelachaud C, Poggi I, and Noot H (2007). Excercises of style for virtual humans. In L Canamero and R Aylett, eds. Animating Expressive Characters for Social Interactions. Amsterdam: John Benjamins. Santi S, Guaitella I, Cavé C, and Konopczynski G, eds (1998). Oralite’ et Gestualite’, Communication Multimodale, Interaction. Paris: L’Harmattan. Scherer K (2000). Emotion. In M Hewstone and W Stroebe, eds. Introduction to Social Psychology: A European perspective, pp.151–91. Oxford: Oxford University Press. Serenari M (2003). Examples from the Berlin Dictionary of Everyday Gestures. In M Rector, I Poggi, and N Trigo, eds. Gestures. Meaning and Use. Porto: Edicoes Universidade Fernando Pessoa. Sparhawk C (1978). Contrastive—identificational features of Persian gestures. Semiotica, 24, 49–86. Stokoe WC (1960). Sign language structure. an outline of the visual communication system of the American deaf. Studies in Linguistics. Occasional Papers, 8. Stokoe WC (1978). Sign Language Structure: An Outline of the Communicative Systems of the American Deaf. Silver Spring: Linstock Press. Streeck J (2007). Gesture in Political Communication. A Case Study of the Democratic Presidential Candidates during the 2004 Primary Campaign. To be published in Research on Language and Social Interaction. Volterra V (1987). LIS. La Lingua Italiana dei Segni. Bologna, Il Mulino. Wachsmuth I and Sowa T, eds (2002). Gesture and Sign Language in Human-Computer Interaction. Berlin: Springer. Wallbott HG (1998). Bodily expression of emotion. European Journal of Social Psychology, 28, 879–96. Wallbott HG and Scherer K (1986). Cues and channels in emotion recognition. Journal of Personality and Social Psychology, 51, 690–699.