Mandel's seminal paper on “Iconic devices in American Sign Language” (1977) ... underpinnings, Müller's insight that the hands can become tools that act, mold, ...
The multidimensionality of pointing Julius Hassemer1,2 & Leland McCleary1 1
Universidade de São Paulo, Faculdade de Filosofia, Letras e Ciências Humanas 2
Humboldt-Universität zu Berlin, Berlin School of Mind and Brain
Abstract The analysis of pointing gestures presented here assumes that the meaning of a gesture is enabled by a series of cognitive operations leading from the physical form of an acting articulator, through a limited number of abstract topological conceptualisations, to its referent. Seven proposed operations can be combined in unique permutations of two to six elements, each permutation of which defines a gesture type. Each type may be iconic and/or deictic thus affording the functions of representing point-like, linear, surface-like or volumetric forms (iconicity) as well as location or direction (deixis). Illustrated from available corpora, 27 types of pointing gesture types fall under two categories: one in which the articulator is profiled as a vector, the feature most commonly associated with pointing; and a second, surprisingly more diverse category, in which it is profiled as a surface. Keywords: gesture form, physical form, pointing, multidimensionality, classification.
Graphical summary (typology) Figure 1: A typology tree of manual pointing gestures. See Figure 2 for illustration and comprehensive legend including the seven spatial operations that are the modular building blocks 1 of this typology. For naturalistic gestures coded in these categories, see open data of the Object
Description Study.
2
Figure 2: Same gesture typology in order of mention in the text, including schematic illustrations.
1
For equal access and better reproducibility, we follow an open-data policy. The majority of the data sources for this paper were made freely available by the first author and other researchers. The repositories referred to as “open” can be accessed through links in the footnotes. They include videos, 3D motion-capture data, transcriptions and methods. 2 This is a motion-capture study conducted by the first author in the Natural Media Lab Aachen, in which 17 participants describe physical objects that they had held in their hands (Hassemer, 2016). Videos, 3D data, and resources for coding gesture form analysis are available. These resources include the coding system itself (ELAN template file and coding manuals in English and Portuguese) as well as annotation files of codings by ten expert and novice raters. Link to all data in a CLARIN Repository: http://hdl.handle.net/11022/1009-0000-0007-C34C-8.
1
Introduction When someone indicates a direction or draws a line in the air with a straight index finger, a human observer will perceive not only a moving hand, but something else that is not physically present but which is at least as important as the action of the hand itself: the ray that is understood to emanate from the finger (in the case of direction giving) or the line that is traced in the air (in the case of drawing). However, these essential imaginary forms have too often been mentioned only peripherally in discussions of gesture form. The aim of this paper is to advocate for a more precise and complete definition of gesture form, first by distinguishing it from the physical forms of articulators and then by proposing the inclusion of a set of cognitive operations which appear to mediate between the action of articulators and the meanings attributed to the resulting gestures. Our claims for gesture form will be demonstrated through a systematic analysis of pointing gestures. In short, the interpretation of gesture form is understood to involve the extraction of specifically topological information from gesturing articulators and their environment in the form of abstract, imaginary forms of zero to three dimensions; that is, points, lines, surfaces and solids. Our analyses of form in pointing gestures provide further evidence for the reported “bewildering variety of morphologies actually used for pointing in the wild” (Cooperrider, 2011: 165) and resonate with literature opposing “the oft-assumed simplicity of ‘pointing gestures’” (Haviland, 2003: 139) perhaps rooted in focussing too much on index-finger pointing (Wilkins, 2003). Beyond confirming the claim of the diversity of pointing gestures, we use gesture form analysis (this paper and Hassemer, 2016) to map out their underlying structural diversity, resulting in a typology of pointing illustrated in Figures 1 and 2. Gesture form analysis finds resonance and overlap with Talmy’s (2018) monograph The targeting system of language, specifically the chapter “Gestures as cues to a target”. Talmy’s approach is wider in that it considers gesture as one of several ways to target in language use, while this paper has a more narrow focus on gesture and its consistent structures. Yet, they both regard gesture interpretation as a succession of conceptual tasks that build on one other, in what Talmy calls a “fictive chain”. One 2
“proposed characteristic is that the fictive chain is fully connective without gaps. That is, the hearer imagines fictive constructs in an unbroken sequence from the gesture to the target.” There are also basic structural overlaps with Talmy’ less recent work on cognitive semantics (2000), in which he distinguishes spatial schemas evoked by language, for example by prepositions, as zero, one, two, or three dimensional. We also adhere to Talmy’s topological understanding of dimensions that includes curved and connected forms. That means that both a straight line and a curved line, including a curved line coming full circle, is considered one-dimensional (see gesture types 6 to 8 in Figure 2). A second approach dealing with form dimensions in communicative manual action is Mandel’s seminal paper on “Iconic devices in American Sign Language” (1977), which proposes a system in which imaginary forms are created under specific movement constraints. A pointed implement (pencil or fingertip) leaves a line as its trace, either straight or curved depending on the motion. A linear implement (piece of chalk held flat, or a straight or bent finger) leaves a surface, whose shape depends on the shape of the implement (straight or curved) and on the motion. A linear implement can also move parallel to itself, e.g., a straight finger moving like an arrow to depict a straight line. A surface implement (trowel or ice-cream scoop, a whole hand held in various flat or curved surface shapes) can leave a trace shaped like itself, when moved parallel to itself—flat hand moved flat, curved hand moved as if along a curved surface—or it can leave a solid trace perpendicular to itself, as if one indicated the height of a stack of papers by putting his hand palm down on the table and then raising it to the height of the stack. Mandel (1977: 67)
These gestures and the relation amongst them will be encompassed by our multidimensional typology. Our system retypologises Mandel’s “iconic devices” according to the cognitive operations, or topological conceptualisations, required to lead from a dynamic articulator configuration to a referent, or, in Mandel’s terms, to the gesture’s “meaning”. Similarly, we build on, by revealing aspects of its cognitive underpinnings, Müller’s insight that the hands can become tools that act, mold, draw or represent (Müller, 1998; 2014) and Mittelberg and Waugh’s demonstrations of how imaginary objects can be presented in an external but adjacent spatial relation to the articulator (Mittelberg & Waugh, 2009). Finally, with regard to pointing handshapes, the 3
typology elaborates on Haviland’s (2003) and Fricke’s (2007) distinction that a single, linear, finger is preferred for pointing at an entity, while a flat, surface-shaped hand is preferred for pointing in a direction. The following general claims about gesture form analysis are adopted from Hassemer (2016; see also Hassemer et al., 2011), upon which this paper elaborates theoretically and empirically. Our first analysis in the section “Pointing At Object” illustrates the described sequence in detail. 1. The constituent imaginary forms that are part of the conceptualisation of every gesture can be classified as being of zero to three dimensions. 2. Each imaginary form is the output of spatial, or more precisely geometrical, operations. Exactly seven different spatial operations (legend of Figure 2) allow for analysing all gestures. The form dimensions that each operation outputs are predictable (e.g., two parallel one-dimensional lines limit exactly twodimensional space – the operation of limiting adds one dimension). 3. All forms and spatial operations show the sequence of derivations that lead from the gesturing body to the imaginary object or to another interpretable spatial entity. 4. Every gesture type is defined by the permutation of spatial operations that are necessary for an observer to understand the gesture. A type thus only specifies the gesture form structure, including the dimensionality of all constituent forms (excluding specifics of gesture form such as location, length or curvature). The bulk of this paper is divided into two sections. The following section analyses gestures in which single linear articulators are the basis of the gesture form, for example a straight index finger. This section includes many of the stereotypical ways of pointing, beginning with the classical pointing at a distant object with the index finger. The subsequent section, in contrast, analyses a broad palette of gestures in which several fingers, often all the fingers on one or both hands, are used to point. Among these gestures is a pointing type that is most typically used for the indication of direction (at least in many cultures): a flat sideways-oriented hand with the fingers pointing forward, which often includes motion on the sagittal plane. While indicating direction is the most common use of this pointing type, it is also used for pointing at large volumes of space using various hand shapes and movement patterns.
4
Pointing with a 1D profile We begin our analyses with the typical deictic gestures of pointing with a 1D profile. In this branch, we have Pointing At Object, Pointing At Location, and Directing (indicating a particular direction by pointing).
Pointing At Object Figure 3 shows a speaker of Ukaan, a language spoken in Nigeria’s Akoko mountains, from open data by Salffner (2014) 3. The speaker is engaged in the task of explaining to the participant on the other side of the curtain in their second language English the block construction he is building. In detailing that the white block stands on the red block, he points at the red block while uttering “for top of that eeh red” (meaning “on top of that red block”). The handshape shown in Figure 3b is maintained while the hand is briefly moved down twice. This is an example of what may be the most stereotypical type of pointing gesture, Pointing At Object (No Contact)GT1. This gesture affords singling out an object not in contact with the gesturer. The type number will appear as a subscript in the name only when gesture types are introduced or discriminated, to facilitate cross reference in Figures 1 and 2. In order to infer the referent of this gesture on the basis of the configuration and the movement of the gesturer’s body, five spatial operations are required (Figure 3b, upper row of the flowchart). The first input form is always the gesturer’s whole body. The first operation – Articulator Profiling – consists of profiling the parts of the body that are relevant for this gesture, here the index finger of the right hand, more precisely the distal phalanx. This operation selects a specific three-dimensional portion of the threedimensional body. The second operation is Shape Profiling, which extracts from the profiled articulator the predominant geometrical form, in this case, the axis of the (linear) index finger, a slightly curved line segment (1D, illustrated in Figure 3b in green). These first two operations are obligatory and common to all gesture types. The third operation (in this case) is Extension, which prolongs the line segment away from the body, maintaining its one-dimensionality (illustrated in orange). The fourth operation
3
ISO639-3:kcf; bundle: ikaan589-84.045, time: 02min 41s; speaker Adesonmi Obaude; creator/research assistant: Funmi Yun-Hsin Chang. Data archived as part of the collection Salffner, 2014. Farming, food and yam: language and cultural practices among Ikaan speakers: an archive of language and cultural material from the Akaan people of Ikakumo (Ondo State and Edo State, Nigeria). London: SOAS, Endangered Languages Archive. URL: http://elar.soas.ac.uk/deposit/0259. Accessed on April 1st, 2015.
5
(also in this case) is the Intersection of the extended line (1D) with a surface (2D) at a point (0D, illustrated in blue). The fifth and last operation for this gesture type is Proximity, which selects an entity close to this point (usually adjacent), here the threedimensional red block. Each of these five operations is necessary for an observer of this gesture to identify the red block as the referent or, in general, for any observer to understand the gesture type Pointing At Object (No Contact).
a)
b) Figure 3: Pointing At Object (No Contact)GT1 directed at a red block lying down. (a) Original screenshot. (b) Zoomed pictures and flowchart. The right-hand picture and the
flowchart constitute a “graphical gesture form analysis” and their colour-coding is consistent with the diagrams and illustrations in the typology (Figure 1 and Figure 2). Source: Salffner, 2014.
This analysis is consistent with what is found in the gesture studies literature. For Kendon and Versante (2003: 112), “Gestures understood as pointing gestures are regarded as indicating an object or a location that is discovered by projecting a straight line from the furthest point of the body part that has been extended outward into the environment.” Kita opens his edited volume on pointing by stating that the prototypical pointing gesture “projects a vector from a body part” (2003: 1). In fact, the conceptualisation of pointing as involving a one-dimensional form emitted by the articulator is found in research on pointing from a range of disciplines such as linguistics (e.g., Bühler, 2000[1934] (“Pfeil” = “arrow”); Fricke, 2007; Streeck, 2009); sign linguistics (e.g., Liddell, 2003); psychology (e.g., Butterworth, 2003; Kita & Lausberg, 2008); ethnography (e.g., Haviland, 2003; Wilkins, 2003; Enfield, 2009); and computer science (e.g., Kranstedt, et al., 2006; Pfeiffer, 2010). Our analysis differs from these other accounts by explicitly situating the presence of the “vector” or “projecting straight line” within a conceptualisation process common to all 6
manual gesture interpretation, specifically as the result of one of several cognitive operations on the physical action of the articulators. As one of several possible constituent forms, each describable crucially in terms of its dimensionality, the presence or absence of such a linear extension and how it is ultimately interpreted as “pointing” at an object (or at a location, or in a direction) makes it possible to discern fine-grained relationships among the form structures of closely related gestures, and thus arrive at a structurally-based typology of gesture types. The discriminatory use of breaking gesture form into its constituent, predominantly imaginary, forms is exemplified by comparing the gesture type just described, Pointing At Object (No Contact), with its sister gesture type, Pointing At Object (Contact) GT2. An example of the latter may be someone touching a book with her index finger and saying: “I haven’t read this one”. If the (No Contact) version is described as above and illustrated as in Figure 3b, it becomes clear that the (Contact) version of Pointing at Object would differ only by the elimination of the operation of extension. In the (Contact) version, there would still be a linear segment, given by the operation of shape profiling of the articulator (green), and there would still be a point of intersection (blue), but the extended (orange) segment would be absent. In making this structural comparison between two gesture types, we can see that although one contains an imaginary extended straight line and the other does not, we are dealing with two gestures that are otherwise structurally identical: they both profile an index finger as articulator, the finger is profiled formally as a line segment (1D), the line intersects with a surface (2D) at a point (0D) and there is a proximate object (3D) which can be understood as the referent. Thus rather than characterising pointing as basically one dimensional, we are able to show how its structure is intrinsically complex and multidimensional.
Pointing At Location An example of a similar structure is Pointing At Location. Figure 4 from Andrén (2010: 186) shows a boy performing another deictic gesture indicating with a straight index finger the spot where the next puzzle piece should go, while saying “There!” This gesture can be interpreted as Pointing At Location (Contact)GT4 and is characteristic of young children’s preference for touching reachable objects when they are pointing at them (Andrén, 2010: 184). This gesture type overlaps with the previously mentioned Pointing At Object gesture types in that a one-dimensional line intersects with a two7
dimensional surface in a zero-dimensional point. The crucial difference between them is that in Pointing At Location, we have no final operation of proximity, which picks out a referent object in the vicinity of the point of contact. Instead, the location of the point of intersection (0D) is already what is referred to by this gesture; it is itself the referent form.
Figure 4: Pointing at Location (Contact)GT4 at where the puzzle piece should go. Source: Andrén, 2010.
Inserting the operation of extension results in Pointing At Location (No Contact)GT3 which points at distant locations. We have thus shown, by applying structural mulitdimensional gesture form analysis, the precise similarities and differences of four closely related gesture types: Pointing At Location (Contact)GT4 and (No Contact)GT3 and Pointing at Object (Contact)GT2 and (No Contact)GT1, as illustrated in Figure 1. In an alternative interpretation of the gesture in Figure 4, it could be understood not to refer only to the point of intersection, but to the particular area the puzzle piece should occupy. In this interpretation, a higher dimensional shape, a two-dimensional flat area would be the referent form. This would require the additional operation of proximity, resulting in the type Pointing At Object (Contact), which was introduced at the end of the last section as having a referent form (the object) of up to three dimensions. Thus it is that the referent form of each type has a particular dimensionality, as noted below each gesture type box in Figures 1 and 2. Both gesture types – Pointing At Location and Pointing At Object – are equally precise in one of their constituent forms, the point of intersection, but by adding proximity, we can gain a referent form of up to three dimensions, of any shape, size or materiality, depending on the context. The example shows that gesture form analysis is amenable to ambiguity in gesture interpretation and 8
allows specifying among which gesture types and referent forms a specific gesture token is ambiguous.
Directing Another pointing type with a linear shape profile is DirectingGT5, illustrated in Figure 5, which comes from a motion-capture study conducted by the first author at the Natural Media Lab, Aachen, Germany, containing a variety of spatial tasks, hence Spatial Tasks Study (Hassemer, 2016). In response to the question “Where is north?”, the participant raises her right hand with the index-finger pointing in a specific orientation. In order to conceptualise this gesture as indicating north, the index finger is first profiled as the relevant articulator by articulator profiling; the finger is then shape-profiled as a one-dimensional line segment (green); then by extension this line segment produces the referent form of a ray leading away from the body in a particular direction (orange), which, on its own, is a sufficient response to the given task. Without all three of these conceptualisations – picking out the articulator, conceptualising the articulator in its predominant dimensional form, and then extending that imaginary form – the gesture could not be understood to mean what it does.
It is worth pointing out that the referent forms so far analysed have been both real (in the case of the red block) or imaginary (in the case of an imaginary extended line showing direction).
9
None of the gestures discussed so far have required a consideration of motion to be interpreted correctly. Yet, co-axial or parallel movement in the pointing direction can be used redundantly in these gestures to emphasise the direction (Kendon, 2004: 200). In order to reduce unnecessary complexity, only operations that have an impact on any of the ensuing forms including the referent form are annotated. Thus, motion is annotated of as part of the analysis only in those cases in which it actually influences the referent form. One such example is analysed next.
Dynamic Directing Various examples in this paper are taken from the open data corpus Gesture and Speech Alignment (SaGA) by Lücking and colleagues (2010), 4 which contains route descriptions of participants that went on a bus tour through the 3D environment of a virtual city. Figure 6 shows one participant of this study performing a two-handed gesture, of which the right hand is analysed here. In describing part of the route that passes by a lake in a park, the participant makes a movement that begins in front of his chest and then performs a large circular movement to depict a path around the |lake| 5 while saying “and you go around it” (“und du gehst außen rum”). During the first part of the movement, the index finger is straight or overstretched (Figure 6a) as the hand moves forward and to the right. In Figure 6c, the finger has been redirected upward in the direction of movement and by Figure 6d it is curved ventrally as the hand begins the curve to the left. These changes in the configuration of the articulator – always pointing roughly in the direction of movement – indicate that there are changing directions and degrees of curvature in the depicted path: thus the name Dynamic DirectingGT6. In this gesture, the referent form (the path) is not produced by extending the shape profile (curved line segment, indicated in Figure 6e in green) beyond the tip of the articulator. In Dynamic Directing, the path is produced by the operation of Trace Leaving, in which the colinear movement of the articulator itself is understood to leave a trace (indicated in Figure 6e in pink). In the sense that trace leaving has a path as its output form, it is equivalent to Mandel’s ‘atemporal motion’ (1977: 66-67) and is the gestural analogue of Talmy’s ‘fictive motion’ (2000: 99).
4
http://hdl.handle.net/11022/1009-0000-0000-DEC1-C Following Liddell (2003: 148), the notation of vertical bars enclosing an expression x means that |x| refers to a surrogate or imaginary x represented in real space. 5
10
a)
c)
b)
d)
e) Figure 6: Dynamic DirectingGT6 describing a path around a |lake|. Image e) is an overlaid composite of a) through d). In image e), the right hand corresponding to image c) is partially hidden by the arm in image d). Source: Lückinget al., 2010.
In contrast to the above gesture type, the following four gesture types will include the articulator moving orthogonally (or at least non-colinearly) in relation to the shape profile of the articulator, thus adding one dimension to the output form.
11
Drawing The pointing gestures with a 1D profile described up to here have had a strong deictic function in that they serve to individualise an object, refer to a location or indicate a direction. DrawingGT7-8, the gesture type described next, may be more readily understood to serve an iconic function than a deictic one. Nevertheless, it is included here because it can be used with a clear deictic function and also because of its major structural overlap with the pointing gestures mentioned up to here. Note that while ‘deictic’ and ‘iconic' are often used as basic gesture categories, there are at least two general objections to doing so which should become clearer throughout this paper. First, some of the ‘deictic’ gesture types that are discussed here can also have clear iconic uses. A ‘deictic’ gesture such as Dynamic Directing, for example, can simultaneously serve an iconic function as, for example, in the example above, depicting the shape of a static trail. Second, indexicality in gesture, as in semiosis in general, is inseparable from and co-dependent on iconicity (for a discussion, see Deacon, 1997: 47-101). In our data, for example, the gestural production of a pointing ray is only possible due to the conceived similarity of the profiled articulator to a line. Drawing prototypically focuses on the shape of the drawn line, though it is also a feature of these gestures that they are always produced at some position in space and therefore always contain information concerning location and orientation, whether or not that information is meaningful in the context of a particular gesture performance. In most typologies of gesture […], iconic gestures and deictic (pointing) gestures are treated as separate kinds of gesture. This does not seem to be correct. Pointing gestures can trace the shape of what is being pointed at, and thus superimpose an iconic display on a deictic point within the performance of a single gesture. Goodwin (2003: 229, emphasis in the original)
The modularity of gesture form analysis makes it possible to specify in which aspects of spatial structure two gesture types share features and in which they differ. Thus Figure 1 exhibits how closely related DrawingGT7-8 is to Pointing At LocationGT3-4. They share a 1D shape profile of the articulator and a 0D point of intersection. While Pointing At Location stops at the point of intersection, Drawing will add one additional operation (trace leaving). Figure 7 shows a participant of the Object Description Study performing a gesture of the type Drawing (Contact)GT8. The participant emphasises the curved shape 12
represented by the left hand (not part of this analysis) by drawing part of a circle on the middle finger and thumb of the left hand with the index finger of the right hand. In consonance with Goodwin’s assessment, at the same time the index finger is pointing at its target at the 0D point of intersection with the fingers of the left hand, it is also producing, by the operation of trace leaving, a 1D trace as it is moved along the surface. The result is a referent form of a one-dimensional curved line. Or, in Kita’s formulation: “a pointing gesture can create an iconic representation by tracing a shape or movement trajectory” (2003: 2).
a)
b)
c)
Figure 7: Drawing (Contact)GT8 along the |torus|. (a) Original. (b) Gesture form analysis. (c) Stimulus object. Source: Object Description Study.
Just as Pointing At has a (Contact) and (No Contact) version, so Drawing can also be performed both in contact with a real or imaginary surface or at a distance, in which case the operation of extension is required. An example of Drawing (No Contact)GT7 is shown in Figure 8, taken from the Spatial Tasks Study. In this example, the particpant expresses uncertainty about where exactly the train station is from his current position. He begins by attempting to establish a southward direction, since this is a direction that he feels more certain about and he knows the train station is roughly to the south. He stammers “In any case so hm yes so now southwards since south is there.” (“auf jeden Fall also hm ja also südlich jetzt dort ist ja Süden”) while gazing toward the floor and extending his left hand and straight index finger. In producing his gesture, he moves his hand upward, in a controlled motion, without arriving at a clear end point. The function of this gesture is very similar to the ‘north’ gesture analysed above (Figure 5); but the fact that there is a continuous, controlled movement indicates that the raising of the hand is meaningful. The structure of this example of Drawing (No Contact) 13
“southwards” includes the shape profile of the index finger (green) plus its linear extension (orange) from the tip of the finger until it hits the surface of the ground. As the arm is rotated upwards, the point of intersection with the floor projects progressively farther away from the gesturer, such that a long line (potentially extending to the horizon) is drawn on the ground and directed away from the participant in a southerly direction (pink).
a)
b)
c)
14
d) Figure 8: Drawing (No Contact)GT7 southern direction. (a-c) Movement stills. (d) Gesture form analysis. Source: Spatial Tasks Study.
The similarity of context of this gesture to the ‘north’ gesture (Figure 5) contrasts with their clear differences in performance and conceptualisation. This one (Figure 8) uses motion to create gesture form; the other does not. This one exploits the spatial effect of intersection and trace leaving; the other does not. The comparison of the two types illustrates that different spatial strategies may be used to communicate very similar referents, for example, a cardinal direction. Another strategy for the same function, using a flat hand (2D shape profile), will be seen in the section on intersecting surface pointing.
15
Silhouetting Just as Drawing manifests clear similarities and differences in relation to Pointing At, so does SilhouettingGT9-10. As shown in Figure 1, Silhouetting is Drawing with one additional operation: Limiting. The screenshot in Figure 9a also stems from the Object Description Study. The participant is describing the small spherical object shown in Figure 9b and says “The object was about this big and [ ] round“ (“Der Gegenstand war so groß und [ ] rund”). The line drawn in Figure 9a (shown in pink) specifically focusses on a curved path (1D), produced by trace leaving, this time the result of the movement of a point of intersection of the fingertip in contact with an imagined |object| (note the participant’s gaze directed at his fingertip). In this case, however, unlike with Drawing, the referent form is not the traced line itself, but rather the outlined two-dimensional object, or in this case a two-dimensional projection (black stripes in Figure 9a) of a three-dimensional object (the stimulus in Figure 9b). Thus Silhouetting (Contact)GT10 adds to Drawing (Contact) the operation of limiting, in which the drawn line is understood to delimit an enclosed area. In this gesture type, both the line (1D) and the fill (2D) are relevant. And just as Drawing may have a (Contact) and a (No Contact) version, so may Silhouetting.
a)
b)
Figure 9: Silhouetting (Contact)GT10 the circumference of a small ball. (a) Gesture form analysis. (b) Stimulus object. Source: Object Description Study.
16
Preliminary conclusions Lines or vectors and their ability to indicate a direction are prominent in all of the pointing gestures analysed thus far. However, they occur within different stages of conceptualisation and in conjunction with other conceptually relevant forms, for example the point. The point occurs at an important stage of gesture conceptualisation, precisely at the intersection between the direction-indicating line (either the shapeprofiled articulator or a line extending from it) and the surface of an entity (real or imagined) pointed at. The point is a form which, due to its minimal (in fact, nonexistent) shape, affords communicating only a narrow range of functions (Hassemer, 2016). The most prominent of these is to mark a specific location in space, precisely because of its lack of extent. The analysis of the communicative functions of forms of different dimensions can be seen as an application of Gibson’s theory of affordances (1977). Gibson considers the affordances of objects and surfaces, but we will also argue for and empirically show the affordances of other-dimensional forms, the point and the line, to account for the communicative potential of gestures. The deictic potentials of the pointing gestures investigated above can be broken down to geometric constructs that begin with the three-dimensional form of an articulator, end in spatial entities of varying dimensions, and whose core is made up of the point (0D) and diverse linear constituents (1D). Beyond the general affordances of forms of specific dimensions, the application of a perspective of affordances can shed light on the combined forms within specific gesture types. In the often considered stereotypical Pointing At Object (No Contact)GT1 with index finger (Cooperrider, 2011), the linear extent of a single finger is exploited to present a line in a specific orientation in space. This line affords a clear direction when being extended colinear to its axis. The extension of this line, the ray, then allows bridging any distance between the gesturer and the referent. The two-dimensional surface (real or imagined) with which the line intersects affords ending the otherwise endless imaginary ray while also producing, in conjunction with the one-dimensional ray, exactly that kind of form, a zero-dimensional point, which is ideal for communicating location. The extremely adaptive operation of proximity finally provides Pointing At Object (No Contact) with the ability to indicate a variety of referent shapes of any dimensionality. While this gesture type profits from the precision of a point, it can also be used to point to an object as large as a mountain or a barn or a referent of a different dimensionality, like a surface or a line. As this example illustrates, gestural 17
function can be broken down to the affordances of the constituent forms, thus permitting an explicit link between form and function.
Pointing with a 2D profile The gestures discussed in this section are performed with an open hand, not with a single outstretched finger as in the pointing types with a 1D profile. Some 2D-profile types point in a broad fashion, while others can precisely indicate a direction similar to the already analysed Directing (Figure 5) and Drawing (No Contact) (Figure 8). Still others serve to demarcate certain spaces and shapes within the gesturer’s reach. To disentangle the functions and the different constituents of pointing with a twodimensional shape profile, gesture form analyses show how the spatial information of a flat or curved hand is processed outputting broad forms (2D or 3D) as well as narrow forms (1D). In contrast to gestures with a one-dimensional shape profile, the structure within surface pointing types is less straightforward and less discussed in the literature. There are many structurally distinct types which appear to occur only in specific contexts and the differences in function, or in the portion of space they mark, is sometimes minor. Thus we will group structurally related gesture types for ease of exposition. The order of gesture types presented here will have a tendency to increase in complexity, in vagueness and in marking larger and more distant spaces. The four groups that involve 2D-profile gestures are (1) Surface-edge marking, (2) Non-intersecting surface pointing, (3) Intersecting surface pointing, and (4) Sketching with surface. In addition, we will discuss the 2D gesture type Non-Specific HoldingGT26. In Figure 2, the groups are indicated by headings in the left-most column of gesture types. Immediately to the right of this column, you see the spatial structure of each gesture type in colour-coded abbreviations of operations. This gives an overview of the core spatial operations that types of each category have in common: Putting aside the two initial profiling operations that are common to all types, Surface-edge marking (group 1) is characterised by the operation of intersection; Non-intersecting surface pointing (group 2) by extension; Intersecting surface pointing (group 3) by extension and intersection; and Sketching with surface (group 4) by trace leaving.
18
Group 1: Surface-edge marking The four gesture types within the group Surface-edge marking are performed in contact with an object. They use the flat or curved hand as their articulator, more specifically its edge, whether it is the broad lateral edge opposite the thumb and continuing along the side of the little finger, or the smaller edges formed at the tips of two or more fingers held side by side. For the purposes of this paper we will exemplify each group by analysing only two or three gestures that are characteristic of it. For the group Surfaceedge marking, we will analyse the static gesture type Surface-Edge StampingGT11, and the dynamic type Surface-Edge DrawingGT12. An example of Surface-Edge StampingGT11 is taken from a recording of the political talk show Friedmans Agenda with four discussants on the German special interest channel “lettra”. Political arguments on education policies are exchanged and a member of the political party Die Linke demands an answer to the question “Are the already-trained going to be further supported or not?” ("Werden die jetzt Ausgebildeten weiter gefördert oder nicht?"). During this and other sequences of her contributions, she moves her hands markedly downward onto the table. In this sequence, she performs this beat-like movement that may be perceived as ‘chopping’ four times, each time a little farther from her body. A screenshot of the first of these gestures is shown in Figure 10a. The palms face inward and the fingers, apparently held tightly side by side, are sharply bent at the base, such that the fingertips of each hand are close to each other.
a) 19
b) Figure 10: Surface-Edge StampingGT11 different |career steps|. (a) Screenshot while the hands are beginning to approach the table. (b) Schematic illustration of the hands at the moment when touching the table top. Source: Friedmans Agenda (lettra, 02.07.2008).
As shown in Figure 10b, one spatial operation necessary for understanding these gestures is profiling the four aligned fingers of both hands. They are roughly co-planar to each other, forming two surfaces, which are the profiled shapes, parallel to her frontal plane. Secondly, emphasised by the marked downward movement, the intersection of the edge of the hands and the surface of the table is in focus. We interpret these gestures as illustrating the concept of advancement by “stamping” lines onto the table (Mandel, 1977: 67). The lines at advancing positions metaphorically represent progressive stages in professional development. A structural relative of this gesture type is Surface-Edge DemarcatingGT13, in which, with the additional operation of limiting, two stamped lines demarcate the area between them. The first dynamic type of Surface-edge marking is Surface-Edge DrawingGT12 as exemplified in Figure 11 by a woman drawing the vertical pole of a standard lamp (floor lamp). The data was collected by the second author and contains two participants discussing the possible furnishing of an empty living area. One participant draws the stem of the lamp on the table in front of her with a straight movement towards her body with her flat right hand, palm sideways, middle and ring finger touching the table top. Simultaneously she says “Those lamps that are like this, a rod and a lamp” (“Aqueles abajures que são assim tipo, um pauzinho, e o abajur.”). In addition to a spatial structure as the one in Surface-Edge StampingGT11, the operation of trace leaving is necessary to understand the extent of the motion as the extent of a static object. The 20
difference of whether this operation is added before the intersection with the table top or after it does not impact the referent form of a straight line. Thus both permutations of operations are one and the same gesture type.
a)
b) Figure 11: Surface-Edge DrawingGT12 the |rod of a standard lamp|. (a) Original screenshot, beginning of movement. (b) Gesture form analysis. Source: McCleary (unpublished).
If this gesture was performed with two hands, one beside the other, with palms facing each other and delimiting a certain area between them, it would be Surface-Edge SilhouettingGT14, the last type in this category. This is not the case in Figure 11. Rather, the static left hand seems to depict the lower end of the rod by performing SurfaceEdge StampingGT11 (first type of the current group). The referent forms of a line or a surface produced by these gestures afford both communicating different locations in space (as in Figure 10) and communicating specific shapes (as in Figure 11), showing once more that deictic and iconic functions of gesture are not exclusive and that depending on the context they can be one or the other or, to varying degrees, both.
21
Group 2: Non-intersecting surface pointing In Non-intersecting surface pointing and most of the following groups, just as in most of the gesture types among the pointing gestures with a one-dimensional profile, the profiled shape gets extended beyond the outermost part of the articulator – with the difference that in 2D-profile pointing it is a surface and not a line that is first being profiled and then being extended. Hand shapes in this and the following groups extend in the pointing direction (often forward, away from the gesturer). In contrast to 1Dprofile pointing, they also extend horizontally or vertically. The profiled surface looks like a narrow plane (when the fingers are parallel to each other and in contact), or like a plane that spreads wide (when the flat hand is spread) – or like an empty cone widening forward when the hand is loosely open (see illustration in Figure 2). The analysis of Non-intersecting surface pointing begins with an example of Static Planar DirectingGT15 (Figure 12), in which the profiled plane is narrow, because the fingers are held together, and because only the index, middle, ring and little finger (below “fingers 2-5”) seem to be profiled, with the thumb hanging down loosely. The participant in this example says “the house is U-formed. You sort of look into the U so to speak.” (“Das Haus ist halt U-förmig. Man guckt halt in das U rein sozusagen”), while performing the gesture. He continues to use Static Planar Directing at other times while saying for example: “if this here is the observer position ...” (“wenn das hier die Betrachterposition ist...”).
22
Figure 12: Static Planar Directing into a |U-shaped building|. Source: Lücking et al. (2010).
The participant describes the field of vision (or the range of viewing directions) as a flat surface that extends in the direction of the fingers. An actual field of vision also extends vertically. However, he specifically depicts the extent of lateral vision that is relevant for recognising a U-shaped form of a building when one is looking into the “U”. In this respect, conceptually reducing vision to a planar, horizontal shape abstracts the field of vision to the two dimensions which are relevant. In Arnheim’s words, the “gesture limits itself intelligently to emphasizing what matters” (Arnheim 2004[1969]: 177).” Static 1Dprofile pointing would not have sufficed in representing "looking into the U”, as in this example. The participant in Figure 13 also holds her hands horizontally, except that this time both hands are used and the fingers (including the profiled thumb) are spread roughly horizontally. At the moment of the gesture, she is a bit at a loss in her description of the virtual tour, so she decides laughingly “before I forget about it, I will tell you already what the fountain looks like” (“bevor ich’s vergesse sag ich dir schon mal wie der Brunnen aussieht”). Nevertheless, the gesture clearly does not refer to the fountain; it has more of a discursive function of inhibiting action (including verbal action) on the part of the interlocutor. The spatial interpretation of this gesture could be that the planar profile of her hands is extended in the direction of the interlocutor (whose hand and lap
23
are partially visible at the lower right of Figure 13a, b). The extended plane reaches the interlocutor’s body above the hands that are on the lap. Black vertical stripes indicate the space beneath the extended surface, which is arrived at by the operation of limiting. The extended surface (yellow) is the upper limit of this volume, which has the effect of containment, used here metaphorically. There are different gestures of this type that point with a flat hand at another person or body part, but slightly off to the side, thus delimiting the indicated space at one side (in contrast to the example here, with the hand is usually oriented palm-up, oblique or sideways; see also Talmy, 2018).
a)
b)
Figure 13: Static Volumetric DirectingGT16 with flat hands palm down to keep the interlocutor from interrupting. Source: Lücking et al. (2010).
We do not take the gesture in Figure 13b to include motion as an aspect that influences gesture form. If it did, for example, if the gesturer moved her spread hands apart so as to address multiple listeners that sit next to each other, this would include parallel trace leaving and result in the type Dynamic Volumetric DirectingGT17, the last type in this group (see also Wilkins 2003: 194-196). Another example of Non-intersecting surface pointing is performed not with a planar, but with a conic hand shape, sometimes with the fingers straight, sometimes loosely curved. Figure 14 is taken from a word search for “all over” in the TV News Archive, which offers quickly accessible open data in form of videos and transcriptions. The possibility of finding examples of Static Volumetric DirectingGT16 in the vicinity of the words “all over,” is support for this gesture type fulfilling a function of somewhat more vague pointing or pointing at larger chunks of space. On CBS Evening News on March 24
15, 2010, 6:39pm 6, a correspondent at the scene reports “this is what we're seeing all over. Huge trees blocking the roads and repair crews working on adrenaline just trying to clear the way.” While he speaks, he performs several beat-like movements in the direction of the trees. In contrast to Figure 12, the thumb here seems to be part of the profiled articulators, resulting in an empty conic shape, again with the cone widening in the direction of pointing. By the operation of limiting, all of the three-dimensional space within the hollow cone is filled out (black hatching). With this expanding three-dimensional cone, the reporter directs the viewer's attention to the central part of the scene where fallen trees, workers, and other objects are located.
Figure 14: Static Volumetric DirectingGT16 with a conic shape at a scene with fallen trees. Source: Evening News (CBS).
In this gesture type, the order of operations is not fixed. The filling in of the twodimensional conic surface to become a three-dimensional cone can occur, as shown above, as the last step, after extending the cone; but it could also occur right after the shape profile, outputting a hand-sized volumetric cone that is then extended toward the scene. Note that the referent form is identical in both permutations of spatial operations. An alternative interpretation of Figure 14 could entail the forward movement of this gesture example to be understood not just as a beat, but rather as an expressive movement corroborating the pointing direction. The forward movement could thus be 6
.
25
interpreted as leaving a trace parallel to the sagittal extent of the cone. Since this minor movement would not contribute to specifying the pointing direction, it is not coded as an additional operation (which would make it a distinct gesture type).
Group 3: Intersecting surface pointing As shown in Figures 1 and 2, the first 2D-profile group Surface-edge marking is characterised by containing gesture types in which the intersection with a surface is required, but in which there is no extension of the profiled shape. The second group, Non-intersecting surface pointing is characterised by types that require the extension of the profiled shape but no intersection with a surface. The group at hand is characterised by types that require both the extension of the profiled surface and the intersection of the extended surface with another surface. The first gesture in the group Intersecting surface pointing is Projecting Line With SurfaceGT18, which, in one of its variants, is frequent in direction giving, having been reported on by Fricke (2007: 108-115), Wilkins (2003: 196-198), Müller (2008: 228-233) and others. This gesture indicates a direction of a straight or a curved path by a flat or curved vertical palm with the fingers forward. Despite the use of the whole hand or at least multiple fingers, this gesture is not vague or broad in its pointing. It intersects with the ground (over which the interlocutor is supposed to walk or drive) in a line (1D), thus referencing a precise, linear path. One empirical token is a participant in the SaGA corpus mentioning where to leave a roundabout by saying "a::nd you simply want straight out” (“u::nd du möchtest einfach nur grade wieder raus”). During this phrase, he quickly moves his flat vertical right hand down twice in a beat-like fashion, with the palm facing sideways. The second of these two beats is depicted in Figure 15.
26
Figure 15: Projecting Line With SurfaceGT18 for directing straight ahead. Source: Lücking et al. (2010).
The linear referent form of Projecting Line With Surface indicates the direction that is central to the utterance at hand. Since the downward movement does not influence the resulting form, and may also be coded as a superimposed beat, it is not coded as part of gesture form. Projecting Line With SurfaceGT18 and DirectingGT5 (1D), analysed in Figure 5, have distinct affordances. The extension of the pointing surface in the former will intersect with the ground regardless of changes of its contour, for example by a hill or valley (for an analogous affordance, see DrawingGT7, analysed in Figure 8). Directing does not afford this adaptation as an inherent part the gesture, but instead affords communicating a specific slope, which Projecting Line With Surface cannot. For example, in one attested token, Projecting Line With Surface is used, with a downward beat, to refer to a path going up a flight of stairs. While it sucessfully adapts to any upstairs path, it cannot indicate the incline or height of the stairs. In another example of Intersecting surface pointing by the same participant one minute later (Figure 16), motion is a non-redundant factor. While saying “Then eh::m it bends 27
at some point away from the river like this” (“dann eem biegt das irgendwann so vom Fluss weg”), the participant twice performs a gesture with a flat vertical hand, palm facing sideways. At least in the second occurrence of the gesture (the first one is less pronounced), motion does contribute new information. Throughout the short, accelerating movement, the curvature of the hand increases (Figure 16c). The necessary additional operation is trace leaving, completing the structure of the gesture type Drawing Line Projected By SurfaceGT19 (Figure 16b).
a)
b)
c)
Figure 16: Drawing Line Projected by SurfaceGT19 for path curving away from the river. (a) Cropped original screenshot. (b) Graphical gesture form analysis. (c) Temporal succession in 2-frame intervals showing increasing curvature and increasing lateral movement throughout the gesture stroke from image 1 to image 4 (time points 1 and 4 are displayed in (b). Source: Lücking et al. (2010).
A token of the most complex type in this group, Silhouetting Surface With Line Projected By SurfaceGT21, is again taken from the second author’s decoration data. In Figure 17, the participant depicts a rectangular rug while saying “So you put a rug here like that” (“aí cê coloca um tapetinho aqui assim”). The extended surfaces emitted by the flat hands intersect with the floor in lines which extend forward from the gesturer’s body, in accordance with the trace-leaving movement of raising the hands. If the same gesture were to involve static hands (no trace leaving), it would be Delimiting Surface By Line Projected By SurfaceGT20, the remaining type in the current group 3. 28
Figure 17: Silhouetting Surface With Line Projected By SurfaceGT21 to depict a rectangular rug. Source: McCleary (unpublished).
Given its rectangular shape, the rug has a clear orientation: it extends in a specific direction. The deictic information of direction or orientation is conveyed by interpreting the linear projections of the planar hands on the ground to leave a trace when the hands are moved. Once again we see a gesture not limited to either iconic depiction or deictic pointing. Depending on the context, it can have one, the other, or both functions.
Group 4: Sketching with surface All four types of the group Sketching with surface reaffirm Mandel’s systematics of a device of virtual depiction, “sketching” (1977: 67; see also Wilkins, 2003: 194). All types of Sketching with surface profile the hand as a surface and all types crucially include the operation of trace leaving. Conveniently for our purposes, all four types can be illustrated by the same gesture, whose ambiguity makes it amenable to alternative analyses (see Wilkins, 2003: 196). 29
In Figure 18, a Copenhagen city guide on a boat tour explains the reason for the typical proportions of houses: “this is also why we s- we have [unintelligible] see so many narrow but very tall buildings”. Located in front of the speaker are the seated tourists, so that the gesture clearly does not point to physically present houses, but rather creates imaginary house miniatures in or near the guide’s gesture space. The analysed gesture occurs during the first words of the utterance, sketching a |row of houses| on which ensuing gestures will further elaborate by specifying the narrowness and height of single |houses|. The left hand is held vertical with the fingers directed forward, and moves to the left, roughly orthogonally to its plane.
a)
b)
c)
d)
Figure 18: SKETCHING WITH A SURFACECC4 for a row of |houses| in four different interpretations. (a) Sketching Volume With SurfaceGT22. (b) Sketching Volume With Extended SurfaceGT23. (c)
30
Sketching Surface With Line Projected by SurfaceGT24. (d) Sketching Surface With Line Projected By Extended SurfaceGT25. Source: Copenhagen tour guide (Hassemer, 2016).
In Figures 18a, b, we see examples of a surface leaving a volume in its trace as it moves. Under this interpretation, the gestures depict the |row of houses| in their volume, in Figure 18a immediately in front of the gesturer, his hand itself sketching the volume, and in Figure 18b, at a distance, the volume being sketched by an extension of the shape profiling of the hand. Differently, Figures 18c, d depict the same |row of houses| as a (depth-less) |facade of the row of houses|, also at different distances, either at his fingertips, as in Figure 18c, or more distantly, as in Figure 18d. The vertical line (blue in the illustrations) is the result of the intersection of the plane of the hand with an imaginary facing plane in front of the gesturer (consonant with Fricke’s ‘screenlike models’, “bildschirmähnliche Modelle”, Fricke 2007: 267-8). The names of the gestures types are chosen to exhibit the core form constituents, but beginning with the referent and ending with the shape profile, as in: Sketching Surface With Line Projected By Continued SurfaceGT25 in Figure 18d. These awkwardly long names are used for gestures that do not have a familiar counterpart in an everyday action such as DrawingGT7, and have the advantage of immediately disclosing the relations within the spatial construct. While each of these possible interpretations is the result of different structural analyses, the basic spatial concept of a row of buildings is captured with all of them; the interpretations in Figures 18a-d are functionally homogeneous (Hassemer, 2016). Note that, if the gesturer understood the gesture as illustrated in Figure 18a and his interlocutors conceptualised the same gesture as shown in Figure 18d, communication would most likely not be impaired. This kind of ambiguity in spatial conceptualisation seems to be a common feature of gestural expression. A gestural depiction specifies only the spatial characteristics of current interest, not all characteristics. In other words, gestures disambiguate forms that make a difference for the communicative act, and may not specify those forms that are irrelevant to the overall utterance.
Non-Specific Holding As argued in Hassemer (2016), the large group of gestures that involve imagining the hand touching, holding, or moulding an object in a more or less specific way have their roots in surface contact being an important part of visual and tactile perception, because gestures often imitate actions involving surface contact. These gestures are 31
traditionally considered iconic. While part of our general claim is that any of these surface-contact types have the potential to be used to indicate the location of a referent, we restrict the mention of gesture types within this group to Non-Specific HoldingGT26 whose deictic function is dominant and which is used repeatedly with this function in one of the source corpora. An example of such a gesture is performed by a participant in the SaGA corpus while saying: “to your right, as you know, you see this one thing with the two towers and on the left this this dome“ ("du siehst halt rechts von dir das eine Teil mit den zwei Türmen und links diese diese Kuppel"). The participant alternately gestures with her left hand to indicate the |dome’s| position to her left (Figure 19) and similarly gestures with her right hand to her right to indicate the |thing with the two towers|. In these gestures, the |object| is highly underspecified. The |dome| is not specified in its size or shape nor in its exact position (we do not regard the fingers’ slight curvature to represent the dome’s curvature). We only know that it is located beyond the surface of the hand, which the participant holds as if touching an |object| that possesses a rather plane vertical front. One could imagine a scaled-down scene, in which the dome was as close as being in contact with the hand. More likely is the interpretation that the |dome| is not in contact with the hand, but occupying a portion of space farther away from the gesturer. Both interpretations locate the |dome| somewhere in the vaguely indicated three-dimensional volume in surface contact with the profiled surface of the hand (note the “ANY” in the illustration indicating any shape of any dimension in this space). Regardless of the position of the |dome|, or because of the lack positional specificity, this gesture is called Non-Specific Holding, showing some similarity to the act of holding off something, but serving primarily as a locating device in this case (compare Talmy, 2000: 107-108 for how planes “face” in a specific direction).
32
Figure 19: Non-Specific HoldingGT26 a |dome| to her left. Source: Lücking et al. (2010).
Representing In 0D The last gesture type of this paper, Representing In 0D, diverges substantially in performance (no specific handshape needed) and conceptualisation (only the two profiling operations are required) from all the earlier gestures and is a final example of the broad structural bandwidth of pointing gestures. The gesture is taken from the show Friedmans Agenda and is performed by the moderator when asking, with reference to new elite universities in Germany: “Will there really be more competition or less competition?” (“Ist wirklich mehr Wettbewerb dann da oder weniger Wettbewerb?”) while looking up from his notes to the participant to his right. In Figure 20a, he drops his loosely joined hands on the table towards his left and leaves them there (Figure 20b) while saying with emphasis “more competition”. He then moves them over toward the centre of his body (Figure 20c) until he says “less competition”, emphasised with another head nod (Figure 20d).
33
Figure 20: Two gestures of Representing In 0DGT27, one position for more, one for less competition. Source: Friedmans Agenda (02.07.2007).
The double gesture in Figure 20 does not convey a specific shape: neither linear, nor planar, nor three-dimensional. Rather, both hands together are merely relevant in their location, despite the imprecision of their configuration. Location is a zero-dimensional form, which is used in these two tokens of Representing In 0DGT27 to contrast the hands’ position in Figure 20b with that in Figure 20d, in accordance with the contrast in speech.
Discussion In the analyses in this paper, we have illustrated how gesture form, which includes a sequence of imaginary forms starting with the shape determination of the relevant articulator, is the basis of meaning making in gesture. We covered a broad range of spatial structures in gestures within the fuzzy boundaries of what is usually referred to as pointing. Yet many of these gestures could also be called not only deictic, or even mainly deictic, but also iconic, metaphorical or discursive, not because the gestures
34
themselves fall into these categories, but because, given their possible referents, they may acquire these functions in the context of the composite utterance of which they are a part (Enfield, 2009). What an action identified as a gesture contributes is the conveyance of complex spatial information. We call this complex spatial information gesture form. Gesture form can either stand for itself, as a path in a route description, or direct attention to physical or imaginary entities, for example the precise (or quite vague) pointing at a co-present object, or the metaphorical marking to progressive steps of someone’s career. In conceptualising gestural action, gesture form, as introduced here, is the foundation on which conceptual strategies in gesture build. Gesture form analysis (originally elaborated in Hassemer, 2016) incorporates insights into various structural aspects of spatial conceptualisation in bodily communication. These include Talmy’s (2000: 26-31) distinction of topological dimensions to gesture: Topological dimensions are neutral with respect to “magnitude”, “shape”, and “closure”. These neutralities explain how a single gesture type Drawing, which is here claimed to output a one-dimensional referent, can create as diverse referential shapes as a line that is long or short, straight or curved or even connected at its ends (e. g., forming a circle). These neutralities allow pointing gestures to indicate objects of any size, at any distance, along differently curved pointing paths; and also allow surfaces or volumes to point. In general, they allow capturing the spatial structure of pointing gestures through the systematic application of seven spatial cognitive operations. The combination of static and dynamic constituent forms within gesture form makes it possible for a single gesture to simultaneously indicate the location of a referent (deictic function), depict its shape (iconic function), and become the vehicle for beats (discursive function), in which, for example, the direction of a conic pointing gesture can indicate a focus of attention, the handshape can suggest its volumetric extent and position in time of the marked forward movements can emphasise elements of the cogestural speech. Thus gesture form analysis argues against using the terms deixis, iconicity, and beat as gesture categories, noting their lack of mutual exclusivity in both gesture types and gesture tokens; they are functions afforded by gesture form. Deictic information is conveyed by lines, surfaces and volumes shape-profiled by or extending away from the profiled articulator; iconic information is communicated equally by the static presentation of one- to three-dimensional shapes and by the motion of zero- to three-dimensional forms through space; and beat-like markings of a point in time and in
35
space are realised by thrust reversal in articulatory motion: and they can be afforded inseparably and simultaneously. All gesture form analyses were integrated in a typology of pointing gestures (Figure 1). The modular character of this typology allows deriving conclusions with regard to the structure of pointing gestures, such as the fact that Pointing AtGT1-4 gestures may be perceived as the most prototypical pointing types because their spatial structure is the only one that contains a precisely defined (0D) point in space (by the intersection of the 1D pointing vector with a physical or imaginary 2D surface). On the other hand, several pointing types were analysed that do not contain a vector, as commonly associated with prototypical pointing (Kita, 2003). Rather, it was seen that pointing gestures may use the whole range of zero- to three-dimensional forms to create meaning – even within a single type, such as Pointing At ObjectGT1-2. Further, and perhaps surprisingly, the most structural variation lies within pointing types that include profiling the hand as a two-dimensional shape, some of them not even containing a line or vector. Pointing is thus ‘multidimensional’ in several regards. Forms of different dimensions occur in different permutations across gesture types and each form has a variable degree of salience and a different role in creating the referent form, the salient end product of gesture form. A single gesture has multiple constituent forms and every analysis in this paper proposes that each constituent form and spatial operation is a necessary part of a gesture’s conceptualisation. In one way or another, the brain has to compute these geometrical operations in order to arrive at the referent. The claims regarding the structure of specific gesture tokens are deliberately weaker than for example the basic proposition that imaginary forms are part of gesture conceptualisation. If the reader rejects a specific gesture form analysis, this invites the proposal of an alternative series of constructs to account for the referent form that the given gesture communicates. Both accepting and rejecting the analyses presented here draws attention to spatial affordances and constraints in how the spatial modality of gesture affords the meaning conveyed. The proposed spatial operations may also be subject to change. New data or new perspectives on gesture conceptualisations may require adding other operations or changing, conflating, or rendering obsolete operations presented here. In the light of fundamental problems of universal typologies (Heath, 2016), this seems only natural. Gesture form analysis also enables testing specific hypotheses about gesture perception and production (Hassemer & Winter 2016; Hassemer & Winter, submitted). Candidates for such testing are the perceived 36
differences between any two gesture types, since they have to be somehow distinguished in gesture performance for successful communication. Claims on structural aspects of gesture form are not to be seen in competition with other more semiotic, functional, or speech-integrated approaches. The contrary is the case. The framework presented here distinguishes gesture form from other sources of meaning that impact gestural communication. This is done not merely to introduce an updated understanding of gesture form, but also to highlight its limitations. Clearer limitations of gesture form should also help teasing apart the independent contributions of other aspects of gesture interpretation: movement qualities such as velocity, acceleration, steadiness, laxness; mechanical implications of imitated physical action; different aspects of speech context such as lexical meaning and intonation; and the social, psychological and emotional context of the composite utterance. The typology presented here covers large parts of a complete gesture typology (Hassemer, 2016). It does not seem to be coincidental that pointing types reveal a large variability in spatial structure. Other gesture types, for example those that imply the imaginary holding of an object, sliding over it, or representing it fully in its threedimensionality – have simpler spatial structures, but they make use of already rich kinds of spatial forms (2D and 3D forms) whose detailed configuration is exploited and whose mechanical implications vary considerably. In Pointing, the focus lies on directing the interlocutor’s attention to spaces of different dimensions, sizes and shapes.
References Andrén, M. (2010). Children’s Gestures from 18 to 30 Months. Dissertation, Lunds Universitet. Arnheim, R. (2004[1969]). Visual Thinking. Berkeley: University of California Press. Bühler, K. (2000). Sprachtheorie. In L. Hoffmann (Ed.), Sprachwissenschaft. Ein Reader (2nd ed., pp. 51–71). Berlin: Walter de Gruyter. Butterworth, G. (2003). Pointing Is the Royal Road to Language for Babies. In S. Kita (Ed.), Pointing: Where language, culture, and cognition meet (pp. 9–34). London: Laurence Erlbaum. Cooperrider, K. A. (2011). Reference in Action: Links between pointing and language. Dissertation, UC San Diego.
37
Deacon, T. W. (1997). The Symbolic Species: The co-evolution of language and the brain. New York/London: W. W. Norton and Company. Enfield, N. J. (2009). The Anatomy of Meaning. Speech, gesture, and composite utterances. Cambridge/New York: Cambridge University Press. Fricke, E. (2007). Origo, Geste und Raum. Lokaldeixis im Deutschen. Berlin/New York: Walter de Gruyter. Gibson, J. J. (1977). The theory of affordances. In R. Shaw (Ed.), Perceiving, Acting, and Knowing. Towards an ecological psychology. Hillsdale: Lawrence Erlbaum. Goodwin, C. (2003). Pointing as situated practice. In S. Kita (Ed.), Pointing: Where language, culture, and cognition meet (pp. 217–242). London: Lawrence Erlbaum. Hassemer, J. (2016). Towards a Theory of Gesture Form Analysis. Imaginary forms as part of gesture conceptualisation, with empirical support from motion-capture data. Dissertation, RWTH Aachen University. Hassemer, J., Joue, G., Willmes, K., & Mittelberg, I. (2011). Dimensions and mechanisms of form constitution: Towards a formal description of gestures. In Proceedings of Gesture and Speech in Interaction (GESPIN) 2011. Bielefeld. Hassemer, J., & Winter, B. (2016). Producing and perceiving gestures conveying height or shape. Gesture, 15(3), 404–424. Winter, B., & Hassemer, J. (submitted). Decoding gestural iconicity. Cognitive Science. Haviland, J. B. (2003). How to point in Zinacantán. In S. Kita (Ed.), Pointing: Where language, culture, and cognition meet (pp. 139–170). London: Lawrence Erlbaum. Heath, J. (2016). Type-ology or typ-ology?. Linguistic Typology, 20(3): 479-495. Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge/New York: Cambridge University Press. Kendon, A., & Versante, L. (2003). Pointing by hand in “Neapolitan.” In S. Kita (Ed.), Pointing: Where Language, Culture, and Cognition Meet (pp. 109–138). London: Lawrence Erlbaum. Kita, S. (2003). Pointing: Where language, culture, and cognition meet. London: Lawrence Erlbaum. Kita, S., & Lausberg, H. Generation of co-speech gestures based on spatial imagery from the right-hemisphere: Evidence from split-brain patients. Cortex, 44 (2008): 131– 39. Kranstedt, A., Lücking, A., Pfeiffer, T., Rieser, H., & Staudacher, M. (2006). Measuring and Reconstructing Pointing in Visual Contexts. In Proceedings of the brandial 2006 -
38
The 10th Workshop on the Semantics and Pragmatics of Dialogue (pp. 82–89). Universitätsverlag Potsdam. Liddell, S. K. (2003). Grammar, Gesture, and Meaning in American Sign Language. Cambridge: Cambridge University Press. Lücking, A., Bergmann, K., Hahn, F., Kopp, S., & Rieser, H. (2010). The Bielefeld Speech and Gesture Alignment Corpus (SaGA). In 4th Workshop on the Representation and Processing of Sign Languages (LREC 2010): Corpora and Sign Language Technologies. University of Hamburg. Mandel, M. A. (1977). Iconic devices in American Sign Language. In L. A. Friedman (Ed.), On the Other Hand: New perspectives on American Sign Language (pp. 57– 107). New York: Academic Press. Mittelberg, I., & Waugh, L. R. (2009). Metonymy first, metaphor second: A cognitivesemiotic approach to multimodal figures of thought in co-speech gesture. In C. Forceville & E. Urios-Aparisi (Eds.), Multimodal Metaphor (pp. 329–356). Berlin/New York: Mouton de Gruyter. Müller, C. (1998). Redebegleitende Gesten. Kulturgeschichte – Theorie – Sprachvergleich. Berlin: Arno Spitz. Müller, C. (2008). What gestures reveal about the nature of metaphor. In A. Cienki & C. Müller (Eds.), Metaphor and Gesture. Amsterdam/Philadelphia: John Benjamins. Müller, C. (2014). Gestural modes of representation as techniques of depiction. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & J. Bressem (Eds.), Body – Language – Communication: An international handbook on multimodality in human interaction, Vol. 2 (pp. 1687–1702). Berlin/Boston: Mouton de Gruyter. Pfeiffer, T. (2010). Understanding Multimodal Deixis with Gaze and Gesture in Conversational Interfaces. Dissertation, Bielefeld University. Salffner, S. 2014. Farming, food and yam: language and cultural practices among Ikaan speakers: An archive of language and cultural material from the Akaan people of Ikakumo (Ondo State and Edo State, Nigeria). London: SOAS, Endangered Languages Archive. URL: http://elar.soas.ac.uk/deposit/0259 (accessed April1st, 2015.) Streeck, J. (2009). Gesturecraft: The manu-facture of meaning. Amsterdam: John Benjamins. Talmy, L. (2000). Toward a Cognitive Semantics. Vol. I Concept structuring systems. Cambridge: MIT Press. Talmy, L. (2018). The Targeting System of Language. Cambridge: MIT Press.
39
Wilkins, D. P. (2003). Why pointing with the index finger is not a universal (in sociocultural and semiotic terms). In S. Kita (Ed.), Pointing: Where language, culture, and cognition meet (pp. 171–216). London: Lawrence Erlbaum.
