Sound Design Education in Interaction Design at

AIA-DAGA 2013 Merano

Sound Design Education in Interaction Design at Zurich University of the Arts: Enabling Sound Thinking for Sound Design Daniel Hug Zurich University of the Arts, 8000 Zurich, Switzerland, Email: [email protected]

Introduction At the Interaction Design field of the Design Department of the Zurich University of the Arts, sound has been an important element in the curriculum for years. This is motivated by the advantages of multimodal interfaces, and the fact that many everyday commodities have small or no screens, or even disappear in our environment. Electroacoustic sound can be a powerful way of shaping the aesthetic and functional aspects of such artifacts, beyond the modification of material and mechanical properties. From a pedagogic standpoint, several challenges are associated with integrating sound in the design practice. First, we need to enable our students to make informed decisions about when and how to use sound in potential future applications. We also need to take into account that sound is but one of many aspects they need to consider, with varying relevance for their professional future. And finally, there is a need for design methods suitable for the specific demands of interaction scenarios employing sound. Instead of aiming at mastering certain tools and typical production tasks, our strategy focuses on the development of the understanding of sound as material for creative expression in the context of interaction design sound thinking for sound design. Based on this strategy we have devised a series of exercises and project tasks, which also constitute a prototypical design process which is integrated with our general interaction design process. Parts of this educational framework have been described in [7, 9, 11]. In this paper we will present a comprehensive overview over our approach to sound education in the field of interaction design1 .

Speaking about Sound A spectre is haunting the community of sound designers, the spectre of the myth of the sound designer as a kind of magician who receives an assignment, sits down in his windowless basement studio and does some obscure work with a result that nobody really understands. The sound designer’s reality, however, is much more earthly and requires the ability to formulate and communicate ideas, concepts and experiences in a systematic way to fellow designers, clients, and other stakeholders [12]. Thus, the first pillar of our sound design curriculum is focused at the ability to listen and understand sonic phenomena 1 Two example courses are documented at: http://blogs.iad.zhdk.ch/sounddesign-hs12/ http://blogs.iad.zhdk.ch/sid-hs12/

from all possible perspectives, and also to get acquainted with the various vocabularies and conceptual frameworks that can be used in order to describe sonic experiences. It all starts with sensitizing towards listening and then builds on methodical modules which broach all dimensions of sound, from acoustics to sound phenomenology and sound semiotics. In addition, an understanding of objects (and their sounds) as meaningful elements in socio-cultural processes and in meaning making is conveyed. This knowledge forms the basis for designing ”object-sound reconfigurations”, i.e. a sound design, which is aware of the fact that it deals with the dialectic relationship between sounds, objects and actions, and the ”schizophonic” condition that emerges from combining electroacoustic sounds with actual physical objects (see [8] for details).

Beyond Des(eye)gn: Sensitizing for Sound and Listening The majority of our students (and designers in general) have a predominantly visual understanding of design. To think about the sonic dimension of our world, and the artifacts we design for it, is unfamiliar to many of them. Thus we have to sensitize their hearing and stimulate ”critical listening”. For this purpose, we perform a socalled ”ear cleaning”, inspired by Schafer’s didactical suggestions [17], but extending Schafer’s approach by the close, analytic listening to works of experimental electronic music, in particular of musique concrète.These works not only challenge our abilities to describe sounds, but also convey an idea of what ”sound design” can mean. Also a short overview over the history of sensory culture and the various shifts of the paradigms and ideologies associated with hearing and vision, from the greek antique via the area of enlightenment up to the current revival of the auditive and oral [21], is provided.

Acoustics and Hearing In this module the students are familiarized with the fundamentals of sound generation and perception. The key is not so much in a comprehensive understanding of acoustics and psychoacoustics, but rather in understanding key concepts and being able to differentiate between acoustic and perceptual dimensions of sound. This improves precision in communication about sound and provides links to design solutions2 .

2 For instance, the shepard-risset tone can be used as example of a means of blending sounds based on their overtones.

2160


Sound Object/Event, Sounding Objects In this episode, two fundamental phenomenological understandings of sounds are discussed. One perspective emphasizes sound as ”object” of perception. For their investigation the ”écoute reduite” [16], the reduced listening, which focuses at the analysis of the ”sound object” as such, independent from its source and possible meaning, is employed. The other, seemingly opposing, understanding emphasizes that sonic phenomena often are strongly linked to the (physical) event that caused them, and that they can be best understood through a causal listening [5]. Both understandings have their value and thus are used in a pragmatic way in our educational framework. In the context of interaction design, another dimension of meaning emerges from the basic action-sound relationship in the physical world, of an enactive understanding of how our perception and interpretation of objects and processes are influenced by sound. This includes also the area of product sound design and ”product sound quality” [1, 3]. In order to acquire competences in this area, everyday objects are analyzed in terms of their identity and character, the reasons for why they sound the way they sound and how sound contributes to their perceived quality. Using the typology of sound objects proposed by Schafer [17], the sounds are visualized as abstract ”sonic objects”. Sketching a visual representation of a sound’s essential characteristics contributes to both the descriptive and prescriptive understanding of sound in the context of objects and actions.

in the context of interaction are the fields of Auditory Display and Sonification, ranging from basic alarms, status informations etc. to the concepts of Auditory Icons and Earcons and the complex sonifications of multidimensional data (see, e.g. [6]). Building on the understanding of sound as semiotic device, its use in building narratives is scrutinized. Not conventional storytelling is of interest here, but rather the use of sound to tell something about a specific interaction with an artifact. For this the framework of ”narrative metatopics”3 is used [9]. A selection of clips and associated metatopics are discussed, to familiarize participants with elaborated sound design strategies.

Contextualized scape Studies

Relationships:

Sound-

An important precondition for the creation of sounds, that can enter a meaningful functional and aesthetic relationship with other sounds in a given setting, is sensitizing designers for the ”acoustic ecology” [18]. For this purpose, the concept of the soundscape as a system, constituted by interacting sounding objects and agents in a specific environment, is introduced. An emphasis lies on the understanding of sound in the context of acoustic communication and acoustic community [18, 19]. This is complemented by suitable visualization frameworks, such as those proposed in [14], which can serve students to develop their own visualization of soundscapes (see Figure 2).

Designing Sounds While in our process many of the necessary processes and tools for actually producing sounds are quite conventional the task of designing sounds for interaction scenarios requires a different approach than ”usual” sound design. Signal paths, or details of post-processing and mastering are less relevant, more relevant are methods that allow to convey a possible experience in a plausible, dynamic and improvisational way, in order to support the design for inherently dynamic interactions. In our courses, the actual production of sound is always approached in the context of larger application design scenarios such as Service Design or Physical Computing projects. This also requires introductions to related fields, such as Sonic Branding. In the following we describe the specific steps and didactical elements used for sound creation.

Everyday Fiction Figure 1: Visualisation of a sound event/object according to Schafer [17].

Sound as Sign and Sonic Narratives After investigating the fundamentals of sound, more complex ways of creating meaning through sound are looked at. This involves semiotic codes and second order semantics. A central source here comes from the analysis of film sounds [4] and from sound semiotics in general [13, 20]. Another relevant area for meaningful sound

Building on the narrative analysis of film clips described above, this activity introduces a quick and dirty method of prototyping sound experiences audiovisually. Participants are asked to document everyday interactions and objects using video recordings. Then they are asked 3 Narrative metatopics are recurring high-level narratives which emerged from the analysis of a large set of clips from fictional media such as film and games, in which sound played a central interpretive role. They provide a powerful means to navigate a semantic space associated with interactions, and help to ground sound design decisions [9].

2161


ing soundscapes as a whole. Thiis is achieved by investigating a specific soundscape, e.g. of a shop, or a children playground, and then recreating sounds of interactions and products found in this space by ”transplanting” desired properties from another domain. For instance the soundscape of a shop may be re-modeled by sonic characteristics found in the living room.

Performance-Oriented Mockup Strategy The previous methods and exercises are focusing on the elaboration of sonic details and modeling their sonic context. They are also ideal to get acquainted with the more traditional tools for sound design, such as recording, wave editing and multitrack sequencing. To move towards prototyping and evaluating interactive experience an additional step, which is called ”experience prototyping” [2], is added. , For this method, which is common in interaction design, techniques from performative arts can be used. A prominent example is ”bodystorming”, a method of generating ideas by acting out scenarios [15]. As consequence, live performance and improvisation of sounds is a necessary element when using such methods. In the following we outline the methods and related exercises (for details please refer to [10]). Figure 2: A visualisation of soundscape elements designed by students.

to find narrative links between the fictional metatopics identified and the everyday experiences documented, and to redesign the sounds of the everyday experiences, using the available sound designs identified in the fictional scenarios of the clips. This allows the rapid prototyping of design ideas, relying on the elaboration and quality of professional sound design, without having to go through an elaborate sound design process. It also allows a preliminary verification of the usefulness and transferability of the fictional metatopics to everyday artifacts. The same method can be used to create early draft video prototypes of interactions.

Sonic Transplantation This activity familiarizes the participants with the fundamental process of modeling sound using another sound as ”material”. This is also one of the core strategies used in sound design to create sounds that are familiar and novel at the same time. For this activity, the participants are also required to create their own sound library. Thus they do not only gather experiences with recording, but they can also activate their knowledge about describing and categorizing sounds. Simultaneously, this activity serves to teach the minimal skills required for using multitrack sequencing and digital signal processing (via plugins and automatization). This reflects our philosophy regarding ”teaching” tools, which is always embedded in a overall design method and application context.

Semantic Foley and Live-Foley Exercises: As prerequisite for creating performance-oriented mockups we need to practice the ability to ”sketch” with sound ”on the fly” by using soundmaking tools which are readily available, such as objects, the voice and the body. This is inspired by common techniques used in film Foley. We employ two activities: In the ”Semantic Foley” activity, the participants express combinations of a process (e.g. charging, activation, connection) and an attribute (e.g. dangerous, weak, significant), using only the sound sketching methods described above. The audience is asked to interpret the sounds. This way, both Foley techniques and critical listening are trained. An additional step for training live sound making skills is a live Foley exercise using one of the film clips which have been discussed earlier in relation to ”narrative metatopics” (see Figure 3). This exercise creates a transition between the understanding of filmic sound narratives and the performative quality of Foley sound making.

Soundscape Redesign Building on a combination of soundscape analysis and sonic transplantation, the participants employ another method of sonic prototyping, which focuses on redesign-

Figure 3: A group performing realtime-foley to a film clip.

A Two-Stage Experience Mockup Process: The final step of the educational framework and the related

2162


design process is the actual experience prototype. This is developed in two subsequent steps, which both employ a variation of the ”Wizard of Oz” mockup technique. The first prototype only uses Foley techniques. The sound is picked up by a microphone and transported to the location of the performance using loudspeakers, achieving a ventriloquist effect. In the second iteration, the sounds are designed using electroacoustic tools and DSP, and then played live to participants interacting with the prototypes, using real-time MIDI controls. Further details can be found in [10]

reconfigurations for interactive commodities. In Proceedings of Audiomostly 2008, 3rd Conference on Interaction With Sound, pages 56–63, Pitea, Sweden, 2008. [9] Daniel Hug. Investigating narrative and performative sound design strategies for interactive commodities. In Sølvi Ystad, Mitsuko Aramaki, Richard Kronland-Martinet, and Kristoffer Jensen, editors, Auditory Display - 6th International Symposium, CMMR/ICAD 2009, Copenhagen, Denmark, May 18-22, 2009, Revised Papers, volume 5954 of Lecture Notes in Computer Science, pages 12–40. Springer, 2010.

Conclusion In this paper, we have outlined an educational framework for teaching sound design to interaction designers. The methods and exercises are not meant to be for ”classroom use” only, but are intended for the use in real-life design and research practice. One main concern is the appropriate sensitization of designers for issues related to sound, both as aesthetic experience and functional element in the interaction process, and to understand the relevant contextual factors. The other focus lies on enabling interaction designers, who usually are not particularly trained in music or sound design, to create convincing experience prototypes of interaction scenarios with sound. This strategy enables interaction designers to approach sound not as an obscure and rather unsettling aspect of design, but to integrate it as an equivalent element of a holistic, multisensory design strategy.

References [1] Jens Blauert and Ulrike Jekosch. Sound quality evaluation - a multi-layered problem. Acustica, 83:747–753, 1997. [2] Marion Buchenau and Jane Fulton Suri. Experience prototyping. In Proceedings of the Conference on Designing Interactive Systems, pages 424–433, Brooklyn, New York, 2000. [3] H. Fastl. Psycho-acoustics and sound quality. In J. Blauert, editor, Communication Acoustics. Springer, Berlin-Heidelberg, 2005. [4] Barbara Fl¨ uckiger. Sounddesign: Die virtuelle Klangwelt des Films. Sch¨ uren Verlag, Marburg, 2001. [5] W. W. Gaver. What in the world do we hear? An ecological approach to auditory event perception. Ecological Psychology, (5):1–29, 1993. [6] Thomas Hermann, Andy Hunt, and John G. Neuhoff, editors. The Sonification Handbook. Logos Publishing House, Berlin, Germany, 2011. [7] Daniel Hug. Game Sound Education at ZHdK - Between Research Laboratory and Experimental Education. In Proceedings of Audio Mostly 2007 2nd Conference on Interaction with Sound, 2007. [8] Daniel Hug.

Genie in a bottle:

Object-sound

[10] Daniel Hug. Performativity in design and evaluation of sounding interactive commodities. In Proceedings of Audio Mostly 2010 - 5th Conference on Interaction with Sound, 2010. [11] Daniel Hug. Barking wallets and poetic flasks exploring sound design for interactive commodities. In Karmen Franinovic and Stefania Serafin, editors, Sonic Interaction Design. MIT Press, Cambridge, Mass., 2013. [12] Daniel Hug and Nicolas Misdariis. Towards a conceptual framework to integrate designerly and scientific sound design methods. In Proceedings of Audio Mostly 2011 - 6th Conference on Interaction with Sound, Coimbra, Portugal, 2011. [13] Ute Jekosch. Assigning Meaning to Sounds Semiotics in the Context of Product-Sound Design. In Jens Blauert, editor, Communication Acoustics, pages 193–221. Springer, 2005. [14] Iain McGregor. Soundscape mapping: A tool for evaluating sounds and auditory environments. In Proceedings of the 16th international conference on Auditory Display, Washington D.C., 2010. [15] Antti Oulasvirta, Esko Kurvinen, and Tomi Kankainen. Understanding contexts by being there: case studies in bodystorming. Personal Ubiquitous Comput., 7(2):125–134, 2003. [16] Pierre Schaeffer. Traité des objets musicaux. Seuil, Paris, 1966. [17] R. Murray Schafer. Ear Cleaning: Notes for an Experimental Music Course. Associated Music Publishers, Inc., New York, 1967. [18] R. Murray Schafer. The Soundscape: Our Sonic Environment and the Tuning of the World. Destiny Books, New York, 2nd edition, 1977. [19] Barry Truax. Acoustic Communication. Greenwood Press, Westport, CT, 2nd edition, 2001. [20] Theo van Leeuwen. Speech, Music, Sound. Palgrave Macmillan, Houndmills and London, 1999. [21] Wolfgang Welsch. Auf dem Weg zu einer Kultur ¨ des Hörens? In Grenzg¨ ange der Asthetik. Reclam, Stuttgart, 1996.

2163