Sound perception and design in multimodal ...

Sound perception and design in multimodal environments PerMagnus Lindborg Respondent /doctoral candidate Tal Musik Hörsel /Speech, Music, Hearing School of Computer Science and Communication KTH Royal Institute of Technology Main supervisor: Assoc Prof Dr Anders Friberg, KTH Asst supervisor: Assoc Prof Dr Ser Wee, NTU, Singapore Chairman of the dissertation: Prof Dr Sten Ternström, KTH Opponent: Prof Dr Dr Daniel Västfjäll, Linköping University, Sweden Examination committee: Prof Dr Mats E. Nilsson, Stockholm University, Sweden Asst Prof Dr Sarah Payne, Heriot-Watt University, UK Dr Pawel Herman, KTH

Monday, December 21, 2015

Disputation /viva voce 10 am Friday 11 Dec. 2015 Kollegiesalen, Brinellvägen 8, KTH

Sound perception and design in multimodal environments This dissertation is about sound in context. Since sensory processing is inherently multimodal, research in sound is necessarily multidisciplinary. The present work has been guided by principles of systematicity, ecological validity, complementarity of methods, and integration of science and art. The main tools to investigate the mediating relationship of people and environment through sound have been empiricism and psychophysics. Four papers focus on perception, and three on design. They constitute an investigation into how sound affects us, and what sound means to us.

2 Monday, December 21, 2015

Sound perception and design in multimodal environments semantic /visual / motoric association perceived expressive ratings

sound scapes music

‘features’

features

computed musical /psychoacoustic audio /acoustic

evoked


personality traits people individual differences self-report

physiological reaction


Fig. Cloud of the most frequent words (among 83,014 in total) in the dissertation. Made with Wordle (Feinberg 2014).

…sonic word-cloud?


Sound perception and design in multimodal environments Papers Future work

Perception Design Environments

Fig. CoreSound TetraMic for Ambisonic recordings.


Sound perception and design in multimodal environments

Papers


Papers Paper A

Lindborg PM (2013, June). “Physiological measures regress onto acoustic and perceptual features of soundscapes”. ICME3.

Paper B

Lindborg PM & Friberg AK (2015, submitted). “Personality traits influence perception of soundscape quality”. Noise & Health.

Paper C

Lindborg PM & Friberg AK (2015, Dec.). "Colour Association with Music is Mediated by Emotion: Evidence from an Experiment using a CIE Lab Interface and Interviews". PLoS ONE.

Paper D

Lindborg PM & Kwan NAK (2015, May). “Audio Quality Moderates Localisation Accuracy: Two Distinct Perceptual Effects?”. AES Conv.

Paper E

Lindborg PM (2015, March). “Psychoacoustic, Physical, and Perceptual Features of Restaurants: A Field Survey in Singapore”. Applied Acoustics.

Paper F

Lindborg PM (2015, submitted). “A taxonomy of sound sources in restaurants”. Applied Acoustics.

Paper G

Lindborg PM (2015, accepted). “Interactive Sonification of Weather Data for The Locust Wrath, a Multimedia Dance Performance”. Leonardo.


Papers PM (2013, June). “Physiological measures regress onto b o d y p e r c eLindborg p t i o n acoustic and perceptual features of soundscapes”. ICME3.

Paper A Paper B

Lindborg PMp&e Friberg AKn (2015, submitted). “Personality traits r s o a l i t y influence perception of soundscape quality”. Noise & Health.

Paper C

Lindborg PM & Friberg AK (2015, Dec.). "Colour Association with Music by Emotion o s sis -Mediated mod a l a: Evidence s s o from c i an a Experiment t i o n using a CIE Lab Interface and Interviews". PLoS ONE.

Paper D

Lindborg PM & Kwan NAK (2015, May). “Audio Quality Moderates s o u r c e l o c a l i s a t i o n Localisation Accuracy: Two Distinct Perceptual Effects?”. AES Conv.

Paper E

Lindborg PM (2015, March). “Psychoacoustic, Physical, and Perceptual Features of Restaurants: A Field Survey in Singapore”. Applied Acoustics.

Paper F

Lindborg PM (2015, submitted). “A taxonomy of sound sources in restaurants”. Applied Acoustics.

Paper G

Lindborg PM (2015, accepted). “Interactive Sonification of Weather Data for The Locust Wrath, a Multimedia Dance Performance”. Leonardo.

cr


How is the body affected by the sonic environment? -1.0 -0.5

play

1

1.0

1.0

0.5

grand mean = 0.199 slope (r) = -0.851

-1.0 -0.5

0.0

0.5 0.0 -1.0 -0.5

3

1.0

1.0

0.0 -1.0 -0.5 1.0

2

SC.r onto SM and VF interaction (95% conf.int.) r=0.680, VFp=0.015

0.0

ark

:cr

1

et

0.5

0

1:m

ow d 10 :wa ter

r 4:c 2:9h afé :aow riok el

7:r 8:ses hoto p

0.0ople 0.5

-1


0.5

0.0 -1.0 -0.5 1.0

-2 10

:pe

on 5:bstruc 6:n olly t igh t

3:c

-1.0

le :pe op

et ark 1:m

12

2

-1.0 -0.5

0

grand mean = 0.026 Example: SoundMass caused slope (≈ (r) = loudness) 0.975 change in Peripheral Temperature.

-1.0 -0.5

-1

Temp.m onto C2C (95% conf.int.) SM (SD) r=-0.377, SMp=0.038

:cr ow d

grand mean = 0.331 grand mean = -0.211 slope (r) =(r) -0.585 slope = -0.755

t

7 2:h:res

59:b :oor lly i 160:n ol :wigah tetr

4


0.5

0.5 0.0 -0.5

1.0

1.0

aw to 0.5 -1.0 -0.53 8:0.0 :c4oshokper :ncas tfréu 0.5 -1.0 -0.5 0.0 c

-0.5

1.0 0.0 -1.0 -0.5

-2

grand mean = 0.246 slope (r) = 0.916

Analysis revealed relationships between audio features and Autonomic Nervous System activation. 8:shop, Temp

12:people, Temp 7:resto, Temp



Sensation / involuntary reaction BPM.r

1.0 0.5 0.0 -1.0 -0.5

1.0 0.5 0.0 -1.0 -0.5

0.5 0.0

∆˚C (SD)

3:construct, Temp

Fig. A blind-folded participant in the 3D sound installation listening to reproductions of Ambisonic recordings. Note BPM.r onto VF (95% conf.int.) 5:bolly, Temp 6:night, Temp r=-0.498, p=0.037 sensors and “backpack” with wireless transmitter.

12

0.0 -1.0 -0.5

0.5

0.0 -1.0 -0.5

0.5

0.0 0.5 0.0 -1.0 -0.5

grand mean = 0.153 grand mean = 0.116 slope (r) =(r) -0.839 slope = -0.932

11:crowd, Temp

0.5

2:hawker, Temp

8:shop, TempTemp.r onto SM (95% conf.int.) 9:oriol, Temp 4:café, Temp r=-0.678, p=0.018 grand mean = 0.026 grand mean = 0.279 grand mean = 0.067 slope (r) = 0.975 slope (r) =(r) 0.986 slope = -0.981

1.0

∆˚C

-1.0 -0.5

0.5


1.0

6:night, TempTemp 1:market,

1.0

1.0

5:bolly, Temp

11

play

11

-1.0 -0.5

0.0

0.5

1.0

2:hawker, Temp 3:construct, Temp Lindborg PM (2013, June). “Physiological measures regress onto grand mean = 0.08 grand mean = 0.199 slope (r) = 0.897 slope (r) = -0.851 acoustic and perceptual features of soundscapes”. ICME3.

0.0

0.5

1.0

Paper A

9:oriol, Temp grand mean = 0.279 slope (r) = 0.986

Paper B Lindborg PM & Friberg AK (2015, submitted). “Personality traits influence perception of soundscape quality”. Noise & Health. Can personality explain how people perceive soundscapes? Openness

Emotional Stability

Conscientiousness

Agreeableness

Extraversion Fig. Part of screen-based rating interface for Swedish Soundscape Quality Protocol (Nilsson, Axelsson, Berglund 2010)

r = -0.065

0.5

40

0.5

0.5

Fig. Big Five dimensions (e.g. McCrae & Costa 1997)

r = -0.139

r = -0.064

7

2

3

4

5

EmoStability Monday, December 21, 2015

6

7

0.0

Analysis revealed the influence of personality traits on quality ratings. -0.5

Eventfulness

0.0

Example: effect size of personality traits was up to one-tenth of psychoacoustic descriptors. -1.0

-1.0

-0.5

Pleasantness

0.0 -0.5 -1.0

Pleasantness

Cognitive appraisal

-30

-20

-10

0

Weinstein

10

20

11

1

2

3

4

5

Extraversion

6

7

Paper C Lindborg PM & Friberg AK (2015, Dec.). "Colour Association with Music is Mediated by Emotion: Evidence from an Experiment using a CIE Lab Interface and Interviews". PLoS ONE. How does perception of music relate to colour? play

Valence colour = audio + emotion + Ɛ

audio features

audio features and dimensional emotions

audio features and discrete emotions

Fig. Response interface in action.

Crossmodal correspondence

0.6

Analysis revealed that emotions contributed over and above audio features in explaining variation in colour responses. Mixed methodology approach with participant interviews strengthened emotion mediation hypothesis.

0.4

0.5

Crossvalidated adjusted R2

0.7

0.8

PLS models predicting colour patch parameters


Size Lightness a* b*

12

Future work (paper C) Lindborg PM, Hendinata J, et al. (2016). "Colour Association with Music: a web experiment” Can personality traits predict crossmodal association?

http://soundislands.com/webexp/CAS/ 13 Monday, December 21, 2015

Paper D Lindborg PM & Kwan NAK (2015, May). “Audio Quality Moderates Localisation Accuracy: Two Distinct Perceptual Effects?”. AES Conv. Does audio quality affect the perceived auditory space? -20 -10˚ (-30˚) ˚

10˚ 20˚

(30˚)

play

play

Crossmodal effect audio ‘clarity’ → space higher audio quality

enables higher localisation accuracy

{ causes a widening of the soundstage 14


Future work (paper D) Lindborg PM (2016). "Audio quality influences perceived spatial image” What aspects of audio quality provoke the image widening effect?

Exp. 1

Fig. Participant in source localisation experiment 2. Note blindfold, head fixation, and joining of left-right indeces.

Exp. 2



Paper A Paper B


Paper C


Paper D


cr

design

Paper E

Lindborg PM (2015, March). “Psychoacoustic, Physical, and Perceptual environment Features of Restaurants: A Field Survey in Singapore”. Applied Acoustics.

Paper F

Lindborg PM (2015, submitted). “A taxonomy of sound osources u r inc restaurants”. e v s . Applied e nAcoustics. vironmen

Paper G

s c

Lindborg PM (2015, accepted). “Interactive Sonification of Weather Locust rData o sfors The -m o dWrath a l, a Multimedia a s s oDance c i aPerformance”. t i o n Leonardo. 16


t

Paper E

4

4

4

3

3

3

1

1

1

Lindborg PM (2015, March).“Psychoacoustic, Physical, and Perceptual 2 2 Features of Restaurants : A Field Survey in Singapore”. Applied2 Acoustics. Which design features are important BB Café FF Hawk Din for restaurant soundscape quality?

BB

FF

Hawk

Din

BB

FF

Hawk

Din

N50 Qual Visual

5

25

4

4

2.0

72 3

70

20

3

Priciness 2.5

5

74 4

Café

LAeq Sonic Qual

Fan

5

Café

3

1.5 68

2

66

1 BB

Café

FF

Hawk

Din

1.02 1.00 0.98

3

0.96

2

0.94

1

0.90

5

0.88

0.92

occupancy

4

4 3

4

5

3

mean ratings (Likert scale)


2

2

1.0

1 BB Café Café FF

FF HawkHawk Din Din

BB BBCaféCaféFF FFHawkHawk Din Din

Nature 5

2

15

1

BB

Correlation between Occupancy and ratings of Sonic and Visual Environment Overall Quality

1

2

BB

Human 5

4

4

3

3

Typology of Acoustic Design Elements On-site quality ratings (SSQP), observations, 2 recordings, photography in 112 restaurants.

Hawk

2

1 Annotations of interior1 design materials supported BB Café FF Hawk Din BB Café Hawk Din BB classification by Design Style andFFFood Style.

Fan Sonic Qual Example: occupancy and perceived 5 environmental quality were negatively related.

FF

Traffic

5

Restaurant sonic environment

17

Café

Café

FF

Hawk

Visual Qual 5

4

4

3

3

2

2

Paper F Lindborg PM (2015, submitted). “A taxonomy of sound sources in restaurants”. Applied Acoustics. Which sounds are liked or disliked in restaurants? talking kitchen/washing clatter/sound.tableware eating/slurping machine.cooking machine/traffic signal steps/walking negative crying/shouting/ringtone noise screech/sound./chair steps/walking.people

-1.0

-0.5

0.0 Liking

Restaurant sonic environment Systematic classification of free-form annotations of characteristic sounds yielded a taxonomy (+validated). Analysis revealed perceptual and crossmodal effects. Example: voice-related annotations of characteristic sounds where ‘people’ was included as a specifier were more liked: possible emotional crossmodal association mechanism. 18 Monday, December 21, 2015

Future work (paper F) Lindborg PM et al. (). "A restaurant soundscape simulator /virtual environment”


Paper G Lindborg PM (2015, accepted). “Interactive Sonification of Weather Data for The Locust Wrath, a Multimedia Dance Performance”. Leonardo. How can sound be designed so that its meaning is understood intuitively?

Crossmodal design principles Hierarchical auditory system tasks informed mapping strategy. 1) “How does the sound physically manifest itself?” Reflexive, conditioned response (danger alert). Precipitation → Pluck Pressure → Detuning 2) “Where does the sound come from? Approaching or receding?” Spatial cues, pre-attentive, kinaesthetic action-sound coupling (environmental orientation). Geographical lattice --> Spatialisation and Tessitura 3) “What does the sound evoke in me? Aggressive or inviting?” Experiential listening, semantic /emotional crossmodal association (appraisal). Temperature --> Resonance Humidity --> Vibrato depth Wind Speed --> Sharpness 20 Monday, December 21, 2015


Paper A Paper B


Paper C


Paper D


cr

design

Paper E

Lindborg PM (2015, March). “Psychoacoustic, Physical, and Perceptual environment Features of Restaurants: A Field Survey in Singapore”. Applied Acoustics.

Paper F

Lindborg PM (2015, submitted). “A taxonomy of sound osources u r inc restaurants”. e v s . Applied e nAcoustics. vironmen

Paper G

s c

Lindborg PM (2015, accepted). “Interactive Sonification of Weather Locust rData o sfors The -m o dWrath a l, a Multimedia a s s oDance c i aPerformance”. t i o n Leonardo. 21


t


Kappa (Introduction)


Perception (1) Information processing stages

(Schnupp et al. 2011; Mesulam 1998)

Sensation physio-neurological, mainly reflexive (Moore & Linthicum 2011). brain stem = ancient brain structure, subserves auditory perception crucial to survival, “active 24/7”

Perception neurological mediates between sensation and cognition (cf. afferent & efferent innervation)

Cognition psychoacoustic models (Fastl & Zwicker 2007) neuro-psychological “computation in mentalese” (Fodor 1975)

Affect

auditory cognition, e.g. BRECVEMA (Juslin & Västfjäll 2008; Juslin 2013) all evaluative mental states (emotion, mood, preference… Juslin & Västfjäll 2008).

Emotion affective states = valenced (Osgood et al. 1957; Mehrabian & Russell 1974; Russell 1979) relatively brief duration (cf. mood)

Appraisal distinction induced vs. perceived emotion ≈ blurred (Gabrielsson & Lindström 2010) most emotions encountered in everyday listening, especially music (Juslin 2013) Swedish Soundscape Quality Protocol (Axelsson, Nilsson & Berglund 2010; Axelsson 2011)

Individual broad personality traits (John & Srivastava 1999; Russell & Mehrabian 1977) differences narrow construct: noise sensitivity (Weinstein 1978; Belojevic et al. 2012) 23 Monday, December 21, 2015

Perception (2) Sound as soundscape

(Schafer 1977; Truax 2001; Augoyard et al. 2006; Kang 2010)

“perceptual construct originating in sound sources, distributed in space and time, in a physical environment” (BS/ISO 2014)

Ubiquity

Metabole

Sound as event

Source Significance

“everywhere” (Amphoux 1995) diffuse, unstable, omnidirectional sound (Hellström 2003) “metabolic effect is in time what ubiquity is in space” (Chelkoff 1995; 2006) whole soundscape perceived as a static entity ≈ blurred detail

event = smallest self-contained part of a soundscape (Schafer 1977) two typological divisions of the sonic realm referential aspects ∈ {'natural', 'human', 'society', 'mechanical', 'silence', 'indicators') purpose ∈ { ‘keynote’, ‘signal’, ‘soundmark’}

anthrophony geophony

biophony

Fig. Three classes of sound by source. After Krause (2008).

Synechdoche

evaluation and selection (Thibaud 1995; cf. Västfjäll 2003; Bregman 1990) organises perception of time; enables experience of duration (Thibaud 2006) recognition (cf. Tuuri & Eerola 2012), language learning (Thibaud 2006) 24


Design Acoustic design sound = temporal ≠ {visual, architectural, written, commercial} = static

Clairaudience respect for the ear and voice

awareness of sound symbolism knowledge of the natural soundscape and its balancing mechanisms (Schafer 1994; Hellström 2003; cf. Tuuri & Eerola 2012)

Restaurants

'(/$3)

= servicescapes characterised by “elaborate physical complexity” (Bitner 1992)

!,)*(+*&#"$/#*$,) !"#$%#&'$()!(

/$-&"#*$,)

*.2,",3. -,'(" *#+,),-.

/.)*4(/$/ -(#/&%(-()*

1#"$'#*$,) ,0/(%1#*$,)

/,&)' 2(%!(2*$,)

9("#*$,)/4$280(*:(()8!,)!(2*/


6

5 2%,'&!*$1(78!"#$%&'"(

6

5 $)*(%2%(*#*$1(78!")*+,-#'"(

Environments Complexity

unimodal

multimodal

bimodal

Plasticity Designed music, film, and multimedia performance (Wishart 1994, 1996; Chion 2009) Semi-designed servicescapes (e.g. restaurants), streetscapes, urban parks

"every urban moment has a sound signature” (Augoyard & Torgue 2006)

Non-designed pristine wilderness, extreme urban environments Communication and meaning

Soundscape, Music, Speech = systems of acoustic communication (Truax 2001)

Acoustic archetypes Designability (control, intentionality)

nature

+

servicescape soundscape

music

art

speech

density /repertoire size

syntactical strictness


+

Sound Perception and Design in Multimodal Environments

servicescape

sensorial

Complexity

nature

multimodal

wilderness

nature parks

farmland

urban parks

streetscape

shops

dinescape (restaurants)

art multimedia performance

installation, game

film

unimodal non-designed

designed

semi-designed

Paper A

Paper B

Papers E, F


electroacoustic music

Paper G

Paper C

Paper D

Plasticity

“Swordle”

Wordle©

audiovisual perceptualisation of dissertation text in 95.2 seconds


[Swordle] https://vimeo.com/user24324396



Thank you for your attention! http://permagnus.org/


References Amphoux P (1995). “Ubiquité”. In Augoyard JF & Torgue H (eds.) A l’écoute de l’environment. Répértoire des effets sonores. p. 141-158. Augoyard JF & Torgue H (eds.) (1995). A l’écoute de l’environment. Répértoire des effets sonores. Editions paranthèses, France. Augoyard JF & Torgue H (eds.) (2006). Sonic Experience. A Guide to Everyday Sounds. McGill-Queen’s University Press, Quebec, Canada. Axelsson Ö (2011). Aesthetic Appreciation Explicated [Thesis]. Stockholm University. Axelsson Ö, Nilsson ME & Berglund B (2012). “The Swedish soundscape-quality protocol”. Journal of the Acoustical Society of America, 131(4), 3476-3476. Belojevic G, Jakovljevic B & Slepcevic V (2003). “Noise and mental performance : Personality attributes and noise sensitivity”. Noise & Health, 6:21, 77-89, http:// www.noiseandhealth.org/text.asp?2003/6/21/77/31680 (31 Oct. 2015). BS/ISO (2014). Standard 12913-1 on Acoustics. Soundscape. Part 1: Definition and conceptual framework. ICS Classification ISBN 978-0-580-78309-8. Chelkoff G (2006). “Metamorphosis”. In Augoyard JF & Torgue H (eds.) Sonic Experience. p. 73-78. Chelkoff G (1995). “Métabole”. In Augoyard JF & Torgue H (eds.) A l’écoute de l’environment. p. 86-91. Chion M (2009). Film, A Sound Art. Columbia University Press. Fastl H & Zwicker E (2007). Psychoacoustics: Facts and Models. 3rd ed. Springer-Verlag, Germany. Feinberg J (2014). Wordle. URL http://www.wordle.net/ Fodor JA (1998). "The Language of Thought". In Bechtel & Graham (eds.), A Companion to Cognitive Science. Blackwell Publishers. Gabrielsson A & Lindström E (2010). “The role of structure in the musical expression of emotions”. Chapter 14 p. 367-400 in Juslin PN & Sloboda JA (eds.) Handbook of Music and Emotion. Oxford University Press. Hellström (2003). Noise Design. Architectural Modelling and the esthetics of Urban Acoustic Space. (Doctoral thesis). Bo Ejeby Förlag. John OP & Srivastava S (1999). “The Big Five Trait Taxonomy: History, Measurement, and Theoretical Perspectives”. In Pervin & John (eds.) Handbook of Personality. Theory and Research. 2nd ed, Guilford Press, USA, pp. 102-138. Juslin PN (2013)."From everyday emotions to aesthetic emotions: towards a unified theory of musical emotions." Physics of Life Reviews 10.3, 235-266. URL http:// www.sciencedirect.com/science/article/pii/S1571064513000638 (31 Oct. 2015). Juslin PN & Västfjäll D (2008). "Emotional responses to music: The need to consider underlying mechanisms". Behavioural and Brain Studies, 31, 559 – 621. URL http://www.sciencedirect.com/science/article/pii/S1571064513000638 (a31 Oct. 2015). Kang J (2010, Nov.). “From understanding to designing soundscapes”. Frontiers of Architecture and Civil Engineering in China, 4:4, 403-417. Krause B (2008). "Anatomy of the Soundscape". Journal of the Audio Engineering Society 56:1/2, 73-80. Mehrabian A & Russel JA (1974). An Approach to Environmental Psychology. MIT Press. Mesulam MM (1998). “From sensation to cognition”. Brain, 121, 1013–1052. Moore JK & and Linthicum FH (2011). “Auditory System”. Chapter 34 in Mai JK & Paxinos G (eds.) The human nervous system. 2nd edition. Academic press. Osgood CE, George JS & Tannenbaum PH (1957). The measurement of meaning. Urbana, University of Illinois Press. Russell JA (1979). “Affective space is bipolar”. Journal of Personality & Social Psychology 37(3), 345-356. Russell JA & Mehrabian A (1977). “Evidence for a three-factor theory of emotions. Journal of Research Personality, 11(3), 273–294. Schafer RM (1994/1977). The Soundscape: Our Sonic Environment and the Tuning of the World. Destiny Books, USA, pp. 1-320. Schnupp J, Nelken I & King A (2011). Auditory neuroscience: Making sense of sound. MIT Press. Thibaud JP (2006). “Synechdoche”. In Augoyard & Torgue (eds.) Sonic Experience. p. 123-129. Thibaud JP (1995). “Synechdoque”. in Augoyard JF & Torgue H (eds.) A l’écoute de l’environment. Répértoire des effets sonores. p. 134-140. Truax B (2001). Acoustic Communication. Greenwood Publishing Group, USA. pp. 1-284. Tuuri K & Eerola T (2012). “Formulating a Revised Taxonomy for Modes of Listening”. Journal of New Music Research, 41(2), 137-152. Weinstein ND (1978). “Individual Differences in Reactions to Noise: A longitudinal study in a College Dormitory”. Journal of Applied Psychology 63, 458–466. Wishart T (1996). On Sonic Art. Psychology Press. 31 Wishart T (1994). Audible Design. York, Orpheus the Pantomime. Monday, December 21, 2015