Automatic sign language recognition for sign ...

3 downloads 0 Views 9MB Size Report
May 27, 2018 - Interkantonale Hochschule für Heilpädagogik (HfH), CH. • University of Surrey, GB. • Swiss National Science Foundation (SNSF):. • Three-year ...
EALTA Conference Technology-based language assessment: Benefits and challenges

Automatic sign language recognition for sign language assessment May 25-27, 2018

Tobias Haug, Sarah Ebling, Penny Boyes Braem, Katja Tissi, & Sandra Sidler-Miserez

Overview • • • •

Structure of sign languages Swiss context SMILE project Methodological challenges

EALTA Conference

Haug et al., HfH

2

Structure of sign languages •



Manual components • Handshape • Location • Movement • Hand orientation Non-manual components • Eye brows • Forehead • Cheeks • Eye gaze • Mouth • Upper torso

EALTA Conference

Haug et al., HfH

Source: SLAS 2015

3

Structure of sign languages

Handshapes [1]

Overview of handshapes in Swiss German Sign Language (DSGS)

Boyes Braem, 1995 EALTA Conference

Haug et al., HfH

4

Structure of sign languages

Minimal pairs

DSGS-Gebärden SAGEN („say“) und FRAGEN („ask“) Boyes Braem, 1995 EALTA Conference

Haug et al., HfH

5

Swiss context Three national sign languages according to the three main (spoken) language regions • Swiss German Sign Language (Deutschschweizerische Gebärdensprache, DSGS) • ca. 5,500 deaf DSGS users • French Sign Language (Langue des Signes Française, LSF) • ca. 1,700 deaf LSF users • Italian Sign Language (Lingua Italiana dei Segni, LIS) • ca. 300 deaf LIS users Boyes Braem, Haug & Shores, 2012 EALTA Conference

Haug et al., HfH

6

SMILE Scalable Multimodal Sign Language Technology for Sign Language Learning and Assessment

EALTA Conference

Haug et al., HfH

7

SMILE

Project partners • Idiap Research Institute, CH • Interkantonale Hochschule für Heilpädagogik (HfH), CH • University of Surrey, GB • Swiss National Science Foundation (SNSF): • Three-year Sinergia project

EALTA Conference

Haug et al., HfH

8

SMILE

Project team HfH • Sarah Ebling, project coordinator HfH • Penny Boyes Braem • Katja Tissi • Sandra Sidler-Miserez • Tobias Haug

EALTA Conference

Haug et al., HfH

9

SMILE

Project goal

Develop an automatic sign language recognition and assessment system to be used in a DSGS vocabulary L1/L2 translation test

EALTA Conference

Haug et al., HfH

10

SMILE

Automatic sign language recognition

Recognizing the form of a sign

EALTA Conference

Comparing it with a "reference"

Haug et al., HfH

Feedback to the learner

11

SMILE

Automatic sign language recognition

Recognizing the form of a sign

EALTA Conference

Comparing it with a "reference"

Haug et al., HfH

Feedback to the learner

12

SMILE

Cameras • •

1 Kinect (skeleton and depth sensor), HD resolution, 30 frames/second 2 GoPros • •



1x with high framerate (240 frames/second, HD resolution) 1x with high resolution (4K, 30 frames/second)

3 HD cameras (for linguistic analysis)

EALTA Conference

Haug et al., HfH

13

SMILE

What is a “correct” sign? • Data collection of L1 (N = 11) and L2 (N = 19) users of DSGS • Lexical signs sampled from existing DSGS teaching materials (N = 110) • Annotation in iLex • Results: • Six different categories • Four categories were classified as “correct” and two as “incorrect” • Categories served as basis to inform the criterion of correctness Ebling et al., 2018 EALTA Conference

Haug et al., HfH

14

SMILE

Categ ories

Same lexem as target sign?

Same meaning?

Same form?

Test decision

Examples

1

yes

yes

yes

correct

SPRACHE_1A

2

yes

yes

slightly different

correct, accepted variant

PRÜFUNG

3

yes

yes

no

incorrect

NOCHMAL

4

yes

slightly different

slightly different

correct, morphophonemic/ semantic variants

TELEFONIEREN, STRASSE

5

no

yes

no

correct, dialect variants

DANN

6

no

no

no

incorrect

BAUM_1A (Gebärde für anderes Konzept)

Ebling, 2017 EALTA Conference

Haug et al., HfH

15

Current status of assessment and feedback system • Improvement of accuracy • Human and machine rating • Demo event in October 2018 in Zurich, Switzerland

EALTA Conference

Haug et al., HfH

16

Methodological challenges • State of research • Acquisition studies of L1 and L2 users • Available resources, e.g., reference grammar, corpus • Frequency list for vocabulary test • No standardized language (i.e., variability) • «Tradition» of rating • Training of raters

EALTA Conference

Haug et al., HfH

17

Thanks for your attention!

Contact: [email protected]

EALTA Conference

Haug et al., HfH

18

References Boyes Braem, P. (1995). Einführung in die Gebärdensprache und ihre Erforschung (Vol. 11). Hamburg: Signum-Verlag. Boyes Braem, P., Haug, T. & Shores, P. (2012). Gebärdenspracharbeit in der Schweiz: Rückblick und Ausblick. Das Zeichen, 90, S. 58–74. Ebling, S. (2017). Wahlmodul «Gebärdensprachforschung». Interkantonale Hochschule für Heilpädagogik Sarah Ebling, Necati Cihan Camgöz, Penny Boyes Braem, Katja Tissi, Sandra SidlerMiserez, Stephanie Stoll, Simon Hadfield, Tobias Haug, Richard Bowden, Sandrine Tornay, Marzieh Razavi, and Mathew Magimai-Doss. SMILE Swiss German Sign Language Dataset. In Proceedings of the 11th Language Resources and Evaluation Conference (LREC 2018). Mendez, O. (2018, Februar). SMILE sign language assessment: System integration demo. Presentation gehalten auf der SMILE meeting, Interkantonale Hochschule für Heilpädagogik, Zurich.

EALTA Conference

Haug et al., HfH

19