EALTA Conference Technology-based language assessment: Benefits and challenges
Automatic sign language recognition for sign language assessment May 25-27, 2018
Tobias Haug, Sarah Ebling, Penny Boyes Braem, Katja Tissi, & Sandra Sidler-Miserez
Overview • • • •
Structure of sign languages Swiss context SMILE project Methodological challenges
EALTA Conference
Haug et al., HfH
2
Structure of sign languages •
•
Manual components • Handshape • Location • Movement • Hand orientation Non-manual components • Eye brows • Forehead • Cheeks • Eye gaze • Mouth • Upper torso
EALTA Conference
Haug et al., HfH
Source: SLAS 2015
3
Structure of sign languages
Handshapes [1]
Overview of handshapes in Swiss German Sign Language (DSGS)
Boyes Braem, 1995 EALTA Conference
Haug et al., HfH
4
Structure of sign languages
Minimal pairs
DSGS-Gebärden SAGEN („say“) und FRAGEN („ask“) Boyes Braem, 1995 EALTA Conference
Haug et al., HfH
5
Swiss context Three national sign languages according to the three main (spoken) language regions • Swiss German Sign Language (Deutschschweizerische Gebärdensprache, DSGS) • ca. 5,500 deaf DSGS users • French Sign Language (Langue des Signes Française, LSF) • ca. 1,700 deaf LSF users • Italian Sign Language (Lingua Italiana dei Segni, LIS) • ca. 300 deaf LIS users Boyes Braem, Haug & Shores, 2012 EALTA Conference
Haug et al., HfH
6
SMILE Scalable Multimodal Sign Language Technology for Sign Language Learning and Assessment
EALTA Conference
Haug et al., HfH
7
SMILE
Project partners • Idiap Research Institute, CH • Interkantonale Hochschule für Heilpädagogik (HfH), CH • University of Surrey, GB • Swiss National Science Foundation (SNSF): • Three-year Sinergia project
EALTA Conference
Haug et al., HfH
8
SMILE
Project team HfH • Sarah Ebling, project coordinator HfH • Penny Boyes Braem • Katja Tissi • Sandra Sidler-Miserez • Tobias Haug
EALTA Conference
Haug et al., HfH
9
SMILE
Project goal
Develop an automatic sign language recognition and assessment system to be used in a DSGS vocabulary L1/L2 translation test
EALTA Conference
Haug et al., HfH
10
SMILE
Automatic sign language recognition
Recognizing the form of a sign
EALTA Conference
Comparing it with a "reference"
Haug et al., HfH
Feedback to the learner
11
SMILE
Automatic sign language recognition
Recognizing the form of a sign
EALTA Conference
Comparing it with a "reference"
Haug et al., HfH
Feedback to the learner
12
SMILE
Cameras • •
1 Kinect (skeleton and depth sensor), HD resolution, 30 frames/second 2 GoPros • •
•
1x with high framerate (240 frames/second, HD resolution) 1x with high resolution (4K, 30 frames/second)
3 HD cameras (for linguistic analysis)
EALTA Conference
Haug et al., HfH
13
SMILE
What is a “correct” sign? • Data collection of L1 (N = 11) and L2 (N = 19) users of DSGS • Lexical signs sampled from existing DSGS teaching materials (N = 110) • Annotation in iLex • Results: • Six different categories • Four categories were classified as “correct” and two as “incorrect” • Categories served as basis to inform the criterion of correctness Ebling et al., 2018 EALTA Conference
Haug et al., HfH
14
SMILE
Categ ories
Same lexem as target sign?
Same meaning?
Same form?
Test decision
Examples
1
yes
yes
yes
correct
SPRACHE_1A
2
yes
yes
slightly different
correct, accepted variant
PRÜFUNG
3
yes
yes
no
incorrect
NOCHMAL
4
yes
slightly different
slightly different
correct, morphophonemic/ semantic variants
TELEFONIEREN, STRASSE
5
no
yes
no
correct, dialect variants
DANN
6
no
no
no
incorrect
BAUM_1A (Gebärde für anderes Konzept)
Ebling, 2017 EALTA Conference
Haug et al., HfH
15
Current status of assessment and feedback system • Improvement of accuracy • Human and machine rating • Demo event in October 2018 in Zurich, Switzerland
EALTA Conference
Haug et al., HfH
16
Methodological challenges • State of research • Acquisition studies of L1 and L2 users • Available resources, e.g., reference grammar, corpus • Frequency list for vocabulary test • No standardized language (i.e., variability) • «Tradition» of rating • Training of raters
EALTA Conference
Haug et al., HfH
17
Thanks for your attention!
Contact:
[email protected]
EALTA Conference
Haug et al., HfH
18
References Boyes Braem, P. (1995). Einführung in die Gebärdensprache und ihre Erforschung (Vol. 11). Hamburg: Signum-Verlag. Boyes Braem, P., Haug, T. & Shores, P. (2012). Gebärdenspracharbeit in der Schweiz: Rückblick und Ausblick. Das Zeichen, 90, S. 58–74. Ebling, S. (2017). Wahlmodul «Gebärdensprachforschung». Interkantonale Hochschule für Heilpädagogik Sarah Ebling, Necati Cihan Camgöz, Penny Boyes Braem, Katja Tissi, Sandra SidlerMiserez, Stephanie Stoll, Simon Hadfield, Tobias Haug, Richard Bowden, Sandrine Tornay, Marzieh Razavi, and Mathew Magimai-Doss. SMILE Swiss German Sign Language Dataset. In Proceedings of the 11th Language Resources and Evaluation Conference (LREC 2018). Mendez, O. (2018, Februar). SMILE sign language assessment: System integration demo. Presentation gehalten auf der SMILE meeting, Interkantonale Hochschule für Heilpädagogik, Zurich.
EALTA Conference
Haug et al., HfH
19