GESPIN2011 presentationusers.jyu.fi/~tojantun/3BatS/Presentations_files/... · and H. SLOETJES,...
Transcript of GESPIN2011 presentationusers.jyu.fi/~tojantun/3BatS/Presentations_files/... · and H. SLOETJES,...
2.9.2011
1
METHOD FOR VISUALISATION AND ANALYSIS OF HAND AND HEAD MOVEMENTS IN SIGN LANGUAGE VIDEO Tommi Jantunen University of Jyväskylä, Sign Language Centre, Finland
Matti Karppa, Markus Koskela, Jorma Laaksonen & Ville Viitaniemi Aalto University School of Science, Department of Information and Computer Science, Espoo, Finland
GESPIN 2011, Bielefeld (Germany), 6 September 2011
Demo - output of the method imported into ELAN
URL of ELAN: http://www.lat-mpi.eu/tools/elan/ (Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands); See O. CRASBORN, and H. SLOETJES, ”Enhanced ELAN functionality for sign language corpora”. In Proceedings of 3rd Workshop on the Representation and Processing of Sign Languages at 6th LREC conference, Marrakech, Morocco, May–Jun 2008, pp. 39–43.
Processing steps of the method
1. Face detection 2. Detection of skin-coloured regions
3. Skin blob detection
4. Corner point detection 5. Feature point tracking 6. Fitting of active shape models
Demo - feature point tracking & active shape models
J. SHI, and C. TOMASI, ”Good features to track”. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR ’94), Jun 1994, pp. 593–600. T. F. COOTES, D. H. COOPER, C. J. TAYLOR, and J. GRAHAM, ”Active Shape Models - Their training and application”. Computer Vision and Image Understanding, vol. 61, no. 1, pp. 38–59, Jan 1995.
'…how the deaf clubs would func2on without these posts?'
'…but does this mean that deaf clubs are old fashioned?'
Types of motion information we are able to calculate per a tracked body part (both hands and the head)
Feature point tracking only • The number of tracked motion points
Feature point tracking & active shape modeling • The amount (& direction) of horizontal and vertical motion
• Length of the velocity vector (i.e. speed) • Length of the acceleration vector (currently fpt only!)
Results visualised in ELAN and used for annotation
• Tracked data in ELAN helps the annotator to align annotations more precicely
2.9.2011
2
… linguistic analysis
• Example question: What are the phonetic characteristics of headshakes in FinSL?
... and kinematic analysis
• Kinematic analysis without laboratory settings and MoCap equipment
Cf. kinematic analysis of dominant hand speed with MoCap data
T. JANTUNEN, ”Signs and transitions: Do they differ phonetically and does it matter?”. Sign Language Studies, vol. 13, no. 2, 2013, (forthcoming).
Demo - comparison with MoCap data (speed)
Conclusion • The method is already a feasible tool in the annotation
and analysis of sign language and gesture
• Plans for the future • More detailed modeling of articulators (e.g. hands and arms, and
facial gestures) • Development of a more automated mechanism for the annotation
of sign language corpora
• Further comparison with MoCap data
• Exact plans for sharing the software have not yet been made but we wish people to use it (when it's finished)
Thank you! • My colleagues at Aalto University School of Science • Päivi Rainò at HUMAK University of Applied Sciences • Danny De Weerdt, Birgitta Burger, Tuija Wainio at JyU
• For more information, see CoBaSiL home page at http://research.ics.tkk.fi/cbir/cobasil/
… and 3BatS home page at http://users.jyu.fi/~tojantun/3BatS/Home.html
• Please also see our paper in the proceedings for further information