Hear me sign: Sign language translator - development and test...Title [Specialty Track (Specialty...
Transcript of Hear me sign: Sign language translator - development and test...Title [Specialty Track (Specialty...
Copyright 2019 Accenture. All rights reserved.
Hear me sign:Sign language translator -development and testDaria LashkevichAccenture. Riga, Latvia
TAPOST 2019
Copyright 2019 Accenture. All rights reserved.
How you can understand that I need
your urgent help if you don’t know
British sign language?
Copyright 2019 Accenture. All rights reserved.
BusinessHear me sign
BY 2050
900M HEARING LOSS
AND 70M MUTE
PEOPLE
130 DIFFERENT
SIGN LANGUAGES
A LOT OF TIME
LOST
Copyright 2019 Accenture. All rights reserved.
Sign language Basic sentence structure
Source https://www.youtube.com/watch?v=glJvYftJZM4
Copyright 2019 Accenture. All rights reserved.
TaskScope
Create AI application
which can recognize
British Sign Language
alphabet
3 AI Engineers 4 weeks
Copyright 2019 Accenture. All rights reserved.
SOLUTION ArchitectureProcess flow
frontend
video
stream
each frame is
analysed by
skeleton extractor
JSON pose
coordinates
LSTM
module 1
• Signs recognition
• Translation
textJSON
text
text to
speechvoice
backend
AI Pose
recognition
Copyright 2019 Accenture. All rights reserved.
Landscape reviewPossible solutions
IOT GLOVES FOR HAND
TRACKING
CAMERA ONLYIMAGE + DEPTH (E.G.
LEAP MOTION)
https://www.youtube.com/watch?v=oZ_53T2jBGg https://github.com/CMU-Perceptual-Computing-Lab/openposehttps://cdn-blog.adafruit.com/uploads/2016/05/3059616-poster-p-2-these-
students-built-a-glove-that-translates-sign-language-into-english.jpg
Copyright 2019 Accenture. All rights reserved.
Person recognitionBody, face and face key points
Copyright 2019 Accenture. All rights reserved.
Desktop prototype Hear me sign
Copyright 2019 Accenture. All rights reserved.
SolutionHear me signHow to address the problem?
A mobile application translating between sign
language / audio:
Copyright 2019 Accenture. All rights reserved.
SolutionHear me sign
Copyright 2019 Accenture. All rights reserved.
Creating dataset: first problems
• Learning BSL
• Translations problems
• Uncountable quantity
of cases to show sign
Copyright 2019 Accenture. All rights reserved.
Creating dataset: first problems
• Left and right handed signs
Copyright 2019 Accenture. All rights reserved.
Creating dataset: first problems
• Gender differences for coordinate defining
• Not all needed coordinates can be defined
Copyright 2019 Accenture. All rights reserved.
Creating dataset: solutions
• Letters separately (Alphabet first)
• Based on pictures first, after on videos
• Equal quantity of men and women for dataset
• The highest camera quality
• Dataset filtration
Copyright 2019 Accenture. All rights reserved.
Creating dataset: solutions
• All different cases also should be shown in data set
Copyright 2019 Accenture. All rights reserved.
Creating dataset: solutions
• All negative cases also should be shown in data set
Copyright 2019 Accenture. All rights reserved.
Model training – how to test
• Different backgrounds
• Different cameras (desktop and mobile)
• Different people
• (doesn’t matter men or women)
• Testing with British sign language natives
Copyright 2019 Accenture. All rights reserved.
Software/Hardware requirements
Not so easy to test such heavy components:
• GPU NVIDIA driver
• Caffe framework
• Big capacity for video implementation
• Client – server Application
Copyright 2019 Accenture. All rights reserved.
Solution integration: testing
• Testing integration: client – server
• More then one person in the camera –
• who should be recognized
• Unknown signs
• Connection delays
• Fast signs show
• Hard outdoor conditions: train, rain, fog
Copyright 2019 Accenture. All rights reserved.
Recognition
• The second place winner in Global Technology Innovation
Contest 2018 Europe ATC
• The conference SQA-Days EU-1 2019 invited participant
Copyright 2019 Accenture. All rights reserved.
Demo indoor
Copyright 2019 Accenture. All rights reserved.
Demo outdoor
Copyright 2019 Accenture. All rights reserved.
What next?
mobile only wearables + mobileSign language avatar
(inversed translator)
Copyright 2019 Accenture. All rights reserved.
Sign language avatar(inversed translator) - ISL
Copyright 2019 Accenture. All rights reserved.
Sign language avatar(inversed translator) - ISL
Copyright 2019 Accenture. All rights reserved.
Sign language avatar(inversed translator) - ISL
Copyright 2019 Accenture. All rights reserved.
CONTACTS
Accenture RigaLiquid Studio
DARIA LASHKEVICH
Riga Liquid Studio
Innovation Test Team Lead
+371 23202147
Visit us @ Latvia.Accenture.lv