As one of the biggest and most complex tech projects ever for automatic Sign Language recognition and translation, we keep receiving many questions and media inquiries. Ever since gesture recognition was invented, sign language recognition has been one of the enlisted motivations. Even if it is a camera based system or a glove with physical sensors, it is a quite natural idea to use those technologies to enable communication between Deaf and hearing folks.
Calling a system that is able to recognize a set of individual hand signs as a sign language translator is the true evidence of not understanding the challenge. It is venial for a student project – see the remarkable traction of Enable Talk gloves to win the Imagine Cup back in 2012. However, we think that there are many factors that are essential when pursuing a real mission, to provide real value for the Deaf community by respecting Deaf culture.
Here, we highlight the most important three:
- Sentence level translation: just like spoken languages, sign languages are individual languages with their own linguistic structures. A system that recognizes separate signs one-by-one could only provide a translation in a situation where SEE (Signed Exact English) is provided. The whole translation and the used words might be completely changed by a sign that comes later in the sentence, like when the signed sentence turns out to be a question. Furthermore, forcing the signer to stop and wait for the result after every sign could result in poor user experience.
- Tracking the face: all non-manual markers play a significant role in signing. Face is especially important as it supplies numerous sign locations and also because of facial expressions carrying grammatical meanings which need to be considered. No project that disregard the face can ever result in a real sign language translation.
- Engagement of the Deaf Community: There is one true evidence that separates dreamers from real challengers: no good product can be developed without the involvement of the ultimate end users, and this is especially true in this special niche. If there are deaf members in a project, the team will soon realize the two factors above among many others. We believe that the involvement of deaf team members is the most important success factor in all.
See our latest media coverage here. Any new in-depth scientific research in this field could add to the technology development, but we suggest to be conservative with the estimated impact on the community unless you have considered all those factors above.