New AI systems understand sign language and allow communication with hearing impaired or deaf people


Technologies are developing very quickly, and in almost all areas, including AI. Now platforms are gradually beginning to appear that can improve the quality of life of people with various health problems – vision, nervous system, hearing and speech.

Developers create programs and services that help visually impaired people. For example, thanks to Seeing AI, visually impaired people can easily read mail, placing correspondence under the smartphone’s camera. However, in most cases, the easiest solution is to solve some of the problems of people with hearing and speech problems. Recently, developers from the United States presented an AI system that makes it possible to understand sign language and communicate with deaf and dumb people. It also allows people with hearing and speech problems to communicate with each other or with those who have no hearing or speech problems. There are other projects as well. All this is under the cut.

What is this device?

This is a smart device, which helps deaf people communicate without problems, both with those who can hear well, and with each other. The device recognizes sign language and translates to text and speech. In addition, the device is capable of recognizing speech and translating it into text displayed on the LCD screens of the glasses. The device is a system of glasses with headphones. The glasses are equipped with a camera and, as mentioned above, an LCD screen. Plus there is an SD card slot.

Glasses work as an individual sign language translation system, converting sign language into text and audio. As for the audio, it is amplified for those users who can hear, but poorly. The sound volume in the headphones is adjusted according to the severity of the hearing loss of the user.

The sensors on the front of the device recognize the sign language. They track the three-dimensional coordinates of the hands, converting the movement into a digital signal that is transmitted to the microcontroller. Inside is a database of different gestures and their meanings.

The microcontroller also includes a CPU and an ADC. Analog data is converted to digital signals, which are then matched against database templates. As a result of the device’s operation, the sign language is converted into ordinary speech, which sounds in the headphones. The system is just a prototype of a device for understanding sign language. Such a device can be extremely useful for parents of deaf or deaf-mute children, parents who can hear well and can speak without problems. In most cases, it is difficult for people without speech and hearing problems to learn all the nuances of sign language, so communication problems arise.

Moreover, people who are hard of hearing find themselves in an extremely disadvantageous situation, being in an unfamiliar place. It is difficult for them to find out the way, ask about something else or just chat. According to statistics, only 2% of people with hearing and speech problems have everything they need to organize normal communication with others. All others, respectively, are limited to one degree or another in their ability to communicate with society.

How does the device work?

It is also capable of converting ordinary speech, however, not into sign language, but into text that is shown on the LCD screen of the glasses, so that a deaf-mute person can understand what others are saying to him. In addition, if the user is able to hear even to a small extent, he uses headphones through which sound amplified by a special module passes. In general, such a device is an excellent option for creating universal systems that help establish a reliable communication channel.

Other projects

In addition to the already named system, which is still at the initial stage of preparation, there are other projects.

For example, scientists from the Institute for Management Problems. V.A. Trapeznikov RAS (IPU RAS) develop a unique a system for the hearing impaired and deaf people, which, using artificial intelligence, will be able to translate sign language into words, phrases and letters.

“We used an artificial neural cortex (a group of neurons responsible for making decisions – Izvestia) for gesture recognition. It can recognize static gestures. It already recognizes the dactyl alphabet. In the future, we plan to bring this development to an automatic sign language interpreter. will not be able, but in some cases it will help people with hearing problems, “- said the developer of the system, Mais Farhadov.

Another domestic development is the Charlie speech recognition device, which is also based on AI capabilities. According to statistics, about 13 million people in the Russian Federation suffer from hearing loss to one degree or another. Hearing loss ranks next to heart disease on the list of health problems.

The “Sensor-Tech” company has created a device that allows people with hearing impairments to study and work remotely. So, a lecturer at a university can put this device next to him, and special software will convert speech into text, sending information to a specialized site.

There are, of course, projects being developed by large companies. An example is IBM Corporation and its virtual avatars that translate speech into gestures. This project began to be implemented at the beginning of the century. Development took many months. The resulting system, despite its good efficiency, still has drawbacks. For example, people who know and use sign language still treat avatars with suspicion, because in comparison with humans, they cannot convey the expressiveness and accuracy of sign language. Sign language must have a very high precision in the execution of gestures, and every small movement can change their meanings. But be that as it may, the project continues to develop.

As a conclusion, it should be said that such projects are becoming more and more – now there are systems that people with hearing and speech problems can use. And in the near future there will be even more such systems, so that one problem for hearing impaired or deaf people will be less.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *