• Student

Sleeve enables visually impaired to communicate non-verbally

A conversation is more than an exchange of words. What your conversation partner really means is made clear by their gestures, whether they look you in the eye or not, whether they’re smiling, and other non-verbal communication. Team HART hopes to help people who can’t see to pick up such visual signs using a smart sleeve. It is one of the new applications of the device, which was originally developed to transform spoken text into vibrations. Yesterday the team presented the latest updates to the smart sleeve at the Auditorium.

by
photo Sem Jordaan

Ever since it was founded in 2020, the goal of Team HART (Human Augmentation Research & Technology) has been to put human augmentation on the map. To this end, the team presented its first proof of concept last year: a smart sleeve that uses vibrational patterns to enable hearing-impaired people to feel what their conversation partner is saying and that can simultaneously translate the conversation. The past year the students also started to look into other applications for the sleeve, including transforming non-verbal communication into vibrational pattens.

Subtle signals

There’s no completely finished product yet, but all of the separate components are working, says HART member and Applied Physics master’s student Juliette Passariello-Jansen. “The AI model looks for subtle signals and translates these into vibrational patterns.” It doesn’t only notice if someone is nodding or smiling, but also whether they’re actually paying attention to the conversation, Computer Science master’s student Marco Pleket adds. “Knowing who’s the focus of attention is useful for blind people when talking to several people, for instance.” But the application can also be used by people that can see, says Passariello-Jansen. “Take a lecturer who is teaching an online class. The sleeve can detect if students are still paying attention or if an explanation may have left some of them a bit confused.” A research group from Open University helped the students by modifying an existing AI model for facial recognition, which makes these applications possible.

Marco Pleket and Juliette Passariello-Jansen. Photo: Dillon Gonsalves

The students themselves tested how the vibrational patterns feel on one’s arm. 24 motors provide plenty of options to create patterns. “They can vibrate in a different order, or just on one side,” Pleket says. “We can also add an animation that makes the vibration crawl up on your arm.” For the speech application, 39 patterns have already been devised to correspond to phonemes. When interpreting social cues, different vibrational patterns are assigned to different situations and emotions. There is a limit to how many distinguishable patterns you can make, says Passariello-Jansen. “We are working on an AI model that can identify such patterns for us.” Now all that’s left for a well-functioning product is to have the sleeve, including the camera glasses, tested by visually impaired people. Pleket: “The inspiration for this application came from someone who is blind, but now we have to see if it can actually help visually impaired people."

Other applications

In the meantime, the team is also investigating other applications. Might the sleeve, for instance, contribute to our spatial awareness? During a demo at Dutch Design Week, the team had visitors use the sleeve to feel the position of a little square on a screen in front of them, Pleket tells us. “It allowed them to point to where the square was with their eyes closed.” The team is also working on using the sleeve in games. “When something comes at you in a game, for instance, you will generally see this on your screen. But you could also enable people to feel it by having the sleeve vibrate.”

The motors in the sleeve. Photo | Sem Jordaan

Clearly, then, the sleeve is not solely intended to replace missing senses. The team is committed to human augmentation in its broadest definition, Passariello-Jansen says, and to giving everyone the chance to experience the world in a new way. “With our first speech application, we wanted to show that it was possible to replace a sense. Now we’ve entered a phase in which we are actually making the sleeve user-friendly, so it can take a knock. It’s also way more compact than it used to be.” The more comfortably people can wear the sleeve, the bigger the chance they’ll actually want to use it, the students believe.

Share this article