SoundWatch smartwatch app alerts the deaf and hearing-impaired of environmental sounds

Smartwatches have become very common and can offer people a private way of getting notifications about their surroundings, health alerts, phone calls, or package deliveries. Researchers at the University of Washington have developed a new smartwatch app called SoundWatch aimed at the deaf and hard-of-hearing. Smartwatches with the app can pick up the sounds a user is interested in, such as a siren or microwave beeping and identify it and alert the user.The technology is meant to give people a way to experience sounds that require actions, like getting food for the microwave when it beeps. SoundWatch can also help the person feel more connected to the world. Lead author of the research paper Dhruv Jain says that he uses the watch prototypes to notice birds chirping and waterfall sounds when he is hiking. He said that the app allows them to feel present in nature.

When starting the project for the team, the goal was to develop an app that will allow deaf and hard of hearing people know what was going on around their home. The first system the researchers came up with was called HomeSound and relied on Microsoft Surface tablets placed around the home operating as a network of interconnected displays. Each of the displays had a basic floor plan of the home and alerted users to a sound and its source.

The displays also show the sound's waveforms to help users identify them and stored the history of all sounds the user could've missed when not at home. That system was trialed in the homes of six deaf or hard of hearing participants over three weeks. After receiving feedback, the second prototype used machine learning to classify sounds in real-time. The team created a dataset of over 31 hours of 19 common home-related sounds, including a dog bark, cat meow, baby crying, and door knock.

Researchers eventually made their way to the smartwatch system allowing users to get sound alerts wherever they are, even if they are at home. The challenge is the limited storage and processing ability of the smartwatch. That meant researchers needed a system that was easy on the battery but fast and accurate. The answer was a compressed version of the HomeSound classifier. The system sends the sound from the wearable to the user's smartphone for classification.