Algorithm extracts audio from silent video using vibrations

It feels almost like a scene straight out of a spy movie or crime procedural show. You only have a video a of man talking with his back to the camera. No audio to tell you clearly what he's saying and there are no lips to read to give you a clue. But by observing just the movement of the leaves of a plant nearby, you are able to reconstruct what the man is saying, stop his evil plot, defeat the bad guy, and save the day.

OK, so maybe it doesn't work exactly like that or even that well, but that is exactly what a group of researchers from MIT, Microsoft, and Adobe are trying to accomplish. They have developed algorithms that observe an object's vibrations too subtle to be detectable by the naked eye and then analyzes that "visual signal" to reproduce the sound that originally produced the vibrations.

Most of us already know how sound and vibration are related. However, we mostly take for granted that these sounds, like a talking voice, music, or anything for that matter, would cause objects that they hit to vibrate as well. These vibrations are so miniscule that we cannot see them for ourselves. The vibration is so small that it would only represent a shift in position five thousandths the size of a pixel. The researchers had to employ a high-speed camera to properly record and analyze those movements.

But the researchers later on discovered that they may not actually need such expensive equipment after all. They tried exploiting what most consider a bug in the rolling system type of shutter used in consumer digital cameras to try to get the same effect. While not as accurate as high-speed cameras, the artifacts produced by this bug was still enough to allow the researchers to recover sound at several frequencies higher than the video's original framerate.

But more than just recovering sound from video, research paper first author Abe Davis and company are equally excited about the potentials of this technology for the imaging field. Observing vibrations apparently goes both ways. It not only gives information about the sound that produced the vibration but also gives information about the nature of the vibrating object itself, as different materials vibrate differently. This could give scientists of the future the tool to study the material and structure of an object simply by letting a short burst of sound go off, or maybe even just by shouting at it.