MIT Creates A Method Of Seeing A Video Based On Cast Shadows

Scientists from MIT have created something very interesting. They have developed a method of reconstructing a hidden video from the shadows and reflections that can be observed in a pile of clutter. The tech allows a video camera running in the room to reconstruct a video coming from an unseen corner of the room, even if the video falls outside the camera field of view.The team uses an algorithm to observe the shadow and geometry in video and predict the way the light travels in a scene, known as light transport, to recreate that video. The system can estimate the hidden video from the observed shadows and can reconstruct the silhouette of a live-action performance.

The tech has the potential to be useful in a myriad of situations. It could be used to improve self-driving cars to allow them to understand better what's behind corners. Nursing homes could use the tech to enhance the safety of the residents in the home.

MIT's technique is passive and needs no lasers or other interventions to the scene. Currently, the video takes about two hours to process. The team feels that eventually it could be used to reconstruct scenes that aren't in a traditional line of site for multiple applications. The team uses subtle, indirect lighting cues like shadows and highlights from the clutter in the observed area.

The clutter is said to act a bit like a pinhole camera by blocking some light rays but allowing others to pass through. Those paint an image of the surroundings wherever they hit. The algorithm the team developed can make sense of those lighting clues and recover a video that humans can understand, revealing the activity in the hidden scene. The team wants in the future to improve the resolution of the video and to test the video in an uncontrolled environment.