MIT’s New Visual Deprojection Technique Can Turn Images Into Video

By collapsing data comprised of multiple dimensions into just one or two dimensions, much of the information is lost and what remains can be considered somewhat of a shadow of the original. When visual information is collapsed into a lower dimension in this way, MIT calls the product a ‘projection.’ However, by establishing a relationship between higher dimension source data and lower dimension projections, researchers at MIT determined that it is possible essentially ‘deproject’ a projection—by reconstructing the data that is lost during projection.

MIT Is Turning a Shadow Into a Man

To accomplish this feat, MIT developed a neural network that was tasked with learning how to “match low-dimensional projections to their original high-dimensional images and videos.” This essentially means that the neural network developed an archive of data that could then be applied to new images, allowing it to reconstruct the original data that was lost in a projection. As an example application for the technology, the MIT researchers were able to generate accurate video frames of a person walking by using only footage taken from MIT’s recently invented ‘corner cameras.’ These corner cameras are essentially able to detect movement around corners and produce projections that indicate a person’s speed and trajectory as squiggly lines. In another experiment, the researchers produced 35 videos of 30 people walking in a specified area. The frames of these videos were collapsed into projections and used these to train the model. Following this, the model was tasked with deprojecting 6 previously unseen projections, where it was reportedly able to generate 24 frames of the person’s gait, in which both the person’s size and leg positions were accurate.

The Ridiculous Becomes Reality—Almost

The technique is reminiscent of the much-maligned ‘Zoom and Enhance’ trope that was made famous as a technology-based crime-solving technique used on fictional shows such as CSI and CSI: Miami. During a Zoom and Enhance sequence, investigators would be able to extract seemingly ludicrous amounts of information from a blurry or distant photograph, such as being able to enhance a blurry, low resolution of a car at distance to such a degree that a high-resolution image of its number plate could be obtained. However, although it is unlikely that the visual deprojection model will be able to match some of the feats displayed on some fictional television shows and films, it could have quite a significant impact on the medical industry. For example, the technique could be adapted to turn 2D medical images from X-rays into something resembling a 3D CT scan, drastically reducing costs while speeding up the diagnostic process. “If we recover that lost dimension, it can have a lot of important applications,” said Guha Balakrishnan, a postdoc who helped pioneer the new technique. What immediate applications can you think of for MIT’s new data restoration technique? Let us know your thoughts in the comments below.

---

Images are courtesy of Shutterstock.