What is DeepFovea?

Facebook announced that it is releasing DeepFovea, a new state-of-the-art foveate rendering using AI technology.

Engineers at the Facebook Reality Labs have come up with an imagery assistant for creating a “plausible peripheral image” rather than the actual peripheral imagery, which in reality is hazy and unfocused as the gaze is focused on something else. This image rendering is called Foveated Reconstruction, which is done by a 14 times compression of pixels on the RGB (Red, blue, Green) video without compromising on the quality, and which is realistic and gaze-contingent.

DeepFovea is one of the first generative adversarial network (GAN) able to produce natural video sequences, say the facebook developers of the technology. “DeepFovea can decrease the amount of compute resources needed for rendering by as much as 10-14x while any image differences remain imperceptible to the human eye,” according to Facebook.

Foveation means angling an eye to focus vision. When human eyes focus on an object they see that object directly and tend to exclude the peripheral region where objects are registered hazily. DeepFovea uses recent advances in generative adversial networks (GANs), a deep machine learning system that predicts outcomes based on observations that can create the missing peripheral details by generating content that is perceptually consistent. 

DeepFovea is one of the first GAN able to produce natural video sequences.

An immersive AR/VR experience demands precise and accurate eye tracking performance. Cameras inside VR headsets track the movement and position of your pupils, enabling the GPU to know where to focus to render quality images.  Foveated rendering are an essential part of virtual reality (VR) and augmented reality (AR). Foveated renderings can help save 30 percent power in the GPU, which can be used to improve details within the areas viewed by the pupil.

DeepFovea delivers higher quality periphery renderings using greater efficiency compared to what is now available on standard faceted renderings.

Facebook’s claims of using 14 times reduced renderings will enable it to deliver real-time, no-lag videos using gaze detection based on deep machine and AI processes. This will help in building better VR headsets.

The present VR and AR headsets are bulky and impractical with its high power consumption for 3D mapping. The foveated reconstruction technology used in DeepFovea will greatly increase rendering efficiency in the present VR and AR headsets. Facebook first hinted about their work on DeepFovea last year at the Oculus Connect 5. They talked of coming up, within the next five years, with high-end VR and AR headsets. They said foveation combined with deep learning and gaze tracking can be used to deliver high-resolution headsets.

Facebook is releasing a sample version of the network architecture of DeepFovea for researchers, VR engineers, and graphics engineers. The company plans to present the research paper is at Siggraph Asia tonight, and will make the samples available after the event.