The challenges facing mobile VR

We’re finally diving deep into the virtual reality revolution, as some might put it, with hardware and software products aplenty on the market, and resources pouring in to spur on innovations. However, we’re more than a year on since major product launches in this space and we’re still waiting on that killer application to make virtual reality a mainstream success. While we wait, new developments continue to make virtual reality a more viable commercial option, but there are still a number of technical obstacles to overcome, particularly in the mobile VR space.

Limited power budget

The most obvious and well discussed challenge facing mobile virtual reality applications is the much more limited power budget and thermal constraints when compared to its desktop PC equivalent. Running intensive graphics applications from a battery means that lower power components and efficient use of energy is required to preserve battery life. In addition, the proximity of processing hardware to the wearer means that the thermal budget can’t be pushed any higher either. For comparison, mobile is typically operating within a sub-4 watt limit, while a desktop VR GPU can easily consume 150 watts or more.

It’s widely acknowledged that mobile VR isn’t going to match desktop hardware for raw power, but that doesn’t mean that consumers aren’t going to demand immersive 3D experiences at a crisp resolution and with high frame rates, despite the more limited power budget. Between watching 3D video, exploring 360 degree recreated locations, and even gaming, there are still plenty of use cases suited to mobile VR.

Looking back at your typical mobile SoC, this creates additional problems that are less often appreciated. Although mobile SoCs can pack in a decent octa-core CPU arrangement and some notable GPU power, it’s not possible to run these chips at full tilt, due to both the power consumption and thermal constraints mentioned previously. In reality, the CPU in a mobile VR instance wants to run for as little time as possible, freeing up the GPU to consume the bulk of the limited power budget. Not only does this limit the resources available for game logic, physics calculations, and even background mobile processes, but also puts a burden on essential VR tasks, such as draw calls for stereoscopic rendering.

The industry is already working on solutions for this, which don’t just apply to mobile. Multiview rendering is supported in OpenGL 3.0 and ES 3.0, and was developed by contributors from Oculus, Qualcomm, NVIDIA, Google, Epic, ARM, and Sony. Multiview allows for stereoscopic rendering with just a single draw call, rather than one for each view point, reducing the CPU requirements and also shrinking the GPU vertex job too. This technology can improve performance by between 40 and 50 percent. In the mobile space, Multiview is already supported by a number of ARM Mali and Qualcomm Adreno devices.

Another innovation expected to appear in upcoming mobile VR products is foveated rendering. Used in conjunction with eye-tracking technology, foveated rendering lightens the load on a GPU by only rendering the user’s exact focal point at full resolution and reducing the resolution of objects in the peripheral vision. The complements the human vision system nicely and can significantly reduce GPU load, thereby saving on power and/or freeing up more power for other CPU or GPU tasks.

Bandwidth and high-resolutions

While processing power is limited in mobile VR situations, the platform is still beholden to the same requirements as other virtual reality platforms, including the demands of low latency, high resolution display panels. Even those who have viewed VR displays that boasts a QHD (2560 x 1440) resolution or the Rift headset’s 1080×1200 resolution per eye will probably have been a little underwhelmed by the image clarity. Aliasing is especially problematic given that our eyes are so close to the screen, with edges appearing particularly rough looking or jagged during motion.

The brute force solution is to increase the display resolution, with 4K being the next logical progression. However, devices need to maintain a high refresh rate regardless of resolution, with 60Hz considered the minimum but 90 or even 120Hz being much more preferable. This puts a large burden on system memory, with anywhere from two to eight times more than today’s devices. Memory bandwidth is already more limited in mobile VR than it is in desktop products, which use faster dedicated graphics memory rather than a shared pool.

Possible solutions to save on graphics bandwidth include the use of compression technologies, such as ARM and AMD’s Adaptive Scalable Texture Compression (ASTC) standard or the lossless Ericsson Texture Compression format, both of which are official extensions of OpenGL and OpenGL ES. ASTC is also supported in hardware in ARM’s latest Mali GPUs, NVIDIA’s Kepler and Maxwell Tegra SoCs, and Intel’s latest integrated GPUs, and can save on more than 50 percent bandwidth in some scenarios versus the use of uncompressed textures.

Other techniques can be implemented too. The use of tessellation can create more detailed looking geometry from simpler objects, albeit by requiring some other substantial GPU resources. Deferred Rendering and Forward Pixel Kill can avoid rendering occluded pixels, while Binning/Tiling architectures can be used to split the image into smaller grids or tiles that are each rendered separately, all of which can save on bandwidth.

Alternatively, or preferably additionally, developers can make sacrifices to image quality in order to reduce the stress on system bandwidth. Geometry density can be sacrificed or more aggressive culling used to reduce the load, and vertex data resolution can be lowered to 16-bit, down from the traditionally used 32-bit accuracy. Many of these techniques are already being used in various mobile packages, and together they can help reduce the strain on bandwidth.

Not only is memory a major constraint in the mobile VR space, but it’s also a rather large consumer of power too, often equal to the consumption of the CPU or GPU. By making savings on memory bandwidth and usage, portable virtual reality solutions should see longer battery life.

Low latency and display panels

Speaking of latency issues, so far we’ve only seen VR headsets sporting OLED display panels and this is mostly due to fast pixel switching times of under a millisecond. Historically, LCD has been associated with ghosting issues for with very fast refresh rates, making them rather unsuitable for VR. However, very high resolution LCD panels are still cheaper to produce than OLED equivalents, so switching to this technology could help bring the price of VR headsets down to more affordable levels.

Displays are a particularly important part in the overall latency of a virtual reality system, often making the difference between a seemless and a sub-par experience. In an ideal system, motion-to-photon latency – the time taken between moving your head and the display responding – should be less than 20 milliseconds. Clearly a 50ms display is no good here. Ideally panels need to be sub-5ms in order to accommodate sensor and processing latency as well.

Currently there’s a cost performance trade-off that favours OLED, but this could soon change. LCD panels with support for higher refresh rates and low black-to-white response times which make use of cutting edge techniques, such as blinking back lights, could fit the bill nicely. Japan Display showed off just such a panel last year, and we may see other manufacturers announce similar technologies too.

Audio and sensors

While much of the common virtual reality topics revolve around image quality, immersive VR also requires high resolution, spatially accurate 3D audio and low latency sensors. In the mobile realm, this all has to be done within the same restricted power budget that affects the CPU, GPU, and memory, which presents further challenges.

We’ve touched on the sensor latency issues previously, in which a movement must be registered and processed as part of the sub 20ms motion-to-photon latency limit. When we consider that VR headsets use 6 degrees of motion – rotation and yaw in each of the X, Y, and Z axis – plus new technologies such as eye tracking, there’s a considerable amount of constant data to collect and process, all with minimal latency.

Solutions to keep this latency as low as possible pretty much requires an end-to-end approach, with hardware and software both able to perform these tasks in parallel. Fortunately for mobile devices, the use of dedicated low power sensor processors and always-on technology is very common, and these run at fairly low power.

For audio, 3D position is a technique long used for gaming and such, but the use of a head related transfer function (HRTF) and convolution reverb processing, which are required for realistic sounding source positioning, are quite processor intensive tasks. Although these can be performed on the CPU, a dedicated digital signal processor (DSD) can perform these types of processes much more efficiently, both in terms of processing time and also power.

Combining these features with the graphics and display requirements we’ve already mentioned, it’s clear that the use of multiple specialized processors is the most efficient way to meet these needs. We’ve seen Qualcomm make much of the heterogeneous compute capability of its flagship and most recent mid-tier Snapdragon mobile platforms, which combine a variety of processing units into a single package with capabilities that lend nicely towards meeting many of these mobile VR needs. We will likely see type of packages power in a number of mobile VR products, including standalone portable hardware.

Developers and software

Finally, none of these hardware advancements are much good without software suites, game engines, and SDKs to support developers. After all, we can’t have every developer reinventing the wheel for every application. Keeping development costs low and speeds as fast as possible is key if we’re going to see a wide range of applications.

SDKs in particular are essential for implementing key VR processing tasks, such as Asynchronous Timewarp, lens distortion correction, and stereoscopic rendering. Not to mention power, thermal, and processing management in heterogeneous hardware setups.

Fortunately all of the major hardware platform manufacturers offer SDKs to developers, although the market is a rather fragmented resulting in a lack of cross-platform support. For example, Google has its VR SDK for Android and a dedicated SDK for the popular Unity engine, while Oculus has its Mobile SDK built in conjunction with Samsung for the Gear VR. Importantly, the Khronos group recently unveiled its OpenXR initiative which is aiming to provide an API to cover all of the major platforms at both the device and application level layers, in order to facilitate easier cross platform development. OpenXR could see support in its first virtual reality device sometime before 2018.

Wrap Up

Despite some issues, technology is under development, and to some extent already here, that makes mobile virtual reality workable for a number of applications. Mobile VR also has a number of benefits that simply don’t apply to desktop equivalents, which will continue to make it a platform worthy of investment and intrigue. The portability factor makes mobile VR a compelling platform for multimedia experiences and even light gaming, without the need for wires connected up to a more powerful PC.

Furthermore, the sheer number of mobile devices on the market that are increasingly equipped with virtual reality capabilities makes it the platform of choice for reaching the largest target audience. If virtual reality is to become a mainstream platform it needs users, and mobile is the biggest user base around to tap.