Arm Mali-D77 solves some of VR's performance problems

Hype for virtual reality has certainly died down over the past couple of years, owing to a combination of expensive hardware, so-so performance and motion sickness, and a lack of user content. The industry is currently stuck in a bit of a chicken vs. egg scenario, where a lack of consumers deters investment in high-end content. More powerful and cost-effective VR hardware for the masses will be needed to break the deadlock.

Arm is looking to overcome some of these technological hurdles with its first ever display processor (DPU) designed specifically for VR: the Mali-D77. In a nutshell, the Mali-D77 offloads common VR processing tasks from the GPU, freeing up resources for higher frame rates while also helping to reduce motion sickness.

Inside the Arm Mali-D77

Much of the Mali-D77 is based on 2017’s Mali-D71 for flagship smartphones and other high-end applications. It has the same compression decoder, layer scaling, HDR support, and color management units. However, the new design has been optimized to support 3K resolutions at 120fps, with support for 4K resolutions at up to 90fps.

The major changes come in the form of two brand new hardware acceleration units for VR applications. The Mali-D77 supports Lens Correction and Asynchronous Timewarp in hardware, rather than running these algorithms on the GPU. Arm estimates this can free up about 15 percent of GPU resources, which can be put towards boosting frame rates. Moving this load over to the DPU also provides a 40 percent bandwidth saving and 180mW of power per VR layer. Nice.

Lens Correction is required in VR headsets to offset the light curvature of the headset’s lenses. Barrel Distortion is applied to each rendered frame in order to offset the Pincushion Distortion effect of the lenses. Think of this as overcompensating or “reverse distortion” so that the lens distortion actually ends up displaying the correct image. Traditionally this is done on the GPU, taking up extra cycles and time. The Mali-D77 does this all on the DPU.

In addition, the Mali-D77 performs chromatic aberration correction using a similar reverse distortion method. With this applied, image colors will be displayed correctly across the entire lens, including in the corners where color separation distortion can occur.

What is Asynchronous Timewarp?

Lens Correction is rather self-explanatory, but Asynchronous Timewarp is a little more involved. Here, Arm is using the display processor to rotate, skew, and warp images to compensate for the wearer’s movements while mitigating any GPU or other display pipeline latency.

With current generation hardware, X, Y, Z axis movement tracking is updated in sync with the GPU, because the GPU has to render the change in view each time you move. With Asynchronous Timewarp, the two are no-longer-updated together. You can move your head in between GPU frame updates, and the Mali-D77 can warp the current frame to match your head movement.

This is a subtle effect, as it only lasts for a fraction of a second in between rendered frames and doesn’t eradicate the need for fast frame rate rendering. However, it greatly increases the fluidity and smoothness of movement and motion, as updates can occur even more frequently than the GPU frame rate. The disconnect between your body moving and seeing a visual update is the leading cause of motion sickness in VR, so the Mali-D77 can help a lot in this regard.

X, Y, Z axis movement data is fed directly to the Mali-D77 from the CPU, bypassing the GPU stage entirely. This is a very different way of doing things and as such will require developers to utilize a new set of development tools and techniques. This is arguably the biggest hurdle with the D77. Fortunately, Arm engineers work closely with initiatives like OpenXR, so we could see an API announcement for simplified developer support in the future.

Overall, the Arm Mali-D77 is an intelligent and logical progression of the heterogeneous computing idea to help solve some of virtual reality’s biggest hardware issues. There are still other hurdles in the wireless communication, tracking, and cost segments of VR left to solve before mainstream adoption can be reconsidered, but the Mali-D77 helps to crack some of the performance issues.