Smartphone VR: Another 3D fad or the real deal?

This is the second in the three-part series looking at 3D imaging. In the first, we looked at why every time “3D” has failed to become totally mainstream. Today, 3D is back and trying to make a splash in mobile devices – this time in the form of “virtual reality.” Is VR — especially on smartphones — going to be a long-term success, or just another example of a 3D fad?

VR headsets like the Oculus Rift, HTC Vive, PlayStation VR and similar “tethered” products have made great strides in the last few years. So-called “mobile” VR headsets, like Samsung’s Gear VR and Google’s Daydream, have been even more successful (or at least more widespread). They’re basically head mounts for your smartphone with some optics thrown in, and lately it seems like everyone is making one. But will it stick?

Having just looked at the fanfare-and-failure cycles of 3D in general, should we really expect VR to really have staying power? Will it make a big splash and then fade just like its predecessors?

At heart, VR headsets are stereoscopic “3D” displays, with the all same potential problems and an added twist. It’s “virtual reality” because it lets you look around at, and interact with, this illusory three-dimensional world. That requires displaying the correct images to create a stereo effect, figuring out where the viewer is actually looking, and changing the image to match in real time. 

If you move your head to look behind something, then that something had better move out of the way in your field of view, just as though it were really there. VR requires combining a convincing stereoscopic display with the sensors and graphics processing power needed to render and update your virtual view in a smooth, convincing manner. This is part of why I said that augmented reality is an even bigger challenge: if you’re going to, say, place an imaginary creature on a real tabletop, then not only do you have to render the creature correctly but keep it in the proper relationship to its real-world surroundings.

A dedicated, “tethered” VR headset can pull off all of its assigned tasks pretty well. Connecting it to a standalone computer, which could be anything from a barebones notebook to IBM’s Watson, means you can throw as much processing power as you can muster at problems. But the simple fact that it’s a product designed solely for the purpose of VR means that it has displays, optics, head-tracking systems, and so forth than could all be optimized to that goal. That’s not to say these products are going to be the perfect answer, but they’ve at least got a big leg up on the other option.

That other option is “mobile” VR, which is typically a plastic mount with straps to go over your head and lenses over your eyes, and you supply the rest— namely a smartphone, which provides the displays, processing, and position sensing needed to create a virtual world. This is, in my not-so-humble opinion, a remarkably bad idea.

That “s” on “displays” wasn’t a typo. Yes, your phone only has the one display, but here it’s forced to play the role of two. Left-eye and right-eye images have to be shown simultaneously, and it’s up to the optics in the headset to deliver those correctly to the eyes. That means only half the pixels on the screen are available for each image, which leads to an aspect ratio and resolution charitably described as “less than optimal.”

A Galaxy S8 features a 5.8″ 2960 x 1440 OLED screen at 570 PPI. It’s a really nice smartphone display in anyone’s book, but close to a 2:1 aspect ratio. Splitting it in two in a VR headset means each eye gets an almost perfectly square display to use. That’s not good when we’d really like to have a wide field of view. The human eye uses something roughly equivalent to a 5:3 aspect ratio (of course, it’s also not a nice clean rectangle, but rather a sort of fuzzy oval).

There are two ways to fix this. You could use the full area of each half, displaying pre-distorted image on the square space and relying on the optics to stretch the image to the desired wider area— the same sort of trick used in anamorphic movies. However, If the distortion introduced into the image isn’t exactly what the optics were designed to “undo,” you’ve got problems. The other option is to just not use the full height of the display. If, on the S8, we have a 1440 x 1440 space for each image, but we want, say, a 16:9 view, we could just center a 1440 x 810 image in that space and it would be good to go, albeit at well under half the phone’s full resolution.

We could just demand a higher resolution in our phone screens. “But Bob,” I hear you protest, “didn’t you just tell us a few weeks ago that packing more pixels onto a phone was a bad idea?” Yes, I did. That article also generated some comments which took me to task for ignoring the needs of VR. But that was my point: smartphone display choices should ignore VR, at least as a top priority.

Phone-based VR headsets represent the entry level in the VR market. They suffer from too many compromises already to be the choice for serious VR users, and paying for the extreme levels of screen resolution needed to address just that one issue makes no sense. As good as they are, smartphone graphics processing and position/orientation sensors just aren’t up the task of matching what you can do with a dedicated headset and tracking hardware.

Again, consider the Galaxy S8. It’s got an MSRP of more than $700—over $200 higher than Samsung’s own Odyssey VR/MR headset, which features dual 1440 x 1600 OLED displays coupled with a full array of cameras, motion and position sensors, integrated headphones, and adjustments for interpupillary distance. Putting a higher resolution display in a phone just for VR is like paying to put a Ferrari engine in a Toyota Prius. Sure, you’d get a lot more power, but the platform just isn’t meant to do what you want. You’re better off just buying the product meant for that use in the first place.

We could even put a 4K display into a phone, and get a great resolution for each eye. The graphics processing burden goes up geometrically with increased resolution. Even if you build the added power into the processor, it just isn’t the best idea for a battery-powered device. Phone-based VR is best for what it was supposed to be: a quick and relatively economical means of introducing VR into the consumer market. But let’s not make the mistake of thinking it’s the right answer for the serious VR fanatic.

It’s not like dedicated VR headsets are perfect either. They still suffer from all of the other problems we’ve described earlier with stereoscopic displays, with the additional concern that motion tracking and its resulting view is never going to quite match what we see in real life. VR is getting a lot of attention in education circles, to name one interested market, but how long will that love affair last if kids get severe eye fatigue from using it?

There’s a way around even that concern, though. All we need is a display that can produce a real three-dimensional image, one that actually has the appearance of solid objects occupying space, and without any glasses, headsets, head tracking, or any of those burdens. We’ve even already seen examples of this; surely everyone by now has seen a hologram. You’ve probably even got a few in your wallet, on your credit cards.

So when can we replace our old-fashioned flat displays, and free us from all this stereoscopic nonsense?Stay tuned.