Beyond phones: Why Qualcomm is betting big on machine learning, VR and 5G

Qualcomm has made some big announcements this year, introducing its first 5G modem, promising gigabit LTE speeds, and most recently announcing the industry’s first 10nm processor in collaboration with Samsung. Consumers are demanding a lot for from their phones these days, beyond just more power for apps and games.

The trend towards dual cameras requires specialized ISP hardware, while stand-alone and smartphone based virtual reality, which is being pushed by Samsung’s Gear VR and Google’s Daydream, are requiring innovative compromises to slim down into a mobile form factor.

Over the past couple of years, these new demands have changed the way that Qualcomm is approaching processor design, and it seems that the aim is to allow the company to cater to more than just smartphones, as we’ve already seen with drones and virtual reality.

While the Snapdragon 835 is going to be next year’s flagship design, Qualcomm also looks to be building on its existing technologies for lower power IoT devices, cloud computing, and machine learning capabilities too. Here’s what the company has been up to.

Machine learning and heterogeneous compute

While much of the talk around machine and deep learning focuses around cloud compute solutions, there are a growing number of use cases that work best on edge and mobile devices. This is where developments in heterogeneous compute are becoming increasingly important, and Qualcomm has been making strides in this area since the introduction of heterogeneous processing with its Snapdragon 810, as did other SoC developers that made use of ARM’s big.LITTLE technology.

In the mobile space, we first really started talking about heterogeneous compute with the unveiling of Qualcomm’s Snapdragon 820 and how the company planned to improve the performance and energy consumption of image processing and other tasks by running them on the best core in the SoC.

We’re not just talking about loads spread across the CPU and GPU here, but Qualcomm has long been using its Hexagon DSP and Spectra ISP units to offload some tasks too. The idea is that by picking the most efficient component for the task, performance goes up and power consumption goes down.

This trend is sure to be a key part of Qualcomm’s strategy going forward, particularly when used in conjunction with machine learning to improve the features available to consumers. Examples of machine learning applications range widely depending on the hardware, and this isn’t just limited to mobile products.

The automotive market, drones, and smart homes are all primed to make use of machine learning to offer consumers enhanced functionality. This can range all the way from object and voice detection right up to autonomous driving vehicles. In fact, Qualcomm already has a dedicated automotive Snapdragon 820 processor designed with machine learning and communication in mind, although the core features are very similar to the smartphone chip.

Other machine learning examples could include improving device security through facial or voice recognition, to taking a picture and having the software automatically make sure that your family members are in focus. Roughly only 1 percent of smartphone applications are currently making use of machine learning, but International Data Corp expects this number to grow to almost 50 percent of apps in the next two to three years.

Of course, it’s not just Qualcomm and OEMs that are going to be working on machine learning, third party developers are likely to have plenty of good ideas themselves. In order to facilitate easier and optimised development on Snapdragon devices, Qualcomm launched its Neural Processing Engine SDK earlier in the year, which currently supports Snapdragon 820 series processors. The platform supports common deep learning frameworks, including Caffe and CudaConvNet.

There’s also a growing demand for dual camera technology, iris and facial scanning, and virtual reality, which all require an increasing number of complex compute algorithms to be run on today’s smartphones too. However, mobile is limited by very strict power and thermal constraints, which brings its own challenges when it comes to performing these intensive tasks efficiently. Hardware specializations and heterogeneous compete are the keys to overcoming these problems in mobile.

There are a wide range of possible task types with machine learning, some of which run better on CPU type hardware, others on the GPU, and some on dedicated hardware like a DSP. Many of these tasks also need to be done in parallel, so spreading workloads across different cores is essential to bring this type of functionality to consumer.

Eventually, Qualcomm envisions even more dedicated hardware modules included inside SoCs to greatly improve the energy efficiency of the compute heavy tasks, estimated to be anywhere in the region of 4x to 20x more efficient.

We will have to wait and see what type of specializations and tasks are most common before a dedicated pieces of silicon are deemed worthwhile. In the meantime, Qualcomm’s Hexagon DSP, Spectra ISP, and array of smaller sensor processing units, which complement the CPU and GPU that consumers may be more familiar with, are allowing the company to offer optimized hardware for developers looking to get to grips with these new challenges.

We’ve seen a similar take with HiSilicon’s new Kirin 960, which moved ISP hardware into the SoC specifically to deal with enhanced image processing.

Augmented and Virtual reality

Machine learning and heterogeneous computing isn’t just destined for smartphones and cars though, it is also an important part of Qualcomm’s vision for virtual reality products too.

The wide range of sensors for vision and spacial awareness, combined with demanding 3D graphics and a much smaller power budget than PC based equivalents, means that mobile AR and VR platforms have to be especially power and performance efficient.

Here’s just a small set of examples about how different processing requirements could be balanced across a heterogeneous processor.

• CPU – apps, messaging, email, weather, etc
• Sensor Processor – motion tracking, gyroscope, temperature, etc
• ISP – dual / 3D vision cameras, eye tracking, iris detection
• DSP – 3D positional audio and binaural simulation, object detection, facial recognition, gesture detection, noise cancellation, speech recognition and learning
• GPU – Real time graphics, machine learning, and user interface
• Modem – 4G LTE, WiFi, and 5G upload and download for cloud processing

Although augmented and virtual reality will offer users very different experiences, there’s a lot of overlap in terms of hardware and software requirements, particularly when it comes to sensor and graphics processing, and these are really just an extension of today’s smartphone technologies.

Camera sensor counts in VR and AR headsets could reach 4, 8, or higher depending on the use case, and eye tracking is likely to be key to implement important technologies for GPU efficiency like foveated rendering. However, these type of technologies require additional processing power and are often tied into machine learning algorithms, which all ties back into dedicated hardware to make this all run efficiently in compact mobile form factors.

Now, it is possible to provide many of these features with their own dedicated components. An image processor for object recognition, a dedicated DSP for audio, micro-controllers to handle sensors, and a separate CPU to tie the system together. Although highly flexible, this is a very costly and more developer intensive than purchasing a solution that packs all of this into one chip.

Qualcomm has become increasingly focused on providing complete system solutions in a single chip in recent years, as can be seen by the integration of ISP, DSP, and sensor technologies directly into its Snapdragon series. This also allows Qualcomm and OEMs to optimise the hardware to offer these type of features as efficiently as possible, with tight integration between modules for higher peak performance.

There is some risk and trade-offs in predicting the type of features that OEMs are going to want, but Qualcomm is betting that developers are looking for fast-to-market rather than highly custom solutions, especially for emerging fields like virtual and augmented reality.

5G at the heart

While we may know Qualcomm best for its Snapdragon range of application processors, enhanced connectivity – particularly looking towards 5G – is shaping up to be at the heart of many future connected experiences. This applies not only for higher resolution video content, but for streaming VR and AR experiences, sending data for computation in the cloud, and even transmitting location and driver assistance data to vehicles out on the road.

Qualcomm’s recently unveiled X50 5G modem aims to offer up download speeds up to 5 Gbps through support for 8 x 100MHz band carrier aggregation for enhanced bandwidth, up from the 4 x 20MHz CA seen into today’s leading modems. The chip also supports 28GHz millimeter-wave technologies in the form of Verizon’s 5GTF and KT’s 5G-SIG, which could both grow into future 5G standards. It’s a cutting-edge solution that will likely end up powering the first 5G smartphones and tablets in the coming years.

5G isn’t just about providing ever faster data speeds to consumers though, it’s also about connecting up millions of small, low power internet-of-things (IoT) devices across home and industrial markets.

Qualcomm is prepared for this too, with its ultra low-power cellular modems designed for a range of IoT devices. These can support a range of products from smart buildings or appliances that might transfer moderate amounts of data, right down to smart industrial monitoring hardware that may be located on the cell edge and may only need to transfer 10s of Kbps rather than 100s of Mbps.

Specifically for these IoT situations, Qualcomm has its Cat-NB1 compliant MDM9206 and MDM9207 modems already on the market. The MDM9206 can last for several years on just AAA batteries.

In the broader picture, making an early play for 5G is going to give Qualcomm a head start when it comes to not only powering 5G smartphones, but also a wide variety of connected products.

Internet of things

While we are on the subject of IoT, it’s worth noting that it’s not just Qualcomm’s range of Snapdragon processors that are going to be powering this anticipated technology revolution. Qualcomm also offers developers a range of WiFi, Bluetooth, and cellularly connected products complete with an integrated microcontroller with various processing capabilities. These fall under the company’s CSR, FSM, IPQ, and other integrated solution ranges.

Furthermore, Qualcomm is also in the midst of acquiring integrated circuit manufacturer NXP at a cost of $47 billion. No small investment. Once this is completed, Qualcomm will have access to a wider range of of integrated circuit technologies ranging from transistors through to ARM microcontrollers suited for the automotive market and a range of other electronics applications.

This will certainly help the company expand on the more than 1 billion IoT devices already on the market that use Qualcomm chips. The company predicts that there could be as many as 25 billion devices connected to the internet by 2020.

In this regard, and across the mobile and automotive sectors, Qualcomm is looking to provide a selection of integrated solutions that will speed up the development cycle. This can be seen through Qualcomm’s increasing number of development boards, from its Snapdragon Flight Development Kit, through to its Snapdragon VR820 reference headset design. Of course, there’s a trade-off in terms of chip size, stricter thermal limits, and higher costs if developers and manufacturers don’t end up making the most of the additional technologies packed into Qualcomm’s silicon.

Qualcomm is certainly keeping its chips on the cutting edge of emerging consumer and technology trends, but this is just as much a risk as it is an achievement. With IoT still not quite winning over the mainstream and many customers still hesitant about the costs and benefits for virtual reality, not to mention failed AR projects like Google Glass, there’s the risk that simpler, more specialized chips could gain an advantage in the mobile space.

However, if Qualcomm is right and AR, VR, IoT, and smart automotive are the next big fields in consumer electronics, the company is quite far out in front when compared with other smartphone SoC manufacturers.