Why you should test your apps on a range of devices

Almost all app developers will testify to the importance and power of testing. While there are a range of development methodologies in use and a range of SDK options — from Google’s official Java based SDK to third party cross-platform SDKs — every app, regardless of how it is written, needs to be tested.

Testing is in itself a whole branch of software engineering. You can write whole books on testing, testing methodologies, and test automation, in fact lots of people have! Some app developers just pay lip service to testing. The app works OK in the emulator, and it works on their own phone, and that is it. But the problem is this, one sure way for an app fail in the Google Play Store is if it has compatibility issues.

Just go to the Play Store and start to read the feedback left on some apps. “I am using a Samsung XYZ and I get a blank screen at start-up,” or “Works on my Sony ABC, but crashes on my HTCQPR,” and so on. Just replace XYZ, ABC and QPR with the name of a popular model of handset from those manufacturers. That is a sure recipe for disaster.

Diversity

The great thing about the Android ecosystem is its diversity. Some people mistakenly call it fragmentation, but that is really not very accurate. If you look at the desktop PC and laptop market you can see diversity, lots of different sizes, different levels of performance, different GPU manufacturers, different CPU manufacturers, and so on. This is diversity not fragmentation. The same is true of the Android ecosystem, there are phones with 2K screen resolutions and others with 720p or less; there are quad-core phones, hexa-core phones, octa-core phones, etc.; some phones have 512MB of RAM, some 1GB or 2GB, others even more; some handsets supports OpenGL ES 2.0, while others support OpenGL ES 3.0; and so on.

However, like the PC market, the common denominator is the OS, in this case Android. That doesn’t mean that Android ecosystem doesn’t have its problems. In the Windows ecosystem some PCs and laptops are running Windows 7, some are running Windows 8, and so on. For smartphones this means some are running Android 4.1, some 4.4, some 5.0, and so on.

Back in 2012 Google changed the terms and conditions of its SDK to ensure that Android didn’t fragment. The terms and conditions explicitly state that developers using the SDK  do “not take any actions that may cause or result in the fragmentation of Android, including but not limited to distributing, participating in the creation of, or promoting in any way a software development kit derived from the SDK.”

This means that the different derivations of Android, including Amazon’s Fire OS, Cyanogenmod, and MIUI are all still Android at their cores. Another commonality across most Android devices is that they use the same CPU architecture. While Android supports the Intel and MIPS CPU architectures, ARM based processors remain the most prevalent, by a long shot. Not testing your app on an ARM based smartphone is the equivalent to not testing it at all.

Low-end to High-end

One of the main reasons that the ARM architecture has been so successful on mobile is that the architecture is a good fit in all the key market segments. For example, the Samsung Galaxy S6 uses the ARM based Exynos 7420. It is a 64-bit processor with 8 CPU cores (4x ARM Cortex-A57 @ 2.1GHz + 4x Cortex-A53 @ 1.5GHz cores using big.LITTLE), and an ARM Mali-T760 MP8 GPU which supports OpenGL ES 3.1. It is made using the current leading edge manufacturing technologies (14nm FinFET) and supports LPDDR4. In other words it is a beast of a processor.

But at the other end of the scale the ARM architecture is equally well suited. The Android One program is a initiative by Google to enable people in less affluent countries to buy a smartphone. The CPU architecture of choice for the Android One program is ARM. The first generation Android One phones all shipped with a MediaTek MT6582 processor, which includes a quad-core 32-bit Cortex-A7 CPU, and a Mali-400 MP2 GPU which supports OpenGL ES 2.0. It is manufactured using an older chip process technology (28nm).

A Cortex-A7 core is about 3 times slower than a Cortex-A57 core, but it is much cheaper to make and so is great for a program like Android One. But don’t be fooled by the seeming low-end specs of these Android One phones, Google has already released Android 5.1.1 for these devices!

The Android One program highlights the importance of emerging markets. According to Gartner, worldwide smartphone shipments grew by 19 percent during the first quarter of 2015, and that growth was driven mainly by emerging markets. In this market local brands and Chinese vendors recorded an average growth of 73 percent in smartphone sales.

Unity, the popular 3D game engine, has some statistics about what type of devices are being used to play Unity based games. While Android One advocates quad-core processors, the data from Unity shows that dual-core smartphones are still very much in use with just under a third of all smartphones playing Unity based games sporting a dual-core processor. However, quad-core processors are the most popular and account for over half the smartphones in Unity’s dataset, while octa-core phones make up around 4 percent. The same data also shows that 40% of smartphones have less than 1GB of RAM!

Native code, 64-bits, and threading

The official development language of Android is Java, and while that works great for many types of applications, there are times when the need for greater performance means you have to start writing in C or C++. The Android Native Development Toolkit (NDK) is a toolset that allows developers to write large parts of their apps using native-code languages. Google suggest that the NDK is used if you are writing CPU-intensive applications such as game engines, signal processing, and physics simulation.

Since the NDK compiles the C/C++ to native binaries, the only effective way to test the code is on an actual device. For the ARM platform the NDK supports both 32-bit ARMv7 and 64-bit ARMv8.

The NDK also supports ARM’s Advanced SIMD (Single Instruction, Multiple Data) instructions called NEON. They are a set of scalar/vector instructions and registers similar to the MMX/SSE/3DNow! instructions found on x86 desktops. For the ARMv7 architecture NEON was an optional component that might not be included in any given processor. The NDK offers runtime detection to confirm the presence of NEON. As with other native code, the most effective way to test NEON code is on an actual device.

If you’ve written Native (NDK) code to optimize for low end devices or to save battery around hotspots in your code, make sure your compiler flags are compatible across a range of other devices.

If you are using the NDK then you should make sure that your code is 64-bit safe. An increasing number of smartphones are now shipping with 64-bit processors and this trend will continue. While Java apps don’t have to worry about 32-bit vs 64-bit, C and C++ programs do. There are lots of common ‘gotchas’ including magic numbers and the way bit-shifting operations work (especially in overflow situations). It is worth reading 20 issues of porting C++ code on the 64-bit platform to remind yourself of the potential dangers.

While on the topic of performance, it is worth mentioning that one of the best ways to increase the performance of an app is to use multi-threading. Since almost all Android smartphones have at least a dual-core processor and over half have a quad-core processor, it is important to design your code to fully utilize those cores.

Creating multi-threaded apps isn’t hard with Android. Google has lots of information about multi-threading in the Processes and Threads section of the Android documentation. Google also provides several different multi-threaded examples.

However, complex multi-threading programs (those which use semaphores etc.) can behave slightly differently depending on the number of cores, and the way the scheduler is running the threads. One thing is guaranteed, the scheduler will work differently in the emulator than on a real device. The safest course of action is to thoroughly test your app on different devices.

Testing

In an ideal situation you should test you app on lots of different devices under lots of different conditions. But there is obviously a practical limit to the number of devices that can be used for testing, both in terms of costs and time. To help we have put together a guide: Ways to economically test your apps on a range of devices.

Once you have found the means to test your app on multiple devices, it is important to set some criteria for which devices to use. As well as the obvious things like the popularity of a device, the screen resolution, and the version of Android, there are others of the factors you should consider when picking which devices to use:

• GPU – Testing on OpenGL ES 2.0 and 3.0.
• CPU – To check that the performance is acceptable on both high-end and low-end handsets.
• ABI – If you’ve developed any native (C/C++/assembly) code, test it on both 32-bit ARMv7-A and 64-bit ARMv8-A devices.
• SIMD – If you’ve developed any Single Instruction Multiple Data ARM NEON code, test it on both 32-bit and 64-bit devices .

You will want to test your app on devices which only support OpenGL ES 2.0 as well as devices which support OpenGL ES 3.0 and 3.1. You might think that OpenGl ES 2.0 is no longer important, however at the time of writing Google’s Dashboards show that more than half of all Android devices still only support OpenGL ES 2.0. This again highlights the need to test lower-end devices using GPUs like the Mali-400MP and Mali-450MP.

It is also important that you optimize your app for certain GPUs to ensure that you get the best performance (and battery life) from your app. A good starting place is to read our guide: Lighting, console level graphics & ARM – 5 things developers need to know.

In terms of CPU testing, the key is to make sure that your app delivers reasonable performance on low-end devices and isn’t limited to mid- or  high-end only handsets. This means at a minimal that you should test your app on a handset with a quad-core Cortex-A7 based processor, as well as testing it with the latest high-end Samsung or Qualcomm processor.

Wrap up

It is generally accepted that fixing bugs after a product release is more expensive than fixing bugs before release. The reason is that the cost of fixing the bug includes not only the engineering time required to fix the code, manage the change processes, and the build, test, and release of a new version. But it also includes the potential damage done to the app’s reputation including negative scoring and bad reviews on the Google Play Store.

When testing you need to consider which devices to use and rank them in order or priority. Although the Android emulator provides a good starting point to sanity check how an app is running, there is no substitution for running your app on real devices.