OnePlus and Meizu just got busted for benchmark cheating

The OnePlus 3, OnePlus 3T, and Meizu Pro 6 are rigged to cheat in benchmarks, a report from XDA-Developers revealed.

XDA-Developers, with help from Primate Labs (makers of the popular benchmark suite Geekbench), discovered that the three phones are set to artificially increase their CPU performance when running certain benchmarks. This is more than just optimization – the software on the devices specifically looks for a number of widely-used benchmarks and tells the CPU to kick into overdrive whenever they are detected.

The rigging can be observed directly, by monitoring the CPU activity when regular apps and benchmark apps are run. In the case of the OnePlus 3 and OnePlus 3T, evidence of the cheating can also be found in the devices’ firmware, in the form of a hard-coded list of benchmarks: Geekbench, AnTuTu, Androbench, Quadrant, Vellamo, and GFXBench.

It’s telling that a “disguised” version of Geekbench 4 called “Bob’s Mini Golf Putt” did not trigger the high-performance mode on the OnePlus 3T. In other words, OnePlus can’t claim that the CPU kicked in due to the load generated by the benchmark – if that were the case, the phone would have behaved identically when running Geekbench 4 and “Bob’s Mini Golf Putt,” which is the same app with a different name.

CPU shenanigans

So how exactly does the rigging work? In the case of the OnePlus 3 and 3T, the CPU is rigged to idle at 1.29 GHz for the big cores and 0.98 GHz for the small cores, even when there’s no load on the processor. For non-benchmark apps, both the small cores and big cores idle at 0.31 GHz.

Meizu has a different – and curious – approach: on the Pro 6, the big, high-performance cores of the CPU are set to activate whenever benchmarks are detected. The weird thing is these big cores should be kicking in anyway, but they normally don’t do so when running regular apps. So instead of tweaking the processor to work as it’s supposed to (big cores kick in when needed), Meizu programmed the CPU to simulate normal functioning when benchmarks are detected.

The XDA-Developers report has all the technical details, and it’s definitely worth a read. We’ll just note that the results of the cheating are quite modest. As you can see in the chart below, the performance increase generated by the benchmark rigging is small, just a few percentage points.

OnePlus came clean

When confronted by XDA, OnePlus admitted the existence of the benchmark rigging behavior:

“In order to give users a better user experience in resource intensive apps and games, especially graphically intensive ones, we implemented certain mechanisms in the community and Nougat builds to trigger the processor to run more aggressively. The trigger process for benchmarking apps will not be present in upcoming Oxygen OS builds on the OnePlus 3 and OnePlus 3T.”

The company said the OnePlus 3 and 3T CPUs are also hard-coded to go into high-performance mode when running games and that’s not going to change. But while that’s understandable (you want graphic-intensive games to benefit from the extra CPU oomph), there’s no good reason to have the CPU rigged for benchmarks.

To its credit, OnePlus acknowledged its mistake and promised to fix it quickly. And, according to XDA-Developers, the behavior was not present in the firmware when the OnePlus 3 launched, as it was added when the Oxygen OS (global) and Hydrogen OS (China) development teams were merged.

Nevertheless, there is no justification for deceptive behavior that ultimately hurts consumers.

Those who forget history are doomed to repeat it

It’s not the first time phone makers are caught red-handed when it comes to benchmarks. In 2013, AnandTech found that the Galaxy S4 gamed benchmarks and a following survey from the same website showed that multiple devices from HTC, ASUS, LG, Samsung were rigged to fool one or multiple benchmark applications. Following the public outcry, some companies vowed to change their practices, while others, including Samsung, refused to admit any wrongdoing.

Following the 2013 revelations, some benchmark makers tried to proof their apps against manipulation, though it’s only so much they can do against a determined manufacturer.

It’s really hard to understand what OnePlus and Meizu were thinking. The risk of getting caught and exposed as a cheater has got to outweigh the small benefits of ranking a little higher in benchmark databases. And, while many publications (including Android Authority) use benchmarks in their smartphone reviews, benchmark scores are hardly the be-all and end-all of reviews. The general performance and perceived smoothness play a much bigger role in how devices are evaluated, both in media and in real life.

It’s not just OnePlus and Meizu

One last twist in the XDA report: the publication tested other devices from multiple manufacturers and found some that failed the hidden benchmark test that exposed the OnePlus 3 and Meizu Pro 6. The report did not reveal which devices are suspect, pending more analysis, but it did reveal the manufacturers that were not caught cheating: HTC, Xiaomi, HUAWEI, HONOR, Google, and Sony. That leaves some very big names out.

Let us know your thoughts!