MediaTek accused of cheating in benchmarks, accuses 'key competitor' too

We’ve seen several smartphone brands cheat benchmarks over the years, ostensibly in a bid to earn some ill-deserved praise among enthusiasts. But a new report suggests chipset manufacturer MediaTek could be gaming these scores.

Anandtech uncovered evidence of benchmark cheating by MediaTek when it received a Helio P95-powered OPPO Reno 3 Pro (European version) and a standard Dimensity 1000L-powered OPPO Reno 3 (Chinese model).

The outlet’s suspicions were raised when the Reno 3 Pro beat the Reno 3 in the PCMark benchmark utility. This was strange, because the Helio P95’s Cortex-A75 CPU cores are two generations older than the Dimensity 1000L’s Cortex-A77 CPU cores. Furthermore, the P95 only had two of these cores versus the newer chip’s four heavyweight cores.

A stealth version of the PCMark benchmark utility — which manufacturers can’t identify — was installed on the Reno 3 Pro. This revealed a 30% drop in benchmark score compared to the previous questionable score, with Anandtech saying some tests in the benchmark dropped by 75%.

The outlet also tested the Chinese version of the Reno 3 Pro, which swaps the Helio P95 for a Snapdragon 765G, and noted the phone ran the benchmark without resorting to cheating.

MediaTek sports mode uncovered?

Anandtech then dug into the offending device’s firmware files and found references tying benchmark apps to a so-called “sports mode.” It’s believed that this mode ramps up things like the memory controller and scheduler in order to facilitate faster performance.

Further investigation by Anandtech revealed that this wasn’t limited to OPPO’s devices, as it found these references in devices like the realme C3, Xiaomi Redmi Note 8 Pro, and Sony Xperia XA1 as well. The latter device launched in 2017, suggesting this behavior has been going on for a while.

Anandtech was able to extract the references from the Reno 3 Pro running out-of-the-box firmware, but that a subsequent firmware update hid these references away. Nevertheless, it seems like the MediaTek-powered phone was still cheating in benchmarks.

What MediaTek says about cheating claim

MediaTek preemptively published a blog post on its website ahead of the article’s publication, titled “Why MediaTek stands behind our benchmarking practices.” The firm shared the statement it sent to Anandtech as part of the post, seen below:

MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets. We work closely with global device makers when it comes to testing and benchmarking devices powered by our chipsets, but ultimately brands have the flexibility to configure their own devices as they see fit. Many companies design devices to run on the highest possible performance levels when benchmarking tests are running in order to show the full capabilities of the chipset. This reveals what the upper end of performance capabilities are on any given chipset.

Of course, in real world scenarios there are a multitude of factors that will determine how chipsets perform. MediaTek’s chipsets are designed to optimize power and performance to provide the best user experience possible while maximizing battery life. If someone is running a compute-intensive program like a demanding game, the chipset will intelligently adapt to computing patterns to deliver sustained performance. This means that a user will see different levels of performance from different apps as the chipset dynamically manages the CPU, GPU and memory resources according to the power and performance that is required for a great user experience. Additionally, some brands have different types of modes turned on in different regions so device performance can vary based on regional market requirements.

We believe that showcasing the full capabilities of a chipset in benchmarking tests is in line with the practices of other companies and gives consumers an accurate picture of device performance.

The brand also claimed device manufacturers have the final say about optimizations and tweaks for specific APK files. Additionally, MediaTek stated that some brands occasionally give devices in certain regions the ability to enable modes like a “sports mode” or “monster mode” for performance. But it claimed that these modes are disabled in other regions.

MediaTek also criticized Anandtech for questioning its “benchmarking optimization” practices, claiming that this is an industry standard. In fact, the firm says devices using chipsets from other companies do the same thing.

“If they were to review other devices, they would see, as we have, that our key competitor has chipsets that operate in the exact same way — what AnandTech has deemed cheating on device benchmarking tests.”

We have to assume MediaTek is talking about Qualcomm, although it’s clearly steered clear of saying so. We’ve asked the company to clarify the practice as well as if it plans to patch out these tweaks in future firmware releases. We’ll update the article when we hear a response.

This would be a disappointing turn of events if MediaTek is indeed cheating benchmarks, especially after launching its first flagship processor in over two years. It would also be disheartening news in light of the firm bringing more powerful silicon to low-end phones.