UL Delists MediaTek Powered Devices Due To Benchmark Whitelisting

Last week we detailed an article covering MediaTek’s seemingly widespread default inclusion of a benchmark whitelist in their chipset BSP (board support package) – a mechanism that enables more aggressive performance tuning of a device’s power management once it detects that a benchmark application is running.

Yesterday, UL, the developers of the PCMark and 3DMark benchmarking suites, have followed up on our investigation and analysed a wider range of devices, and have made the decision to temporarily delist all devices powered by a wide range of MediaTek SoCs, a list of over 50 devices from over 25 different vendors.

We had worked with UL early on in the investigation, with them providing us alternative anonymised versions of the benchmarks which bypass the whitelist detection, thus exposing the cheating behaviour.

The UL news post states:

“Using hidden mechanisms to detect benchmarking apps by name and make app-specific performance optimizations is not an "accepted industry standard." It is, in fact, the very opposite of the accepted standard.

Likewise, benchmark scores based on hidden app-specific optimizations and settings that are enabled by default and not available to the user do not accurately reflect a device's true performance in everyday use. 

Simply put, a device must run a benchmark as if it was any other application. Performance gains must come from reacting to the nature of the workloads in the test rather than the name of the app itself.

As it has with similar cases in the past, we hope this delisting will help persuade MediaTek to change its approach and join the rest of the industry in adopting benchmarking best practices.”

The statement is a harsh rebuttal of MediaTek’s public response to our article, again pointing out that the practice is anything but an “industry standard” – especially damning since it’s coming from one of the major benchmark developers in the industry.

As UL stated in their blog post, and what we also pointed out in our original piece, we hope that the negative reactions to the matter will convince MediaTek to abandon such practices and reconsider their view of what the “industry standard” is.

Related Reading:

• Mobile Benchmark Cheating: When a SoC Vendor Provides It As A Service
• UL Delists Huawei Devices Due To Inaccurate Benchmarks
• Huawei & Honor's Recent Benchmarking Behaviour: A Cheating Headache
• Futuremark Begins Delisting "Cheating" Mobile Devices from 3DMark Database
• They're (Almost) All Dirty: The State of Cheating in Android Benchmarks