Article 2TH77 OnePlus 5 accused of cheating in benchmarks, Carl Pei responds

OnePlus 5 accused of cheating in benchmarks, Carl Pei responds

by
Harish Jonnalagadda
from Android Central RSS Feed on (#2TH77)

XDA accuses OnePlus 5 of manipulating benchmark scores to gain a 5% performance boost.

Earlier this year, the folks at XDA Developers found that the OnePlus 3T and Meizu Pro 6 Plus were cheating at benchmarks, activating a "performance" mode as soon as a bnechmark test was initialized to eke out a favorable score on testing apps like GeekBench.

With the OnePlus 5 now official, the company is once again under fire, with XDA claiming that units that have been sent out to reviewers manipulate benchmark scores.

oneplus-5-black-8.jpg?itok=y2NM4Je1

With the OnePlus 3T, the company specifically targeted benchmark apps, boosting the CPU frequency when it detected that a particular benchmark was running:

Last time around, OnePlus introduced changes to the behavior of their ROM whenever it detected a benchmark application was opened. Such application names were explicitly listed by their package IDs within the ROM in a manifest that specified the targets.

Then, the ROM would alter the frequency in relation to an adjusted CPU load - our tools showed CPU load would drop to 0% regardless of obvious activity within the application, and the CPU would see a near-minimum frequency of 1.29GHz in the big cores and 0.98GHz in the little cores.

The publication notes that time time around, the cheating is much more "blatant:"

The OnePlus 5, on the other hand, is an entirely different beast - it resorts to the kind of obvious, calculated cheating mechanisms we saw in flagships in the early days of Android, an approach that is clearly intended to maximize scores in the most misleading fashion.

While there are no governor switches when a user enters a benchmark (at least, we can't seem to see that's the case), the minimum frequency of the little cluster jumps to the maximum frequency as seen under performance governors. All little cores are affected and kept at 1.9GHz, and it is through this cheat that OnePlus achieves some of the highest GeekBench 4 scores of a Snapdragon 835 to date.

XDA's OnePlus 5 unit scored 6,700 in GeekBench's multi-core benchmark, which is slightly more than the 6,653 posted by our Xiaomi Mi 6, which is also running a Snapdragon 835. The publication says that OnePlus is able to eke out a 5% uptick in performance in the multi-core benchmark on average, which it detected by using a build of GeekBench with all identifiers removed.

OnePlus co-founder Carl Pei weighed in on the issue on Reddit, stating that the OnePlus 5 treats benchmarks the same way it does resource-intensive games:

We have made it so that when running benchmark apps, the phone performs the same as when running resource intensive apps such as 3D games. We also fully activate our chipset in other parts of OxygenOS, for instance when launching apps to make the launch experience faster and smoother.

We are not making it easier for the chipset to perform, for instance by changing to a lower resolution when detecting a benchmark app. We are not changing the performance of our chipset, for instance by overclocking it.

When users run benchmark apps, which I agree aren't a useful proxy for real life performance, we believe that they want to see the full potential of their device without interference from tampering. That's what we've unlocked.

Every OEM has proprietary performance profiles for their devices, I appreciate that we have a tech enthusiastic following, but feel free to have a look around. :)

OnePlus will have the same build available on consumer devices, with Pei noting that anything otherwise would in fact be considered cheating. With the Snapdragon 835 running full-tilt during intensive games, there is a likelihood of the phone overheating, to which Pei said that the OnePlus 5 will "turn itself off before it gets too hot," and that it has better thermal management from previous generations.

The fact that OnePlus has been caught gaming benchmarks for a second time is disappointing, particularly when there's absolutely no need to do so. Even without artificially-inflated benchmark scores, the OnePlus 5 is one of the fastest devices available in the market today.

And although a small number of users continue rely on benchmarks for purchasing decisions, synthetic benchmarks aren't an accurate representation of a device's real-world performance. As our own Andrew Martonik noted in the Android Central review of the OnePlus 5, the phone is fantastic:

The OnePlus 5 is fantastically fast and smooth in everything I use a smartphone for, right on par with my experience using a Google Pixel XL for several months now. And based on how smooth my OnePlus 3 and 3T have been over time, I don't expect this experience to slow down in the future.

What's your take on the latest OnePlus 5 scandal? Are you convinced by OnePlus' explanation? Let us know in the comments below.

OnePlus 5

OnePlus

L8DlAG1vlY0
External Content
Source RSS or Atom Feed
Feed Location http://www.androidcentral.com/rss.xml
Feed Title Android Central RSS Feed
Feed Link https://www.androidcentral.com/
Reply 0 comments