Jump to content

One plus 5 got caught cheating at benchmarking programs

early an accusation of cheating 

 

"Earlier this year, we published a report that denounced OnePlus (and other companies) for their improper behavior in regards to benchmark manipulation on newer builds of OxygenOS. Today, we sadly must follow-up on our accusations as the company has once more been inappropriately manipulating benchmark scores in the OnePlus 5."  

 

 

getting caught redhanded 

 

"While no customers have a device in their hands (it just launched after all), we have learned about OnePlus’ new benchmark cheating mechanism through our review unit, which we received about ten days ago before the day the embargo breaks and reviewers are allowed to report on the device. Unfortunately, it is almost certain that every single review of the OnePlus 5 that contains a benchmark is using misleading results, as OnePlus provided reviewers a device that cheats on benchmarks." 

 

how is it done? 

 

"When entering certain benchmarking apps, the OnePlus 3T’s cores would stay above 0.98 GHz for the little cores and 1.29 GHz for the big cores, even when the CPU load dropped to 0%. This is quite strange, as normally both sets of cores drop down to 0.31 GHz on the OnePlus 3T when there is no load. Upon first seeing this we were worried that OnePlus’ CPU scaling was simply set a bit strangely, however upon further testing we came to the conclusion that OnePlus must be targeting specific applications. Our hypothesis was that OnePlus was targeting these benchmarks by name, and was entering an alternate CPU scaling mode to pump up their benchmark scores. One of our main concerns was that OnePlus was possibly setting looser thermal restrictions in this mode in order avoid the problems they had with the OnePlus One, OnePlus X, and OnePlus 2, where the phones were handling the additional cores coming online for the multi-core section of Geekbench poorly, and occasionally throttling down substantially as a result (to the point where the OnePlus X sometimes scored lower in the multi-core section than in the single core section). You can find heavy throttling in our OnePlus 2 review, where we found the device could shed off up to 50% of its Geekbench 3 multi-core score. Later, when we began comparing throttling and thermals across devices, the OnePlus 2 became a textbook example of what OEMs should avoid." 

 

why does it matter? 

 

they got caught cheating in the one plus 5 benchmark 

 

"

The OnePlus 5, on the other hand, is an entirely different beast — it resorts to the kind of obvious, calculated cheating mechanisms we saw in flagships in the early days of Android, an approach that is clearly intended to maximize scores in the most misleading fashion. While there are no governor switches when a user enters a benchmark (at least, we can’t seem to see that’s the case), the minimum frequency of the little cluster jumps to the maximum frequency as seen under performance governors. All little cores are affected and kept at 1.9GHz, and it is through this cheat that OnePlus achieves some of the highest GeekBench 4 scores of a Snapdragon 835 to date –  and likely the highest attainable given its no-compromise configuration with its specific configuration. Scores certainly higher than those obtained by similar devices and Qualcomm’s own MSM8998 test device which we were lucky enough to benchmark. Below is a list of benchmark applications affected:

  • AnTuTu (com.antutu.benchmark.full)
  • Androbench (com.andromeda.androbench2)
  • Geekbench 4 (com.primatelabs.geekbench)
  • GFXBench (com.glbenchmark.glbenchmark27)
  • Quadrant (com.aurorasoftworks.quadrant.ui.standard)
  • Nenamark 2 (se.nena.nenamark2)
  • Vellamo (com.quicinc.vellamo) "

 

what did we learn? 

 

 

Fool me Once, Shame on Me; Fool me Twice, Shame on You

 

"It is a bit upsetting that it has gotten to the point where we have to call out the same company twice for manipulating benchmark scores. The fact that all of this was done on review units as well further exacerbates the issue: this cheating mechanism is aimed at maximizing performance and making the device look better or faster in performance sections of reviews. The targeting and manipulation system was packaged in pre-production units sent to journalists who will base their findings on their device from OnePlus, many of them unable or unwilling to verify the existence of cheating in their review unit. It is by no means their fault, but XDA is on the lookout for benchmark manipulation only because we found it in the past, and we thought it was best to inform our readers and potential phone buyers.

We hope this article might rekindle a broader conversation about benchmarks, their role, and their utility in today’s smartphone reviews. Make no mistake, companies like Qualcomm and Samsung do care about benchmarks, and they do consider them a valid, if incomplete, way for customers to judge the performance of their devices even though they have more sophisticated tools to refer to when developing their processors. Ultimately, benchmarks can be of great importance if one understands what the software is measuring, and to which extent its results can be used to deduce the ranking of a particular processor, a particular configuration of hardware, or in more holistic terms, a specific phone with the changes in behavior its software introduces as well. I think that we have come to a time where it is more important to focus on real-world performance and power efficiency than in raw computing or processing prowess, because it is obviously clear at this point that the bottleneck to real-world performance comes from Android and particular implementations of it by OEMs." 

 

source 

-https://www.xda-developers.com/benchmark-cheating-strikes-back-how-oneplus-and-others-got-caught-red-handed-and-what-theyve-done-about-it/

-https://www.xda-developers.com/oneplus-5-benchmark-cheating-reviews/

 

 

TLDR- one plus has a history of getting caught cheating on benchmarks, they are still doing it with the one plus 5. 

OP3T-Score-Comparison.png

OP3T-Single-Core-Throttling.png

OP3T-Multi-Core-Throttling.png

 

one plus 5 

watermarked_GEEKBENCH4_noCHEATING.png

watermarked_GEEKBENCH4_CHEATING2.png

watermarked_OnePlus-5-Score-Comparison.png

Link to comment
Share on other sites

Link to post
Share on other sites

I'm honestly not even that shocked by this. 

 

Tons of companies have done this before. Anandtech usually is pretty good about figuring out if companies are cheating. I recently went back and read the reviews for the htc m7, m8, and m9 and htc cheated in at least one of them.

Make sure to quote me or tag me when responding to me, or I might not know you replied! Examples:

 

Do this:

Quote

And make sure you do it by hitting the quote button at the bottom left of my post, and not the one inside the editor!

Or this:

@DocSwag

 

Buy whatever product is best for you, not what product is "best" for the market.

 

Interested in computer architecture? Still in middle or high school? P.M. me!

 

I love computer hardware and feel free to ask me anything about that (or phones). I especially like SSDs. But please do not ask me anything about Networking, programming, command line stuff, or any relatively hard software stuff. I know next to nothing about that.

 

Compooters:

Spoiler

Desktop:

Spoiler

CPU: i7 6700k, CPU Cooler: be quiet! Dark Rock Pro 3, Motherboard: MSI Z170a KRAIT GAMING, RAM: G.Skill Ripjaws 4 Series 4x4gb DDR4-2666 MHz, Storage: SanDisk SSD Plus 240gb + OCZ Vertex 180 480 GB + Western Digital Caviar Blue 1 TB 7200 RPM, Video Card: EVGA GTX 970 SSC, Case: Fractal Design Define S, Power Supply: Seasonic Focus+ Gold 650w Yay, Keyboard: Logitech G710+, Mouse: Logitech G502 Proteus Spectrum, Headphones: B&O H9i, Monitor: LG 29um67 (2560x1080 75hz freesync)

Home Server:

Spoiler

CPU: Pentium G4400, CPU Cooler: Stock, Motherboard: MSI h110l Pro Mini AC, RAM: Hyper X Fury DDR4 1x8gb 2133 MHz, Storage: PNY CS1311 120gb SSD + two Segate 4tb HDDs in RAID 1, Video Card: Does Intel Integrated Graphics count?, Case: Fractal Design Node 304, Power Supply: Seasonic 360w 80+ Gold, Keyboard+Mouse+Monitor: Does it matter?

Laptop (I use it for school):

Spoiler

Surface book 2 13" with an i7 8650u, 8gb RAM, 256 GB storage, and a GTX 1050

And if you're curious (or a stalker) I have a Just Black Pixel 2 XL 64gb

 

Link to comment
Share on other sites

Link to post
Share on other sites

basically their way of admitting its a crap phone lol

Favorite Threads: PSU Tier List

 

My Stuff n' Things

Spoiler

The Beast (My Rig)   |CPU: AMD Ryzen 9 3900X|  |Cooling: Enermax Liquimax III, 6x 120mm Noctua Redux|  |Motherboard:  MSI MPG B550 Gaming Plus|  |RAM: 4x32gb 3200 G.Skill TridentZ NEO|  |Graphics Card: EVGA(RIP) GeForce RTX 3070TI FTW3|  |Power Supply: Corsair CX-M 750W|  |Case: Corsair 4000D Airflow Mid Tower Case(Black)|  |SSD: 1Tb WD BLACK NVMe, 500gb NVMe, 1Tb Samsung 850 EVO|  |Monitor: MSI Optix MPG341QR 34" Ultrawide|  |Keyboard: Logitech G815|  |Mouse: Logitech G203 LIGHTSYNC|  |Audio Interface: FiiO K7 DAC/Amp|  |Headphones: Sennheiser HD6XX |Webcam: Logitech C920, Logitech C270|

 

My Network Rack  |Switch: Cisco Dell PowerConnect 5548P|  |Router: Unifi USG|  |Rack: 12U|  |Server: HP Z420|  |Services: Proxmox PVE, Wireguard, Pihole, NVR, NAS|

 

Link to comment
Share on other sites

Link to post
Share on other sites

they will be put in bad sport lobbies for a week.

 

 

 

gta reference

Favorite Threads: PSU Tier List

 

My Stuff n' Things

Spoiler

The Beast (My Rig)   |CPU: AMD Ryzen 9 3900X|  |Cooling: Enermax Liquimax III, 6x 120mm Noctua Redux|  |Motherboard:  MSI MPG B550 Gaming Plus|  |RAM: 4x32gb 3200 G.Skill TridentZ NEO|  |Graphics Card: EVGA(RIP) GeForce RTX 3070TI FTW3|  |Power Supply: Corsair CX-M 750W|  |Case: Corsair 4000D Airflow Mid Tower Case(Black)|  |SSD: 1Tb WD BLACK NVMe, 500gb NVMe, 1Tb Samsung 850 EVO|  |Monitor: MSI Optix MPG341QR 34" Ultrawide|  |Keyboard: Logitech G815|  |Mouse: Logitech G203 LIGHTSYNC|  |Audio Interface: FiiO K7 DAC/Amp|  |Headphones: Sennheiser HD6XX |Webcam: Logitech C920, Logitech C270|

 

My Network Rack  |Switch: Cisco Dell PowerConnect 5548P|  |Router: Unifi USG|  |Rack: 12U|  |Server: HP Z420|  |Services: Proxmox PVE, Wireguard, Pihole, NVR, NAS|

 

Link to comment
Share on other sites

Link to post
Share on other sites

Samsung did the same thing some time ago and people shrugged it off cause it was Samsung

Windows 10 Edu | Asus ROG Strix X570-F Gaming | Ryzen 9 3950x | 4x 16GB G.Skill Trident Z RGB| ROG Strix GeForce® RTX 2080 SUPER™ Advanced edition | Samsung 980 PRO 500GB + Samsung 970 Evo Plus 2TB + 8TB Seagate Barracuda | EVGA Supernova 650 G2 | Alienware AW3418DW + LG 34uc87c + Dell u3419w | Asus Zephyrus G14

Link to comment
Share on other sites

Link to post
Share on other sites

Just now, Scruffy90 said:

Samsung did the same thing some time ago and people shrugged it off cause it was Samsung

I'm gonna shrug this one off as well actually. :P

We have a NEW and GLORIOUSER-ER-ER PSU Tier List Now. (dammit @LukeSavenije stop coming up with new ones)

You can check out the old one that gave joy to so many across the land here

 

Computer having a hard time powering on? Troubleshoot it with this guide. (Currently looking for suggestions to update it into the context of <current year> and make it its own thread)

Computer Specs:

Spoiler

Mathresolvermajig: Intel Xeon E3 1240 (Sandy Bridge i7 equivalent)

Chillinmachine: Noctua NH-C14S
Framepainting-inator: EVGA GTX 1080 Ti SC2 Hybrid

Attachcorethingy: Gigabyte H61M-S2V-B3

Infoholdstick: Corsair 2x4GB DDR3 1333

Computerarmor: Silverstone RL06 "Lookalike"

Rememberdoogle: 1TB HDD + 120GB TR150 + 240 SSD Plus + 1TB MX500

AdditionalPylons: Phanteks AMP! 550W (based on Seasonic GX-550)

Letterpad: Rosewill Apollo 9100 (Cherry MX Red)

Buttonrodent: Razer Viper Mini + Huion H430P drawing Tablet

Auralnterface: Sennheiser HD 6xx

Liquidrectangles: LG 27UK850-W 4K HDR

 

Link to comment
Share on other sites

Link to post
Share on other sites

Just now, Pandalf said:

Do people actually care about benchmarks when purchasing a phone with a specific CPU?

I would bet so

We have a NEW and GLORIOUSER-ER-ER PSU Tier List Now. (dammit @LukeSavenije stop coming up with new ones)

You can check out the old one that gave joy to so many across the land here

 

Computer having a hard time powering on? Troubleshoot it with this guide. (Currently looking for suggestions to update it into the context of <current year> and make it its own thread)

Computer Specs:

Spoiler

Mathresolvermajig: Intel Xeon E3 1240 (Sandy Bridge i7 equivalent)

Chillinmachine: Noctua NH-C14S
Framepainting-inator: EVGA GTX 1080 Ti SC2 Hybrid

Attachcorethingy: Gigabyte H61M-S2V-B3

Infoholdstick: Corsair 2x4GB DDR3 1333

Computerarmor: Silverstone RL06 "Lookalike"

Rememberdoogle: 1TB HDD + 120GB TR150 + 240 SSD Plus + 1TB MX500

AdditionalPylons: Phanteks AMP! 550W (based on Seasonic GX-550)

Letterpad: Rosewill Apollo 9100 (Cherry MX Red)

Buttonrodent: Razer Viper Mini + Huion H430P drawing Tablet

Auralnterface: Sennheiser HD 6xx

Liquidrectangles: LG 27UK850-W 4K HDR

 

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, Scruffy90 said:

Samsung did the same thing some time ago and people shrugged it off cause it was Samsung

the sad thing is Samsung stopped doing it after getting caught

Link to comment
Share on other sites

Link to post
Share on other sites

Just now, Pandalf said:

Do people actually care about benchmarks when purchasing a phone with a specific CPU?

i do... If you're paying something that is good as top-tier phones (such as s8 or mi6) for almost half of the price but you notice that is 25% slower (its an example), you should be worried too...

Remember to quote me (or someone else), otherwise we won't going to recieve your answers...

 

PC Specs                   PCPartpicker full performance builds (from350$-1250$)

Link to comment
Share on other sites

Link to post
Share on other sites

4 minutes ago, Blackhole890 said:

i do... If you're paying something that is good as top-tier phones (such as s8 or mi6) for almost half of the price but you notice that is 25% slower (its an example), you should be worried too...

In real world scenarios phones with the same SoC perform indistinguishably from one another, even if they are running a different flavour of Android.

From salty to bath salty in 2.9 seconds

 

Link to comment
Share on other sites

Link to post
Share on other sites

28 minutes ago, Scruffy90 said:

Samsung did the same thing some time ago and people shrugged it off cause it was Samsung

I don't trust Samsung products for a while, people seem to like them a bit too much with a blind following mentality.

I will not buy any phones, SSDs, monitors, devices from them.

after the SSD issues they had I will not buy another one.

if you want to annoy me, then join my teamspeak server ts.benja.cc

Link to comment
Share on other sites

Link to post
Share on other sites

Would a phone company be silly enough to do this knowing it will be found? 

Link to comment
Share on other sites

Link to post
Share on other sites

i dont care much about the specs of my phone, just its battery life and screen. as long as it dose web browsing and plays youtube im pretty much good

I spent $2500 on building my PC and all i do with it is play no games atm & watch anime at 1080p(finally) watch YT and write essays...  nothing, it just sits there collecting dust...

Builds:

The Toaster Project! Northern Bee!

 

The original LAN PC build log! (Old, dead and replaced by The Toaster Project & 5.0)

Spoiler

"Here is some advice that might have gotten lost somewhere along the way in your life. 

 

#1. Treat others as you would like to be treated.

#2. It's best to keep your mouth shut; and appear to be stupid, rather than open it and remove all doubt.

#3. There is nothing "wrong" with being wrong. Learning from a mistake can be more valuable than not making one in the first place.

 

Follow these simple rules in life, and I promise you, things magically get easier. " - MageTank 31-10-2016

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

18 minutes ago, The Benjamins said:

after the SSD issues they had

what SSD issues? source? 

Link to comment
Share on other sites

Link to post
Share on other sites

1 hour ago, Pandalf said:

Do people actually care about benchmarks when purchasing a phone with a specific CPU?

some reason some do. the extra 3% seems to matter. But really as long as thermals are decent its going to be a very similar experiance throughout all the phones 

Link to comment
Share on other sites

Link to post
Share on other sites

Why the hell does this matter? If you are choosing a phone based on benchmarks then you have more problems.

Link to comment
Share on other sites

Link to post
Share on other sites

5 minutes ago, mynameisjuan said:

Why the hell does this matter? If you are choosing a phone based on benchmarks then you have more problems.

false advertising    

Link to comment
Share on other sites

Link to post
Share on other sites

6 minutes ago, nerdslayer1 said:

false advertising    

Synthetic benchmarks IMO are not advertising due to the fact that they can be manipulated. And like I said, only an idiot would chose a phone for benchmarks alone.

Link to comment
Share on other sites

Link to post
Share on other sites

Just now, mynameisjuan said:

Synthetic benchmarks IMO are not advertising due to the fact that they can be manipulated. And like I said, only an idiot would chose a phone for benchmarks alone.

so then, so cheating on benchmarks is justified? 

 

1 minute ago, mynameisjuan said:

not advertising due to the fact that they can be manipulated

so that means everyone should start cheating on benchmarks? 

1 minute ago, mynameisjuan said:

I said, only an idiot would chose a phone for benchmarks alone

interesting how benchmarks are boosting points for these specs driven company. in your opinion what's the best way a person can see how the performance is in one glance without using it?  

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

9 minutes ago, nerdslayer1 said:

so then, so cheating on benchmarks is justified? 

 

so that means everyone should start cheating on benchmarks? 

interesting how benchmarks are boosting points for these specs driven company. in your opinion what's the best way a person can see how the performance is in one glance without using it?  

 

 

1. I dont care about synthetic benchmarks. So who cares if they are cheating

2. What? How is "not advertising due to the fact that they can be manipulated" interpreted into everyone should cheat

3. Benchmarks on phones are stupid and pointless. Time and time again nexus devices have show benchmarks dont mean shit and OS optimization means a hell of a lot more. You want to know performance, look up a similar phone with the same chip and estimate, not rely of half assed benchmarks.

Link to comment
Share on other sites

Link to post
Share on other sites

2 hours ago, nerdslayer1 said:

what SSD issues? source? 

I assume he means the 840 evo issue where old files would slow down -- which they rectified with some firmware patch. My 840 Evo is still going strong and is just as fast as the day I bought it. 

 

1 hour ago, mynameisjuan said:

1. I dont care about synthetic benchmarks. So who cares if they are cheating

2. What? How is "not advertising due to the fact that they can be manipulated" interpreted into everyone should cheat

3. Benchmarks on phones are stupid and pointless. Time and time again nexus devices have show benchmarks dont mean shit and OS optimization means a hell of a lot more. You want to know performance, look up a similar phone with the same chip and estimate, not rely of half assed benchmarks.

 

While I don't care about benchmarks, it's clear that they're trying to manipulate consumers into buying their product (and they clearly think it works) -- otherwise they wouldn't waste their energy trying to cheat at benchmarks.

PSU Tier List | CoC

Gaming Build | FreeNAS Server

Spoiler

i5-4690k || Seidon 240m || GTX780 ACX || MSI Z97s SLI Plus || 8GB 2400mhz || 250GB 840 Evo || 1TB WD Blue || H440 (Black/Blue) || Windows 10 Pro || Dell P2414H & BenQ XL2411Z || Ducky Shine Mini || Logitech G502 Proteus Core

Spoiler

FreeNAS 9.3 - Stable || Xeon E3 1230v2 || Supermicro X9SCM-F || 32GB Crucial ECC DDR3 || 3x4TB WD Red (JBOD) || SYBA SI-PEX40064 sata controller || Corsair CX500m || NZXT Source 210.

Link to comment
Share on other sites

Link to post
Share on other sites

Sad to see, but it won't change my opinion about the phone (which I have not looked into yet, but I don't like the price).

I know which SoC the phone has so I will know how it compares to other phones without needing to look at benchmarks.

Link to comment
Share on other sites

Link to post
Share on other sites

21 minutes ago, LAwLz said:

Sad to see, but it won't change my opinion about the phone (which I have not looked into yet, but I don't like the price).

I know which SoC the phone has so I will know how it compares to other phones without needing to look at benchmarks.

$500.....yeah, for that I'll buy a phone that actually gets customer support. 

PSU Tier List | CoC

Gaming Build | FreeNAS Server

Spoiler

i5-4690k || Seidon 240m || GTX780 ACX || MSI Z97s SLI Plus || 8GB 2400mhz || 250GB 840 Evo || 1TB WD Blue || H440 (Black/Blue) || Windows 10 Pro || Dell P2414H & BenQ XL2411Z || Ducky Shine Mini || Logitech G502 Proteus Core

Spoiler

FreeNAS 9.3 - Stable || Xeon E3 1230v2 || Supermicro X9SCM-F || 32GB Crucial ECC DDR3 || 3x4TB WD Red (JBOD) || SYBA SI-PEX40064 sata controller || Corsair CX500m || NZXT Source 210.

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×