Huawei benchmark cheating? Anandtech examines.

[WMGroomAK]

Interesting article over on Anandtech looks into Huawei enabling higher power limits and thermal headroom on the SoCs for benchmarking tools.  Essentially this allows for Huawei to score higher in benchmark applications than you would expect with this 'high-performance' mode turned off.  When Anandtech questioned Huawei about this at IFA, their response was fairly much a confirmation and stating that they do this to stay competitive with other companies in China that do this to get high scores.  

 

https://www.anandtech.com/show/13318/huawei-benchmark-cheating-headache

Quote

Does anyone remember our articles regarding unscrupulous benchmark behavior back in 2013? At the time we called the industry out on the fact that most vendors were increasing thermal and power limits to boost their scores in common benchmark software. Fast forward to 2018, and it is happening again.

...

When we exposed one vendor, it led to a cascade of discussions and a few more articles investigating more vendor involved in the practice, and then even Futuremark delisting several devices from their benchmark database. Scandal was high on the agenda, and the results were bad for both companies and end users: devices found cheating were tarnishing the brand, and consumers could not take any benchmark data as valid from that company. Even reviewers were misled. It was a deep rabbit hole that should not have been approached – how could a reviewer or customer trust what number was coming out of the phone if it was not in a standard ‘mode’?

 

So thankfully, ever since then, vendors have backed off quite a bit on the practice. Since 2013, for several years it would appear that a significant proportion of devices on the market are behaving within expected parameters. There are some minor exceptions, mostly from Chinese vendors, although this comes in several flavors. Meizu has a reasonable attitude to this, as when a benchmark is launched the device puts up a prompt to confirm entering a benchmark power mode, so at least they’re open and transparent about it. Some other phones have ‘Game Modes’ as well, which either focus on raw performance, or extended battery life.

 

So today we are publishing two front page pieces. This one is a sister article to our piece addressing Huawei’s new GPU Turbo, and while it makes overzealous marketing claims, the technology is sound. Through the testing for that article, we actually stumbled upon this issue, completely orthogonal to GPU turbo, which needs to be published. We also wanted to address something that Andrei has come across while spending more time with this year’s devices, including the newly released Honor Play.

...

Looking back at it now after some re-testing, it seems quite blatant as to what Huawei and seemingly Honor had been doing: the newer devices come with a benchmark detection mechanism that enables a much higher power limit for the SoC with far more generous thermal headroom. Ultimately, on certain whitelisted applications, the device performs super high compared to what a user might expect from other similar non-whitelisted titles. This consumes power, pushes the efficiency of the unit down, and reduces battery life.

...

As usual with investigations like this, we offered Huawei an opportunity to respond. We met with Dr. Wang Chenglu, President of Software at Huawei’s Consumer Business Group, at IFA to discuss this issue, which is purely a software play from Huawei. We covered a number of topics in a non-interview format, which are summarized here.

 

Dr. Wang asked if these benchmarks are the best way to test smartphones as a whole, as he personally feels that these benchmarks are moving away from real world use. A single benchmark number, stated Huawei’s team, does not show the full experience. We also discussed the validity of the current set of benchmarks, and the need for standardized benchmarks. Dr. Wang expressed his preference for a standardized benchmark that is more like the user experience, and they want to be a part of any movement towards such a benchmark.

...

Huawei stated that they have been working with industry partners for over a year to find the best tests closest to the user experience. They like the fact that for items like call quality, there are standardized real-world tests that measure these features that are recognized throughout the industry, and every company works towards a better objective result. But in the same breath, Dr. Wang also expresses that in relation to gaming benchmarking that ‘others do the same testing, get high scores, and Huawei cannot stay silent’.

 

He states that it is much better than it used to be, and that Huawei ‘wants to come together with others in China to find the best verification benchmark for user experience’. He also states that ‘in the Android ecosystem, other manufacturers also mislead with their numbers’, citing one specific popular smartphone manufacturer in China as the biggest culprit, and that it is becoming ‘common practice in China’. Huawei wants to open up to consumers, but have trouble when competitors continually post unrealistic scores.

 

Ultimately Huawei states that they are trying to face off against their major Chinese competition, which they say is difficult when other vendors put their best ‘unrealistic’ score first. They feel that the way forward is standardization on benchmarks, that way it can be a level field, and they want the media to help with that. But in the interim, we can see that Huawei has also been putting its unrealistic scores first too.

Sorry for posting a large quote, but this feels like the most relevant from the first page of the article and I would encourage people to browse the whole article...  As it is, it definitely raises questions about any benchmark you see for Huawei and Honor within the last year in particular, but also any benchmark on any Chinese phone manufacturer.  

[mynameisjuan]

Well I mean it does show peak performance of the chip. 

 

But who the F cares about phone benchmarks anyway. Real world use its all that matters. 

[ARikozuM]

“Everyone is doing it, so we should do so as well.”

 

No, you shouldn’t. If I have no control over that whitelist, don’t use it for yourselves. If I buy a product that does X I want it to do X at all times. 

[Bouzoo]

It's like Huawei can't make a damn good things nowadays. 

[mynameisjuan]

1 minute ago, Bouzoo said:

It's like Huawei can't make a damn good things nowadays. 

Matebook x Pro is legit

[jefire411]

13 minutes ago, Bouzoo said:

It's like Huawei can't make a damn good things nowadays. 

GPU turbo also seems legit but CPU turbo is DOA

[WMGroomAK]

28 minutes ago, mynameisjuan said:

But who the F cares about phone benchmarks anyway. Real world use its all that matters.

 

11 minutes ago, huilun02 said:

This is unacceptable because majority of people who buy flagship phones just want to compare benchmark scores all the time

Browsers and social media apps need cutting edge performance else they just crash and burn

While I agree for the most part that real world use is what matters, the big issue is that the benchmarks are supposed to provide a common ground of comparison & correlation between devices and how the real world, day to day performance is.  

[PocketNerd]

I don't get why CPU/GPU benchmarks for Smartphones exists....

 

[mynameisjuan]

2 minutes ago, WMGroomAK said:

 

While I agree for the most part that real world use is what matters, the big issue is that the benchmarks are supposed to provide a common ground of comparison & correlation between devices and how the real world, day to day performance is.  

Benchmarks =/= real world use but a bunch of program test can give expectations. In the phone world benchmarks are pretty moot as even slower chips are snappy as hell on some phones. 

[DocSwag]

1 hour ago, mynameisjuan said:

Well I mean it does show peak performance of the chip. 

 

But who the F cares about phone benchmarks anyway. Real world use its all that matters. 

Single threaded performance matters for web performance. Efficiency is also a big deal; e.g. watching videos would last longer with a more efficient SoC. Some people play some games on their phone from time to time. Definitely doesn't say everything but it still matters.

44 minutes ago, mynameisjuan said:

Benchmarks =/= real world use but a bunch of program test can give expectations. In the phone world benchmarks are pretty moot as even slower chips are snappy as hell on some phones. 

I would disagree; quite a few flagship phones don't even sustain 60 fps in many basic tasks.

48 minutes ago, PocketNerd said:

I don't get why CPU/GPU benchmarks for Smartphones exists....

 

Because they do matter for reasons I've outlined above.

[LAwLz]

47 minutes ago, PocketNerd said:

I don't get why CPU/GPU benchmarks for Smartphones exists....

For the same reason why CPU/GPU benchmarks for laptops exist.

So that you can compare performance.

 

Even if you don't care about performance because "it's good enough" (which is something people used to say about 1GHz single core processors for desktops) benchmarks are still important. It shows what progress is being made, which solutions work/doesn't work and where improvements are needed.

[LienusLateTips]

EOS 5D camera fakery -> Strike 1

ROM locking -> Strike 2

Benchmark Cheating -> You're out.

[mr moose]

Before this topic turns into a Chinese lynch MOB,  I just want to point out these US and European companies who have also released products that cheat the consumer or on tests:

 

VW (emissions testing)

Apple (battery and processor performance)

Nvida (3.5G debacle)

Intel (compiler anti trust case)

AMD (bait and switch)

I am sure the list  goes on.

 

[dizmo]

Does anyone really pay attention to phone benchmarks?
I'm all about mobile tech. I love phones, and always look forward to releases. Even I don't give a shit about benchmarks though.

[ARikozuM]

4 hours ago, mr moose said:

Before this topic turns into a Chinese lynch MOB,  I just want to point out these US and European companies who have also released products that cheat the consumer or on tests:

 

VW (emissions testing)

Apple (battery and processor performance)

Nvida (3.5G debacle)

Intel (compiler anti trust case)

AMD (bait and switch)

I am sure the list  goes on.

 

 

5 hours ago, ARikozuM said:

“Everyone is doing it, so we should do so as well.”

 

No, you shouldn’t. If I have no control over that whitelist, don’t use it for yourselves. If I buy a product that does X I want it to do X at all times. 

Just because those companies do it doesn’t mean we should overlook them or anyone else.

[D13H4RD]

I don’t get it.

 

They did it because other companies do it too? What?

[mr moose]

2 hours ago, ARikozuM said:

 

Just because those companies do it doesn’t mean we should overlook them or anyone else.

I never said we should over look it.   By all means hold them accountable.  What I said was you shouldn't try to claim they are doing this solely because they are Chinese.  I was very specific about it.

 

52 minutes ago, D13H4RD2L1V3 said:

I don’t get it.

 

They did it because other companies do it too? What?

I think they were trying to say if they don't do it then they look bad against other phones that do, specifically in the Chinese market.  It seems to be becoming a trend in China considering dell used cheating as a selling point for one of their laptops.

[D13H4RD]

5 minutes ago, mr moose said:

I think they were trying to say if they don't do it then they look bad against other phones that do, specifically in the Chinese market.  It seems to be becoming a trend in China considering dell used cheating as a selling point for one of their laptops.

And that's sad because they didn't buck that trend.

[mr moose]

Just now, D13H4RD2L1V3 said:

And that's sad because they didn't buck that trend.

Cultures evolve through consumer demand as much as they do by corporate governance.

[vorticalbox]

Have you ever turn off the thermal limits on a phone? I did with my HTC one x.

 

Thing was running at 98c on one of the frozen things you put in cooler bags.

 

Put out some impressive numbers though.

[AlTech]

12 hours ago, Bouzoo said:

It's like Huawei can't make a damn good things nowadays. 

Yeah, and doing this is just misleading the consumer..

[Amazonsucks]

A company thats basically an arm of the chinese communist party cheating on a benchmark? Say it aint so!

[WMGroomAK]

On 9/4/2018 at 6:50 PM, VegetableStu said:

if huawei wants to do that, at least leave in a "drain my battery" optional mode that throws all efficiencies away ._.

Well the good news for you is that according to Huawei's Official statement that is floating around, they are planning to provide users with access to the 'Performance Mode' meaning you too can enable battery draining and overheating mode on your phone.  

 

https://www.theregister.co.uk/2018/09/05/huawei_on_smartphone_benchmarks/

Quote

In a statement, Huawei told us today:

Quote

Huawei always prioritizes the user experience rather than pursuing high benchmark scores – especially since there isn’t a direct connection between smartphone benchmarks and user experiences. Huawei smartphones use advanced technologies such as AI to optimize the performance of hardware, including the CPU, GPU and NPU.

 

When someone launches a photography app or plays a graphically-intensive game, Huawei’s intelligent software creates a smooth and stable user experience by applying the full capabilities of the hardware, while simultaneously managing the device’s temperature and power efficiency. For applications that aren’t as power intensive like browsing the web, it will only allocate the resources necessary to deliver the performance that’s needed.

 

In normal benchmarking scenarios, once Huawei’s software recognizes a benchmarking application, it intelligently adapts to 'Performance Mode' and delivers optimum performance. Huawei is planning to provide users with access to 'Performance Mode' so they can use the maximum power of their device when they need to.

 

Huawei – as the industry leader – is willing to work with partners to find the best benchmarking standards that can accurately evaluate the user experience.

 

Still seems kind of weird double talk with the whole 'we prioritize user experience over benchmark scores however we enable our performance mode when you benchmark'. 

[Magnetorheological]

Unacceptable! It's time for me to chuck my P20 Pro out the window!

[LAwLz]

3DMark has started delisting Huawei devices from their website because of this.

If you ask me, that's a shame that they are being delisted. Even if the scores were artificially inflated, it would have given a rough indicator of how well they performed.

Then again, hopefully this will make Huawei stop cheating and then they will be allowed back.