UL [Benchmarks] delists Huawei due to inaccurate benchmark results after Anandtech snitches on Huawei "cheating"

[AlTech]

Previously on LTT Tech News:  Anandtech finds Huawei cheating in benchmarks

 

Current News:

UL, the current owner of UL Benchmarks which itself owns Futuremark which created 3DMark and other benchmarking products, has delisted Huawei devices due to inaccurate benchmark results caused by cheating.

 

Huawei devices were discovered to contain software that enabled their devices to "cheat" at benchmarks by running faster than it would normally in day to day usage.

 

Tested Huawei devices have been removed from the UL Benchmark's results database.

 

Anandtech notes that UL has contacted Huawei to address the issue.

 

Quote

A few days ago we published our article addressing Huawei and Honor’s inaccurate benchmarking behaviour. In a nutshell, we had found out that this year’s devices had introduced a new thermal limiting behaviour that quickly throttled power consumption to ~4-4.5W in 3D workloads. While this in itself wasn’t an issue, the problem is that the firmware did not apply this new behaviour to a specific list of whitelisted 3D benchmarks.

 

Anandtech also mentions that they snitched on Huawei to UL as soon as they were aware of the "cheating".

Quote

We work closely with all benchmark vendors, and UL isn’t an exception. We had first given UL note of the behaviour two weeks ago and had been sharing our early results with the development team

 

Quote

Today UL published their independent confirmation of our results on their own devices, and have subsequently decided to remove the tested devices from their results database.

UL stresses that the kind of detection and optimization performed by Huawei infringes the company’s rules for manufacturers. Unfortunately this isn’t the first instance of a vendor being delisted, as most famously a slew of phone manufacturer had been delisted in 2013 after a more in-depth investigation of ours resulted in quite embarrassing results for a lot of vendors.

The UL team further explains that they’ve also been in touch with Huawei, and that the Chinese vendor is planning to address the behaviour by introducing a new “performance mode” that disables the new thermal throttling behaviour. In essence this mode would revert back to the behaviour we’ve seen in the past such as the Mate 9 – where the SoC is allowed higher peak performance figures at a cost of high power.

 

I respect that UL owns their own benchmarks and has their own online results database but de-listing huawei devices because of this seems a bit extreme.

Honestly, I would probably have just made each Huawei device result show a big warning showing that the results are invalid but keep the results up for a short while at the very least.

 

Also, benchmark cheating is bad. And cheating in general is bad in case that wasn't already apparent.

 

Source:

https://www.anandtech.com/show/13334/ul-delists-huawei-devices-due-to-inaccurate-benchmarks

[AlTech]

Just now, valdyrgramr said:

Don't most companies do this?  Should delist them all.

Most of the phone manufacturers did at one point.

 

OnePlus and Samsung are quite famous for trying in the past.

 

But corporate management at UL seems to be more strict than Futuremark was.

[porina]

UL = Underwriter Labs, they have created various standards (e.g. flammability of materials) and took over FutureMark not that long ago.

Unilever is a different company

[AlTech]

25 minutes ago, porina said:

UL = Underwriter Labs, they have created various standards (e.g. flammability of materials) and took over FutureMark not that long ago.

Unilever is a different company

Wow.... was totally under the impression that it was owned by Unilever cos iirc when the acquisition was news, Linus said that Futuremark was owned by Unilever or something like that.

[S w a t s o n]

 

Top comment on the anandtech article, lmfao

[Thaldor]

3 minutes ago, AluminiumTech said:

Wow.... was totally under the impression that it was owned by Unilever cos iirc when the acquisition was news, Linus said that Futuremark was owned by Unileve or something like that.

Nope, totally different company. Underwriter Labs is well known testing laboratories and standardisation company which keeps UL-standards and is also known for UL/"RU"-quality mark. Unilever (only UL I can find is that NYSE stocks for Unilever are labeled UN (not to be confused with United Nations) and UL) is probably biggest global consumer goods megacompany which produces food, beverages, cleaning chemicals and personal hygiene products like AXE, OMO, Dove, Hellman's, Lipton and so on.

 

I can understand why they would rather ban manufacturer from scores rather than just show a warning. Because considering that they want to keep the "leaderboard" as factual as possible for it to be scientifically accurate, having cheaters on there would be a big no no.

[LinusTech]

6 hours ago, AluminiumTech said:

Wow.... was totally under the impression that it was owned by Unilever cos iirc when the acquisition was news, Linus said that Futuremark was owned by Unilever or something like that.

Don't recall ever saying that... Citation needed

[Rune]

41 minutes ago, LinusTech said:

Don't recall ever saying that... Citation needed

Wan show November 7th 2014 @ 1:19:58 according to timestamps. Looking for video.

 

Edit: Here we go:

 

Hm, only ever referred to as "UL"

[Carclis]

I think this says a lot for the relevance of benchmarking apps on mobile devices.

[mr moose]

Why delist them?  just put a qualifier next to the entry so people know how the result was achieved. 

[spartaman64]

12 hours ago, S w a t s o n said:

 

Top comment on the anandtech article, lmfao

i guess capitalizing the To is weird but its English convention to capitalize every word in a title except for a couple of exceptions

[ScratchCat]

2 hours ago, mr moose said:

Why delist them?  just put a qualifier next to the entry so people know how the result was achieved. 

That would just encourage others to do the same - "We can get the highest benchmark number at the cost of a little yellow warning symbol most people will ignore"

 

This isn't the equivalent of GPU Boost 2.0 but overclocking for benchmarks only as well as overriding the default thermal limits. A benchmark is support to be a general indicator of performance, overriding all default settings to get higher scores makes the score useless and therefore should be seen as invalid.

[mr moose]

9 hours ago, ScratchCat said:

That would just encourage others to do the same - "We can get the highest benchmark number at the cost of a little yellow warning symbol most people will ignore"

 

This isn't the equivalent of GPU Boost 2.0 but overclocking for benchmarks only as well as overriding the default thermal limits. A benchmark is support to be a general indicator of performance, overriding all default settings to get higher scores makes the score useless and therefore should be seen as invalid.

It doesn't have to be a yellow symbol, it can be the results in a whole different graph.   So long as the results are qualified and the average user knows the deference then whether that encourages them or not is moot.