Mobile benchmark cheating: Andrei Frumusanu on Anandtech re-opens scandal

[ocanada_techguy1]

Wow.  Kudos to Andrei Frumusanu for his vigilance and investigative works here.   For those who don't remember the article touches upon the earlier discoveries and reporting of cheating by some very major makers of mobile devices.  In this new case there is question that this is being done by and included by the chipmaker placed onto the phones of various OEMs.  In this case the chip maker noticed is MediaTek.  The tweaks cover a huge range of benchmarks and in some cases go so far as to target the copy of the benchmark used by editorial reviewers.  Further, upon discovery and inquiries opened, it would appear said chipmaker simply moved their benchmark cheats to elsewhere but they remain.  Their statement in response is along the lines: everyone does it, normal, nothing to see here.  Some will consider it shockingly blatant, the author seems to think so, while for others this might come as no surprise at all.

 

https://www.anandtech.com/show/15703/mobile-benchmark-cheating-mediatek

 

 

 

[mr moose]

I haven't paid any attention to a mobile bench mark ever,  it would be like making your decision using only the spec sheet.   

[williamcll]

The first false assumption is that MediaTek ever made a fast mobile processor.

[TetraSky]

This is why I have trust issues when it comes to mobile benchmarks.

Seriously, I was contemplating getting a MediaTek based chromebook before, the price was attractive and benchmarks for the chip inside looked excellent, on par with Intel's more expensive quad core CPUs chromebooks and all that.... Then you read actual customer feedback on the devices and for the mediatek it was nothing but "slow, takes a while to do anything" while the intel was "blazing fast, no lag".

 

So this article explains a lot.

[Doobeedoo]

This again, yeah...especially on phones where many of them can't handle prolonged load so they just boost for short bursts for benchmarks. Really need some better way...

[SansVarnic]

-= Topic moved to General discussion =-

 

@ocanada_techguy1 Please update your topic to meet the posting guidelines [see quote below) and your topic will be moved back to the Tech News Sub-forum. Please contact a Mod when completed.

 

Quote

When creating a thread in the News subforum, please make sure your post meets the following criteria:

• Your thread must include some original input to tell the reader why it is relevant to them, and what your personal opinion on the topic is. This needs to be MORE than just a quick, single comment to meet the posting guidelines.
• Your thread must include a link to at least one reputable source. Most of the time, this should be a respected news site.
• Your thread should also include quotes from the cited source(s). While you shouldn't just copy the entire article, your quote should give the reader a summary of the article in a way that gives the key details, but also leaves room for them to read the full article on the linked website. Please use quote tags to show that you have copied this content from another site.
• The title of your thread must be relevant to the topic and should give a reader a good idea of the contents of the thread. Copying the title of the source is permitted but absolutely not required. It should be to the point and not be done in such a way as to mislead a reader, such as clickbait, etc.
• If your article is about a product or some form of media, images are always appreciated, although they are not required.

Failure to comply may result in your thread being locked or removed without warning.

 

[LAwLz]

On 4/9/2020 at 12:59 AM, mr moose said:

I haven't paid any attention to a mobile bench mark ever,  it would be like making your decision using only the spec sheet.   

I don't think it's unreasonable to check benchmarks for phones. It gives a great indication of how things are progressing. I don't think I would ever use something like a single CPU or GPU benchmark to go "I am buying this phone over this one" because it would ignore way too many aspects, but it is an important thing nonetheless. For example if you need a new phone and you used to have a flagship from a few generations ago, it might be interesting to know which midrange chip is equivalent to your old phone. It can make you avoid "side-grades" in terms of SoCs. Or if two phones are somewhat similar but has different SoCs (for example a lot of Chinese brands make very small iterations to their phones so sometimes only the SoC is the difference between two models) it is a good idea to check the performance of the two.

 

 

On 4/9/2020 at 6:50 AM, williamcll said:

The first false assumption is that MediaTek ever made a fast mobile processor.

1) I think phone benchmarks are more important for low and mid-range devices than they are for high end devices. With high-end devices you usually sacrifice a lot of back for your buck in favor of the best of the best, which usually boils down to whichever the latest high-end Qualcomm SoC is.

With midrange devices you got far more options, and it is far more important to know how much faster than 20 dollar more expensive phone really is, and if it's worth the extra money.

 

2) MediaTek will actually make a high end SoC soon. The Dimensity 1000 looks to be a beast.

4x Cortex A77 cores at 2.6GHz

4x Cortex A55 at 2.0GHz

Mali G77MP9 GPU

Pretty decent NPU performance (4.5 TOPs on the NPU alone)

WiFI6 and sub-6 5G modem integrated.

N7 manufacturing process.

4K60p encode and decode for H.264, HEVC and AV1 (decode only for AV1).

 

So I'd say it's mix of very high end (CPU cores, manufacturing node), and match last gen's high end (with stuff like the NPU and GPU).

[mr moose]

28 minutes ago, LAwLz said:

I don't think it's unreasonable to check benchmarks for phones. It gives a great indication of how things are progressing. I don't think I would ever use something like a single CPU or GPU benchmark to go "I am buying this phone over this one" because it would ignore way too many aspects, but it is an important thing nonetheless. For example if you need a new phone and you used to have a flagship from a few generations ago, it might be interesting to know which midrange chip is equivalent to your old phone. It can make you avoid "side-grades" in terms of SoCs. Or if two phones are somewhat similar but has different SoCs (for example a lot of Chinese brands make very small iterations to their phones so sometimes only the SoC is the difference between two models) it is a good idea to check the performance of the two.

 

 

 

I don't think you can ascertain that data from any of the benchmarks available even if they weren't cheating.

 

 

Apart from the fact they are generic/synthetic and therefore not ideal for any one specific user,  changes to the benchmark and updates create differences in results that make comparing old devices to new devices pointless.

 

 

 

[justpoet]

1 hour ago, mr moose said:

I don't think you can ascertain that data from any of the benchmarks available even if they weren't cheating.

 

 

Apart from the fact they are generic/synthetic and therefore not ideal for any one specific user,  changes to the benchmark and updates create differences in results that make comparing old devices to new devices pointless.

Especially on Android devices, where the OS is probably several versions different as well, with different "branding" bloatware depending on which company it is coming from too.