Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
ocanada_techguy1

Mobile benchmark cheating: Andrei Frumusanu on Anandtech re-opens scandal

Recommended Posts

Posted · Original PosterOP

Wow.  Kudos to Andrei Frumusanu for his vigilance and investigative works here.   For those who don't remember the article touches upon the earlier discoveries and reporting of cheating by some very major makers of mobile devices.  In this new case there is question that this is being done by and included by the chipmaker placed onto the phones of various OEMs.  In this case the chip maker noticed is MediaTek.  The tweaks cover a huge range of benchmarks and in some cases go so far as to target the copy of the benchmark used by editorial reviewers.  Further, upon discovery and inquiries opened, it would appear said chipmaker simply moved their benchmark cheats to elsewhere but they remain.  Their statement in response is along the lines: everyone does it, normal, nothing to see here.  Some will consider it shockingly blatant, the author seems to think so, while for others this might come as no surprise at all.

 

https://www.anandtech.com/show/15703/mobile-benchmark-cheating-mediatek

 

 

 

Mobile_Benchmark_Cheating_Anandtech_April_8_2020.png

Link to post
Share on other sites

I haven't paid any attention to a mobile bench mark ever,  it would be like making your decision using only the spec sheet.   


QuicK and DirtY. Read the CoC it's like a guide on how not to be moron.  Also I don't have an issue with the VS series.

Sometimes I miss contractions like n't on the end of words like wouldn't, couldn't and shouldn't.    Please don't be a dick,  make allowances when reading my posts.

Link to post
Share on other sites

Just have benchmark software makers constantly change their app IDs and elude these cheats? Even better, have the benchmark app randomize its app ID every time it is opened.

 

Although in reality I seriously doubt the vast majority of people care that much about benchmark scores, if they even care at all. Most people who settle on Mediatek chipsets are doing so for the lower device cost, and they kind of already acknowledge that they are not getting a cutting edge number crunching device.

 

Many people who insist on flagships from the top brands end up just using it for social media and media consumption and never really use the full potential of the hardware.

 

Nevertheless, cheating is cheating.

 

For me I never buy Mediatek devices anyway. Custom roms are too important for me.


Awareness is key. Never enough, even in the face of futility. Speak the truth as if you may never get to say it again. This world is full of ugly. Change it they say. The only way is to reveal the ugly. To change the truth you must first acknowledge it. Never pretend it isn't there. Never bend the knee.

 

Please quote my post in your reply, so that I will be notified and can respond to it. Thanks.

Link to post
Share on other sites

The first false assumption is that MediaTek ever made a fast mobile processor.


Specs: Motherboard: Asus X470-PLUS TUF gaming (Yes I know it's poor but I wasn't informed) RAM: Corsair VENGEANCE® LPX DDR4 3200Mhz CL16-18-18-36 2x8GB

            CPU: Ryzen 7 2700X @ 4.2Ghz          Case: Antec P8     PSU: G.Storm GS850                        Cooler: Antec K240 with two Noctura Industrial PPC 3000 PWM

            Drives: Samsung 970 EVO plus 250GB, Micron 1100 2TB, Seagate ST4000DM000/1F2168 GPU: EVGA RTX 2080 ti Black edition @ 2Ghz

                                                                                                                             

Link to post
Share on other sites

This is why I have trust issues when it comes to mobile benchmarks.

Seriously, I was contemplating getting a MediaTek based chromebook before, the price was attractive and benchmarks for the chip inside looked excellent, on par with Intel's more expensive quad core CPUs chromebooks and all that.... Then you read actual customer feedback on the devices and for the mediatek it was nothing but "slow, takes a while to do anything" while the intel was "blazing fast, no lag".

 

So this article explains a lot.


CPU: AMD Ryzen 3600 / GPU: Radeon HD7970 GHz 3GB(upgrade pending) / RAM: Corsair Vengeance LPX 2x8GB DDR4-3200
MOBO: MSI B450m Gaming Plus / NVME: Corsair MP510 240GB / Case: TT Core v21 / PSU: Seasonic 750W / OS: Win 10 Pro

Link to post
Share on other sites

This again, yeah...especially on phones where many of them can't handle prolonged load so they just boost for short bursts for benchmarks. Really need some better way...


Ryzen 7 3800X | X570 Aorus Elite | G.Skill 16GB 3200MHz C16 | Radeon RX 5700 XT | Samsung 850 PRO 256GB | Mouse: Zowie S1 | OS: Windows 10

Link to post
Share on other sites

-= Topic moved to General discussion =-

 

@ocanada_techguy1 Please update your topic to meet the posting guidelines [see quote below) and your topic will be moved back to the Tech News Sub-forum. Please contact a Mod when completed.

 

Quote

When creating a thread in the News subforum, please make sure your post meets the following criteria:

  • Your thread must include some original input to tell the reader why it is relevant to them, and what your personal opinion on the topic is. This needs to be MORE than just a quick, single comment to meet the posting guidelines.
  • Your thread must include a link to at least one reputable source. Most of the time, this should be a respected news site.
  • Your thread should also include quotes from the cited source(s). While you shouldn't just copy the entire article, your quote should give the reader a summary of the article in a way that gives the key details, but also leaves room for them to read the full article on the linked website. Please use quote tags to show that you have copied this content from another site.
  • The title of your thread must be relevant to the topic and should give a reader a good idea of the contents of the thread. Copying the title of the source is permitted but absolutely not required. It should be to the point and not be done in such a way as to mislead a reader, such as clickbait, etc.
  • If your article is about a product or some form of media, images are always appreciated, although they are not required.

Failure to comply may result in your thread being locked or removed without warning.

 


Tech News Posting Guidelines - READ BEFORE POSTING | Community Standards | Forum Staff

LTT Folding Users Tips, Tricks and FAQ | F@H Contribution | My Rig | Project Steamroller

 

Spoiler

 †  

 

Character is like a Tree and Reputation like its Shadow. The Shadow is what we think of it; The Tree is the Real thing.  ~ Abraham Lincoln

You have enemies? Good. That means you've stood up for something, sometime in your life.  ~ Winston Churchill

Docendo discimus - "the best way to learn is to teach" ~ Benjamin Jantz

 

I am a StarCitizen are you? My ships: Aegis Eclipse, Aegis Sabre, Aegis Gladius, Aopoa Nox, KI P52 Merlin, KI P72 Archimedes and the RSI Constellation Aquila.

 

My Phones are a Nokia Lumia 925 with WM10 and a Microsoft Lumia 950 XL with WM10 running the Fast Ring insider updates. Broke :(

Samsung Note 9 and a Samsung S9+

 

About Myself:   https://linustechtips.com/main/profile/229093-sansvarnic/?tab=field_core_pfield_46

 

 CHRISTIAN MEMBER 

 

 

Link to post
Share on other sites
On 4/9/2020 at 12:59 AM, mr moose said:

I haven't paid any attention to a mobile bench mark ever,  it would be like making your decision using only the spec sheet.   

I don't think it's unreasonable to check benchmarks for phones. It gives a great indication of how things are progressing. I don't think I would ever use something like a single CPU or GPU benchmark to go "I am buying this phone over this one" because it would ignore way too many aspects, but it is an important thing nonetheless. For example if you need a new phone and you used to have a flagship from a few generations ago, it might be interesting to know which midrange chip is equivalent to your old phone. It can make you avoid "side-grades" in terms of SoCs. Or if two phones are somewhat similar but has different SoCs (for example a lot of Chinese brands make very small iterations to their phones so sometimes only the SoC is the difference between two models) it is a good idea to check the performance of the two.

 

 

On 4/9/2020 at 6:50 AM, williamcll said:

The first false assumption is that MediaTek ever made a fast mobile processor.

1) I think phone benchmarks are more important for low and mid-range devices than they are for high end devices. With high-end devices you usually sacrifice a lot of back for your buck in favor of the best of the best, which usually boils down to whichever the latest high-end Qualcomm SoC is.

With midrange devices you got far more options, and it is far more important to know how much faster than 20 dollar more expensive phone really is, and if it's worth the extra money.

 

2) MediaTek will actually make a high end SoC soon. The Dimensity 1000 looks to be a beast.

4x Cortex A77 cores at 2.6GHz

4x Cortex A55 at 2.0GHz

Mali G77MP9 GPU

Pretty decent NPU performance (4.5 TOPs on the NPU alone)

WiFI6 and sub-6 5G modem integrated.

N7 manufacturing process.

4K60p encode and decode for H.264, HEVC and AV1 (decode only for AV1).

 

So I'd say it's mix of very high end (CPU cores, manufacturing node), and match last gen's high end (with stuff like the NPU and GPU).

Link to post
Share on other sites
28 minutes ago, LAwLz said:

I don't think it's unreasonable to check benchmarks for phones. It gives a great indication of how things are progressing. I don't think I would ever use something like a single CPU or GPU benchmark to go "I am buying this phone over this one" because it would ignore way too many aspects, but it is an important thing nonetheless. For example if you need a new phone and you used to have a flagship from a few generations ago, it might be interesting to know which midrange chip is equivalent to your old phone. It can make you avoid "side-grades" in terms of SoCs. Or if two phones are somewhat similar but has different SoCs (for example a lot of Chinese brands make very small iterations to their phones so sometimes only the SoC is the difference between two models) it is a good idea to check the performance of the two.

 

 

 

I don't think you can ascertain that data from any of the benchmarks available even if they weren't cheating.

 

 

Apart from the fact they are generic/synthetic and therefore not ideal for any one specific user,  changes to the benchmark and updates create differences in results that make comparing old devices to new devices pointless.

 

 

 


QuicK and DirtY. Read the CoC it's like a guide on how not to be moron.  Also I don't have an issue with the VS series.

Sometimes I miss contractions like n't on the end of words like wouldn't, couldn't and shouldn't.    Please don't be a dick,  make allowances when reading my posts.

Link to post
Share on other sites
1 hour ago, mr moose said:

I don't think you can ascertain that data from any of the benchmarks available even if they weren't cheating.

 

 

Apart from the fact they are generic/synthetic and therefore not ideal for any one specific user,  changes to the benchmark and updates create differences in results that make comparing old devices to new devices pointless.

Especially on Android devices, where the OS is probably several versions different as well, with different "branding" bloatware depending on which company it is coming from too.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×