Intel 9th Gen CPU Tests: Flaws in Testing Metheds

[Imbellis]

Edit: Oof, I'm unsure if this thread should fall under duplicate; However, "Intel 9th Gen Paid Benchmarks" focuses more on the results while this focuses on the flawed testing methodologies.

 

Gamer's Nexus, as well as many others, have recently posted a video harshly critiquing the testing methodology due to variance and poorly thought out procedures in their testing methodology from Principled Technologies:

Quote

"Given the level of general incompetence here, you have to start questioning (all of the benchmark results posted)."

 

In addition, others such as those from Forbes, PCGamesN, HawdwareUnboxed, TechSpot, and many others commented on their thoughts regarding the flaws:

Quote

"We don’t know what Z390 board Intel was using, or whether it was capable of hitting a 5GHz all-core speed out-of-the-box, so the actual clock speed of the chip is a bit of a mystery. "

"Principled Technologies either used in-game benchmarking tools or FRAPs to record the average FPS of games that don’t include them. " - PCGamesN (Although FRAPS can measure frames differently from games)

 

 

 

Quote

"The benchmarks carried out by Principled Technologies are even more bogus than we first thought. A few viewers pointed out that the Ryzen 7 2700X was listed as tested in the “Game Mode” within the Ryzen Master software and I foolishly thought they might have just made a simple copy and paste error in their document as they would have used this mode for the 2950X. This does explain why the Threadripper CPUs were faster than the 2700X in every test.

 

What this means is a CCX module in the 2700X was completely disabled, essentially turning it into a quad-core. I’ve gone ahead and re-run the XMP 2933 test with Game Mode enabled and now I’m getting results that are within the margin of error to those published by Principled Technologies." - HardwareUnboxed

 

Quote

"Ryzen doesn’t perform that well with fully populated memory DIMMs, two modules is optimal. However timings are also important and they used Corsair Vengeance memory without loading the extreme memory profile or XMP setting, instead, they just set the memory frequency to 2933 and left the ridiculously loose default memory timings in place. These loose timings ensure compatibility so systems will boot up, but after that point you need to enable the memory profile. It’s misleading to conduct benchmarks without executing this crucial step. " - TechSpot

 

Some of the factors that varied were:

• CPU coolers (Noctua U14S vs AMD's stock wraith cooler.
• Varying motherboards brands
• RAM speeds and RAM timings
• Using median results instead of getting an average
• Using FRAPS instead of the in-game frame counter
• Using GPU tests instead of CPU
• Use of flawed benchmarks (FFXV)

Edit: In addition:

• Possible variation fan speeds
• Omitting or forgetting to include testing constraints and other general info
• Possible variation of benchmark settings for GTA5

 

 

Overall, I think it is safe to say that intel's branding of this data, without any of their own confirmation, is extremely gullible or they are attempting to manipulate consumers (to a further degree than before) to buy their CPUs.

Oh, and there's the fact that Principled Technologies also disclaims any warranty regarding their tests so they're not held liable for these awful tests. Honestly, I think I'll be upgrading to a AMD CPU now instead of Intel.

 

Also, you should just ignore this thing below... the thing they included at the end of their test documentation.

 

 

 

 

[whoislinus]

This is an absolute travesty - if this isn't one of the main topics on the WAN show I'd be very surprised.

 

It's akin to shooting your competition before a race IMHO.

 

Running a 2700x as a quad-core in benchmarks...come on Intel FFS.

[Terryv]

I seriously question their (Intel) intelligence. The marketing department should know these will be picked apart under a microscope. 

 

All this does at best is make them look incompetent , and at worst outright lying. 

 

I hope someone gets fired over this.

[Spongy141]

2 hours ago, Terryv said:

I seriously question their (Intel)ligence. The marketing department should know these will be picked apart under a microscope. 

 

All this does at best is make them look incompetent , and at worst outright lying. 

 

I hope someone gets fired over this.

2

fixed

[Pretzel4000]

GamersNexus interview with PT is up:

https://youtu.be/qzshhrIj2EY

@Imbellis

[Pretzel4000]

@VegetableStu LOL how do you embed?
I just gave up.

[Pretzel4000]

Just now, VegetableStu said:

oh this forum does it automatically when you paste the URL (if it's one of the few embeddable media extensions it supports) o_o

Huh. It didn't work when I did it. That's weird. nvm

[Pretzel4000]

Just now, VegetableStu said:

are you on mobile? on my side I have to do a few extra steps for the forum to recognise it as a link first ,_,

Nope. On my desktop. I edited my post like 3 times and it didn't do any of those times.

[Pretzel4000]

 

It worked when hit return a couple of times.

[MrRavens]

Don't mind me, I'm just hosing down the thread. Steve is roasting these guys left, right and center. 

[AnonymousGuy]

I'm watching this unfold and laughing at the amount of overreaction going on.  9900k is going to be the fastest chip on the block by a wide margin.  You can be as autistic you want complaining about cooler or ram or whatever the fuck, it isn't going to change that outcome.

 

Go read the 50 other reviews going online for more information.  You're an idiot if you're relying on basically marketing material.

 

Yeesh..this shit is like twitter drama for nerds.

[D13H4RD]

2 minutes ago, AnonymousGuy said:

I'm watching this unfold and laughing at the amount of overreaction going on.  9900k is going to be the fastest chip on the block by a wide margin.  You can be as autistic you want complaining about cooler or ram or whatever the fuck, it isn't going to change that outcome.

Except......no one is complaining that the 9900K is faster. 

 

It's the methodology, the gimping of the 2700X and especially releasing benchmarks before the NDA lifts that people have issue with. 

 

This is why you wait for independent reviews made by people whom are trusted by the greater community. It gives you a far better picture of how a system does in a significantly fairer manner 

[bitsandpieces]

12 hours ago, Imbellis said:

Varying motherboards brands

Seriously?

You have variances between models from same manufacturer and he is worried about variances between manufacturers? It is not even the same manufacturer of CPU

[GoldenLag]

28 minutes ago, AnonymousGuy said:

I'm watching this unfold and laughing at the amount of overreaction going on.  9900k is going to be the fastest chip on the block by a wide margin.  You can be as autistic you want complaining about cooler or ram or whatever the fuck, it isn't going to change that outcome.

 

Go read the 50 other reviews going online for more information.  You're an idiot if you're relying on basically marketing material.

 

Yeesh..this shit is like twitter drama for nerds.

You seem to forget something.

 

People know the 9900k is going to be the fastest chip for gaming and productivity. We know, because the 8700k/8086k excist.

 

People take issue with the testing that was done. Like the 9000 series launch was pretty much perfect, except the moment this came along.

[Shally]

30 minutes ago, AnonymousGuy said:

I'm watching this unfold and laughing at the amount of overreaction going on.  9900k is going to be the fastest chip on the block by a wide margin. 

Nobody is arguing it won't be, literally nobody here is. I don't know why you thought that's what people were pissed off about but it's clearly not that at all.

People are pissed off because people who don't spend hours a week researching and watching tech channels and forums will likely buy into the marketing by Intel and be very disappointed when they get their chips. It's bsaically false advertising and is shamed on by this community for very obvious reasons, and if you're okay with skewed and paid results, then maybe you aren't PCMR

[Rune]

1 hour ago, Pretzel4000 said:

 

It worked when hit return a couple of times.

Steve is a hero.

[leadeater]

1 hour ago, KendoSapion said:

then maybe you aren't PCMR

I will never identify as part of that, no thanks. 

[Jito463]

29 minutes ago, leadeater said:

I will never identify as part of that, no thanks. 

[D13H4RD]

3 hours ago, GoldenLag said:

You seem to forget something.

 

People know the 9900k is going to be the fastest chip for gaming and productivity. We know, because the 8700k/8086k excist.

 

People take issue with the testing that was done. Like the 9000 series launch was pretty much perfect, except the moment this came along.

Pretty much this. 

 

The performance metric of the 2700X and 8700K are all very well known at this point. That's why a lot of people, especially those who have used it noticed that the 2700X scored way lower than what it technically should be. 

 

A lot of us know the 9900K will be beefy in games. The 8700K was highly regarded as a good all rounder gaming CPU, so the 9900K would have to be better. But that's not the problem. It's not even close to that.

 

The problem is how the tests are made to skew the bar heavily towards Intel rather than ensuring that both systems are at their most optimal for the best possible scenarios in each. 

 

You can't say that it's fine for anyone to benchmark one system that's made optimal but gimp the other with non-optimal RAM timings, an objectively worse cooling solution and especially enabling a mode that disables a CCX, turning it into a 4C/8T CPU.

 

The thing is that Intel didn't have to do all of this. Why they felt it was needed to deceive even though their CPU would still perform really well in games is beyond me. I would think they've been paranoid of AMD ever since their storming success with Ryzen, but that's my guess 

[Misanthrope]

4 hours ago, AnonymousGuy said:

I'm watching this unfold and laughing at the amount of overreaction going on.  9900k is going to be the fastest chip on the block by a wide margin.  You can be as autistic you want complaining about cooler or ram or whatever the fuck, it isn't going to change that outcome.

 

Go read the 50 other reviews going online for more information.  You're an idiot if you're relying on basically marketing material.

 

Yeesh..this shit is like twitter drama for nerds.

So you really think people wouldn't care about a claim of "30 to 50% faster" vs one that is "5 to 10% faster"? (Neither of this claims is likely to be accurate in most scenarios btw)

 

They're both faster, in both cases the 2700x wouldn't be able to touch it sure. 

 

But if you take either one of these claims in a vacuum the buyers would go from "Woah my games are going to be so much better" to "Well it's nice but might as well invest the extra money on a better GPU and get significantly more performance for my money"

 

This is all about perception and controlling the message and yes, it sucks but people in general are well, fucking idiots and perception counts...a lot.

[mrthuvi]

1 hour ago, VegetableStu said:

 

^{(I don't subscribe to the extreme end of PCMR too ._.)}

It's actually burn his entire family in the Chinese sub.

[TOMPPIX]

watch it 

https://www.intel.com/content/www/us/en/benchmarks/benchmark.html

[yian88]

Conclusion as some other recent intel shit thread, dont buy intel, vote with your wallet, buy AMD this round, support them getting good Zen2/Zen3 down the road or until intel stops fucking with us taking us for idiots, probably never.

If you buy nvidia RTX and intel 9000 series oh boi, dont be surprised when they pull more of this shit. Keep in mind bois and girls, transistor shrinking technology is at its end of the journey, maybe we get 10/7/5nm, but after its all arhitecture/software improvement, and we need less of marketing shitty moves and more proper competition in the next decade, i think this is one of the most important times in desktop CPU/GPU market, if AMD cant get up and become a giant to hold its own against intel/nvidia, once the node shrinking trick is over if AMD cant compete and innovate new technologies, graphite, nanotubes,optics, hbm i dont know what,  intel/nvidia will fuck us hard with 5% improvement each 2 years at 1000$+ pricepoint.

[asus killer]

5 minutes ago, yian88 said:

 

If you buy nvidia RTX and intel 9000 series oh boi, dont be surprised when they pull more of this shit.

and yet RTX sold out... 

[RazorStorm]

Method*

Methodology is the study of methods. plz correct